Mel Slater's Presence Blog

Thoughts about research and radical new applications of virtual reality - a place to write freely without the constraints of academic publishing,and have some fun.

12 November, 2017

In the Presence of Jaron Lanier

In the Presence of Jaron Lanier

The talk by Jaron Lanier Friday 11th November 2017 at UCL was attended by about 150 people. Jaron talked about themes from his new book Dawn of the New Everything: A Journey Through Virtual Reality. It is best to read the book of course, but here I will simply say that he introduced the Thesis - that virtual reality offers a fantastic way for the construction of new shared realities, where we can experience each others' dreams. He also introduced the Antithesis that VR offers a perfect means for behavioural modification - since it can act as a 'Skinner box' where a person's environment is completely controlled, and based on feedback in response to their actions and choices algorithms can move them surreptitiously towards new states of mind. The Synthesis ....

Introducing Jaron Lanier

Start of the talk

The talk was very inspiring and, judging by the questions, and the comments afterwards, was very much appreciated and enjoyed.

Here I want to say that it provided the opportunity for previous members of my group, now friends, to get together even if briefly, and renew old ties, and perhaps plan a future gathering. In the pictures below all but Zillah Watson, Aiden O'Neill (and Jaron Lanier) were in the Virtual Environments and Computer Graphics group at UCL at some time. I have indicated those that were postdocs, and PhD students by the year that they obtained their PhD.

Left to right: Daniela Romano (ex postdoc), Vinoba Vinayagamoorthy (2006), Joel Jordan (2010), Marco Gillies (ex postdoc)

Vinoba with Joel

Lee Pip Bull (2006) with Vinoba

Left to right: Zillah Watson, Anthony Steed (1996),
Aiden O'Neill (Penguin Books), Jaron Lanier, 
Mel Slater, Sylvia Xueni Pan (2010 and postdoc)

26 May, 2017

Ghost Presence

While attending WVRF17 I was lucky to experience two immersive VR movie experiences. The first was Through You by Saschka Unseld and Lily Baldwin and the second Defrost: The Virtual Series by Randal Kleiser and Tanna Frederick. They are both 360 degree stereo movies, rendered in Samsung Gear VR. They are both beautifully made and very exciting advances on VR cinematography.

However, here I want to concentrate on the form rather than the content since they are quite different in the kind of presence illusions they evoke. In Through You, you are a witness to a kind of dance that tells the story of a romance over time: what was, and what might have been. In the set up I experienced you physically stand near a wall covered with what felt like a velvet curtain, and were advised to be facing and touching that wall with both hands in order to maintain balance and orientation. Throughout the virtual performance I found that I did keep at least one hand touching the wall, which acted as a kind of anchor. However, the events were taking place anywhere in the scene, so I was often touching the wall even while turned around 180 degrees from my starting position.

In my theory of presence, the basis of the perceptual illusion of ‘being there’ is that the system supports natural sensorimotor contingencies for visual perception. So the extent to which I can use my body and perceive the virtual world in a normal way (head turning, bending down, looking around, reaching out and touching, and seeing wide field of view, with stereo vision, high resolution etc – for vision) the simplest hypothesis for the brain do adopt is “This is where I am”.

In both movies there were head based visual sensorimotor contingencies. The Gear VR does not support full 6 df head-tracking but only head orientation. But since in both movies the viewer is only required to look around while remaining in the same place, this is not very noticeable. So we can say that there were reasonable head-based visual sensorimotor contingencies.

In Through You it was a shock to look down towards myself and realise that I had no body. Our first ever paper in the field of VR (VRAIS, 1993) concentrated on the importance of having a self representation in the virtual environment, a finding that has been supported by more recent results. So even though as you look around, you see the world in 3D stereo, with reasonable field of view, latency and resolution, so that you get approximate natural sensorimotor contingencies, there is still something critically missing in the scene – You. I personally felt a kind of ghostly ‘place illusion’, I was there and not there, some kind of in-between state.

In Through You you are an observer, a witness. The environment does not respond to you in any way. The actors are completely unaware of your presence. One of my proposed bases of the ‘Plausibility Illusion’ is that the virtual world responds to you, i.e., that you can influence what happens, and another is that there are events that relate personally to you. Plausibility refers to the illusion that ‘This is really happening  - now’ (even though you know that this is not the case). This illusion that events are happening is meant in a very personal sense – that the events can impinge on you – that someone may be looking at you, that they can get into your space, that the events may be dangerous or beneficial to you personally. I guess it is something to do with our basic survival instinct, that we continually evaluate surrounding events for possible threat, and that the Plausibility Illusion is concerned with this feeling. Without this personal reality in relation to the ongoing events, you are an observer – very much like you are the observer of a film or stage play. Of course the events can influence your emotions, your appreciation of the aesthetics, you may relate the events to your own life – but the events themselves are not physically connected to you. We cannot call this Ghost Plausibility, since in this type of situation there is no Plausibility at all in the strict sense of the meaning of ‘Plausibility Illusion’ (Psi).

Nevertheless ‘Ghost Place Illusion’ (sensorimotor contingencies but no self representation) and the absence of Psi do give rise to a strange feeling – I will call it Ghost Presence. You are like a Ghost, being in and simultaneously not being in a scenario that unfolds completely independently of your will. From that perspective you can attain a special perspective on the events that you witness. It is somewhat like watching a conventional movie, but your Ghost Presence gives you a familiarity with the space and a closeness to the characters that you cannot get from a movie. It is a different experience, a different art form. This is also superbly illustrated by the prize winning After Solitary by Emblematic. Here you are placed in a virtual replication of a tiny prison cell in which the (real) protagonist spent five years in solitary confinement. You have a first hand experience of the cell, its size, its squalor. From that you can begin to imagine the horror of living 5 years alone (or even 5 hours) in such conditions. The prisoner is there describing what happened to him and his feelings. However, the prisoner is not addressing you personally, he doesn’t look at you or invoke your presence in any way. He is speaking to a general audience. His story is powerful, he is telling it to anyone and everyone but not you specifically. You have the Ghost Presence, where especially the experience of the space is extremely powerful and unforgettable. The monologue of the prisoner provides additional information that also has a powerful emotional impact.

In Defrost you awake after being frozen in liquid nitrogen for 30 years after suffering a stroke (that was incurable at that time). Now the stroke has been cured. A doctor is telling you all this, and adds that due to the fact of your 30 years of sleep you are unable to move, except for your head. When you look down towards yourself you see you are embodied and in a wheel chair. This is 360 degrees and stereo video playback. Since you can look around but in a fixed position the tracking of the Samsung Gear is sufficient to provide reasonable visual sensorimotor contingencies, and there is corresponding Place Illusion. Since you have a body it is not a ghostly PI. The explanation of being unable to move because of the long sleep helps with respect to Plausibility, and a whole series of characters come to talk with you personally including the doctors and (now grown up) children. Over several episodes a drama unfolds in which you play the major role. As well as being a brilliant idea it is very well designed and acted, and a great example of a new kind of first person narrative experience that is going to emerge from VR.

It is important to note that I am not comparing Defrost and Through You from the point of view of their creativity and execution or as experiences, but rather the illusions that they evoke (in me). Through You results in Ghost Presence, which is a very unique experience, being simultaneously there and not there, in the scene and not in it. The contradictions give rise to a kind of tension which is itself a new experience – one that we cannot get at all from any other media. As I said, it is a new kind of art form. Defrost shows how narratives that exploits the PI and Psi illusions  might develop – and especially the conjunction of PI and Psi in the illusion of body ownership.  Through You and Defrost offer different qualities of experience, each created with very high production values, it is not that one is ‘better’ than the other in any way.

When we have used questionnaires to assess presence (e.g., in) we have invariably included the question:

“When you think back about your experience, do you think of the [environment X] more as images that you saw, or more as somewhere that you visited?”

Now when I spend time in a computer graphics model-based VR, my answer to that question has typically a very high score (e.g., 6 or 7 on a 1-7 scale where 1 means ‘images that you saw’ and 7 means ‘somewhere that you visited’). I think back to some environments I have experienced recently, and it is the case I have the feeling to have been somewhere rather than just seen something.

However, when I think back to my experience of Through You and Defrost, my score is pretty low for both, 2/7 – I remember the images, rather than of being somewhere. For Through You this is understandable, because of the Ghost Presence illusion. For Defrost since there were pretty good visual sensorimotor contingencies, I should have the memory of having been somewhere not just seen images. So why is this?

I think this may be to do with being in a virtual world created by video rather than graphics. We are really used to seeing video, it is everywhere throughout our lives. We have never been in world that looks like video. We know that the characters in video have zero information about us. We know that video is something that ‘just plays’ (most of the time) irrespective of anything we do (other than fast forward/back or switch off). In other words video comes with a whole set of very powerful and ingrained expectations. Model based graphics VR does not – we see an environment and virtual characters, that look ‘sort of’ like the real thing, but are not. We don’t know what to expect. Typically there is strong interaction in these worlds whereas there is very minimal interaction with video. So it is possible that the very powerful expectations and associations with video act as a top down dampening of the illusions normally associated with VR. So although during an experience I might have some feeling of ‘being there’ and that the events are ‘really happening’, as soon as I’ve finished, what I remember is the video. I haven’t ‘been there’ but I’ve ‘seen it’.

This is not an Uncanny Valley effect, where because the video is almost but not quite perfect, we reject it. It is an expectations and associations based effect, where cognition to some extent is overriding perception – since we know in a very deep and experiential way what video is, and therefore if this looks like video (a) I can’t be here and (b) this isn’t really happening right now.

14 November, 2016

The fallacy of the large sample size argument

Another bureaucratic solution to problems of science is being increasingly heard. This is the idea that papers that report experiments with small sample sizes should be "desk rejected". This is a relatively new phenomenon. Recently I have had two referees on two different papers mention this point. I have seen tweets suggesting that anything less than n = 100 won't do. This is supposed to contribute to the solution of the 'crisis of reproducibility' - especially in psychology and related disciplines.

Such proposals for bureaucratic solutions with fixed norms do not take account of elementary statistics. The sample size needed for estimation or hypothesis testing is relative to the variance of the variables involved. 

Take the simplest example - where a normally distributed random variable X has unknown mean m and known variance s2. The experiment will deliver a sample of n independent observations on X. The goal is to estimate m, with a 95% confidence interval, u to v. Critically we want to choose a sample size n such that v-u< e.

The 95% confidence interval is  xbar ± 1.96*s/sqrt(n) where xbar is the sample mean.

Therefore we require:
3.92s/sqrt(n) < e, leading to n > (3.92s/e)^2. 

The sample size needed is proportional to the variance. For example, suppose that s = e. Then a sample size of 16 is good enough.  Suppose that s = 0.80*e, then a sample size of around 10 would be good enough. A sample size of 100 would be needed if s would be 2.55 times e. It is clearly the variance in relation to the required error size that is important.

This really matters because running experiments is typically very expensive. If a sample size of 10 would do why use 100? It would only be to satisfy a slogan, not for any real contribution to more reliable statistical inference.

The supposed failure of reproducibility is built into the system and will not be overcome by adopting slogans. In classical statistical inference with 5% significance levels, according to the theory the null hypothesis will be wrongly rejected approximately 5% of the time anyway. No researcher can know if their experiment is one of the 5%! This is the point of repeat studies. Yet repeat studies are hard to publish (no novelty) and if indeed they do find results at odds with the original study then the researchers on the original study may have their integrity questioned. Yet who can know whether the second study is one of the 20% that will falsely not reject the null hypothesis (assuming a power of 80%)!

The Bayesian approach does not have these problems. An experiment will result in probabilities about hypotheses or probability distributions over parameters rather than fixed answers or conclusions. As more data are collected through repetition studies so these probabilities will be updated. Also it should be noted that a similar argument used in the example above will hold for finding a Bayesian credible interval. 

To paraphrase a well known saying: "It's the variance, stupid!" 

17 April, 2016

Pre-Registration of Analysis of Experiments is Dangerous for Science

The idea of pre-registration of experimental analysis is supposed to be one amongst a set of solutions to the crisis in the explosion of false results making it into the scientific literature - especially psychology and related sciences. It is argued that pre-registration of analyses will curb the over-analyses of data that is carried out in a search for ‘significant’ results.

Here I argue that such a move will not only not solve any crisis but that it will have detrimental effects on science. The idea arises as a false notion of how scientific research operates. In the accepted ideology the scientist has a ‘hypothesis’ and then formulates this as a ‘null hypothesis’ (H0) and an ‘alternative hypothesis’ (H1) (of course there could be sets of these for any particular experiment). H0 is set up to mean that the original scientific hypothesis is ‘false’. Specifically H0 is chosen to satisfy simple mathematical properties that will make the resulting statistical analysis quite easy to do, following well-known text book formulae. For example, the scientist may believe that experimental condition E will produce a higher value on some critical variable y than experimental condition C (other things being equal). In such a case H0 would typically be that μC = μE , and H1 that  μE > μC. Under the null hypothesis the distribution of the test statistic t for the difference of two means is known, but critically under certain statistical assumptions. Depending on the results of the computed test statistic t after the experiment, H0 is rejected (if t falls in some critical region) or not. H0 not being rejected is not evidence for it being 'true'. Classical statistics does not even permit the researcher to give odds or make probability statements about the support for H0  or H1.

Anyone who has ever done an experiment in psychological and related sciences knows that this ideology is just that, an ‘ideology’.  In our own work we carry out experiments in virtual reality where under different conditions we examine participant responses. For example, we want to see how conditions E and C influence participants responses on a set of response variables y1,...,yp, but also where there are covariates x1,...,xk. In the (rare) simplest case p = 1 and k = 0. The conditions E and C are typically ‘Experimental’ and ‘Control’ conditions. Theory will make predictions about how the response variables may be influenced by the experimental conditions. However, it will rarely will tell us the expected influence of the covariates (e.g., things like age, gender, education, etc).

Let’s start with the simplest case - a between groups experimental design with two conditions E and C, and one response variable, and no covariates. The null and alternate hypotheses can be as above. So this is very simple, and we would register this experimental design and the main test would be a t-test for the difference between the two sample means.

The experiment is carried out and the data is collected. Now the problems start. The t-test relies on a various assumptions such as normality of the response variables (under conditions E and C), equal variances and also independent observations. The latter can usually be assured by experimental protocols. You do the analysis and check the normality of the response variables, and they are not normally distributed under E or C and the variances are nowhere near each other. The t-test is inappropriate. A bar chart showing sample means and standard errors indeed shows that the mean response under E is greater than under C, but one of the standard errors is worryingly large. You plot the histograms and find that the response variable is bimodal under both conditions, so even the means are not useful to understand the data. You realise eventually that the bimodality is caused by sex of the participants, and therefore nothing now makes sense without including this. Moreover, the design happened to be balanced for sex.  If you take sex into account the variation in these data is very well explained by the statistical model, if you don’t then nothing is ‘significant’. So now your experimental design is 2x2 with (C,E) and Sex as the two factors. Here everything makes sense, we find the mean(Condition-E-Male) > mean(Condition-C-Male) and mean(Condition-E-Female) > mean(Condition-C-Female), both by far. Everything that needs to be compatible with normality is so. The earlier result is explained because there is overlap between the results of Male and Female.

So what do we do now? Do we register a new experiment and do it all over again? In practice this is often not possible: the PhD student has finished, the grant money has run out, circumstances have changed so that the previous equipment or software is just no longer available, etc.. You kill the paper? In fact it is a kind of mysticism to suppose that what you had registered, i.e., the thoughts you had in your head before the experiment are somehow going to influence what happened. What happened is as we described, the means tell their story in spite of what you may have thought beforehand. Of course doing a post hoc analysis is not supposed to be valid in the classical statistical framework, because you have already ‘looked at’ the data. So are the results to be discounted? The argument of ‘registration’ it is that you should register another experimental design.

So you register another design where now sex is explicitly an experimental factor giving a 2x2 design. But to play safe you also record another set of covariates (age, education, number of hours per week playing computer games, etc). You run the experiment again, and just as before find the difference in means and all is well. Moreover, between the 4 groups of the 2x2 design there are no significant differences in any of the covariates. However, while preparing the paper you are generating graphs, and you notice an almost perfect linear relationship between your response variable and the number of hours a week playing computer games. You then include this as a covariate in an ANCOVA, and find that all other effects are wiped out, and that the explanation may be to do with game playing and not gender and not even E and C. In fact in more depth you find that E and C do have the predicted differential effect but only within the group of females. You find also that game playing is linked to age, and also education, so that there is a complex relationship between all these variables that is impossible to describe with ANOVA or ANCOVA single equation type models and what is needed is a path analysis. You set up a path analysis according to what you suspect and it is an exceedingly good fit to the data, and in fact a very simple model. Unfortunately, this means that you have to throw the data away, and register another experiment, and start again.

And so it goes on.

Is this ‘fishing’? It might be described that way, but in fact it is the principled way to analyse data. The Registrationists would have us believe that you set up the hypotheses, state the tests, do the experiment, run the tests, report the results. It is an ‘input-output’ model of science where thought is eliminated. It is tantamount to saying “No human was involved in the analysis of this data”. That surely eliminates bias, fishing and error, but it is wrong, and can lead to completely false conclusions as the simple example above illustrates. In the example above we would have stopped immediately after the first experiment, reported the results (“No significant difference between the means of E and C”) and that would be it - assuming it were even possible to publish a paper with P > 0.05. The more complex relationships would never have been discovered, and since P > 0.05 no other researcher might be tempted to go into this area.

Of course “fishing” is bad. You do an experiment, find ‘nothing’ and then spend months analysing and reanalysing the data in order to find somewhere a P < 0.05. This is not investigation on the lines above, it is “Let’s try this” “Let’s try that”, it is not driven by the logic of what is actually found in the data; it is not investigative but trial and error.

Now the above was a very simple example. But now suppose there are multiple response variables (p > 1) and covariates (k > 0). There are so many more things that can go wrong. Residual errors of model fits may not be normal, so you have to transform some of the response variables; a subset of the covariates might be highly correlated so that using a principle component score in their stead may lead to a more elegant and simpler model; there may be a clustering effect where response variables are better understood by combining some of them; there may be important non-linearity in the relationships that cannot be dealt with in the ANCOVA model, and so on. Each one of these potentially requires the registration of another experiment, since such circumstances could not have been foreseen and were not included in the registered experiment.

The distinction is rightly made between ‘exploratory’ and ‘confirmatory’ experiments. In practice experiments are a mixture of these - what started out as ‘confirmatory’ can quickly become ‘exploratory’ in the light of data. Perhaps only in physics there are such clear cut ‘experiments’, for example, to confirm the predictions of the theory of relatively by observing light bending during eclipses. Once we deal with data in behavioural, psychological or medical sciences things are just messy, and the attempt to try to force these into the mould of physics is damaging.

Research is not carried out in a vacuum: there are strong economic and personal pressures on researchers - not least of which for some is the issue of ‘tenure’. At the most base level tenure may depend on how many and how often P < 0.05 can be found. Faced with life-changing pressures researchers may choose to run their experiment, then ‘fish’ for some result, and then register it, reporting it only some months after the actual experiment was done.

Registration is going to solve nothing at all. It encourages an input-output type of research and analysis. Register the design, do the experiment, run the results through SPSS, publish the paper (if there is a P < 0.05). In fact too many papers follow this formula - evidenced by the fact that the results section starts with F-tests and t-tests without even presenting first in a straightforward way in tables and graphs what the data shows. This further illustrates the fact that apart from the initial idea that led to the experiment, there is no intervention of thought in the rest of the process - presenting data in a useful way and discussing it before ever presenting a ‘test’ requires thought. In this process discovery is out the window, because this occurs when you get results that are not those that were predicted by the experiment. At best discovery becomes extremely expensive - since the ‘discovery’ could only be followed through by the registration of another experiment that would specifically address this.

Overall ‘registration’ of experiments is a not only not a good idea, it is a damaging one. It encourages wrong views of the process of scientific discovery and will not even prevent what it was designed for - ‘fishing’. It will have a negative impact, reducing everything to formulaic ways of proceeding, that the unscrupulous will find a way to get round, and those interested more in discovery than accumulating P values will find frustrating and alien. 

20 November, 2015

The Presence of VR in the News

This is a response to the comment posted by Jon Angel on my blog post 'In the Presence of Freud'.

My view is that all news however presented is distortion. This ranges from the selection of WHAT to present, HOW to present it, WHO is presenting it etc.. While there may or may not be an underlying 'objectivity' - as soon as someone transforms those events into a medium that is different from the multi sensory, infinitely multi faceted events, there is distortion. This is not a negative statement but just inevitable - when you transform something from one media to another and the transformation is not 1-1, then there cannot be anything else but distortion.

What matters in terms of distortion is the political / economic stance of the people who present the news in relation to those who absorb it. I.e., WHAT is selected depends very much on the political stance of the news maker. How it is interpreted and received depends on the stance of the 'consumer'. Take any news story and see how it is presented in the Guardian and how it is presented on Fox news. Are these even reporting the same set of events?

So I do not see anything special with respect to VR on this issue. It is another medium. It has the exclusive property that it technologically puts the viewer into the action. Note that there is a fundamental change since the 'viewer' is no longer just a viewer but a participant. It is totally different to watch a masacre on tv or read about it in the newspaper than to be in it, all around you. This will give consumer-participants a different understanding of the news events. This understanding is neither superior nor inferior to any other presentation - it is just different, it has different qualities reflecting the fact that VR is a different media. 

Is showing the news on tv more biased or distorting than showing it in a newspaper? No - they are different and they have different qualities of experience associated with them.

So overall - I disagree with the point of the article that VR is something special regarding its 'distortion' capabilities. It can be used to deliver different information, which is neither more or less 'objective' than any other method. 

Finally it depends on the goal of the news presenter. If the goal is an analytical understanding of the events and putting them in some overall context then VR - by itself - might not be  a useful option. If it is to give people an experience of how it might have been to have 'been there' then probably it delivers more appropriate information than just a written story. However, even that is not certain, since people turn written stories into vivid imaginal experiences. These imaginal subjective experiences based on reading are not less 'objective' than watching it on tv or being in VR - basically we perceive what we expect to perceive.

So the only thing I would argue in the end is that VR is a different medium and can deliver quite different experiences.

10 September, 2015

In the Presence of Freud

Counselling Yourself in Immersive Virtual Reality

Sofia Adelaide Osimo, Rodrigo Pizarro, Bernhard Spanlang, Mel Slater (2015)
Conversations between self and self as Sigmund Freud—A virtual body ownership paradigm for self counselling, Scientific Reports, 5, 13899; doi: 10.1038/srep13899

Most people silently talk to themselves in order to help resolve personal problems, and for many other reasons. But when you talk to yourself who are you talking to? Suppose that the ‘other self’ that you're talking to is actually represented as another person who listens to you and replies - but the reply is actually your own reply?

We have used immersive virtual reality to provide a way that you can have a conversation with yourself, but the other ‘you’ that you talk with can be represented as another person (or even another copy of yourself). The idea was to investigate whether this method would reduce the negative feelings associated with personal problems.

In our experiment people had a virtual body that looked like themselves, that they could see when looking directly towards their own body, and also in a mirror. The body moved in synchrony and in correspondence with their own movements.

In one condition of the experiment the participants saw a representation of Dr Sigmund Freud standing at the other side of the virtual room. As they explained their personal problem to him he was gesturing to indicate that he was listening. In the next phase the person was then embodied in the Freud body that also they saw directly by looking towards themselves and in a mirror. The Freud body moved the same as they did. While in the Freud body they saw and heard their own body representation on the other side of the room explain the problem to them. Then as Freud they could offer counselling.

They then switched back to their own body, and could see and hear the Freud body delivering his counselling. The voice of the Freud body was their own voice, but with a deeper pitch so as to disguise it. They could switch back and forward between their own body and the Freud body until they felt they had a resolution of the problem.

It has been found in many other studies that when a virtual body substitutes your own and moves the same as you do then you have the perceptual illusion that it is your body - irrespective of whether the body looks like you. For example, it could be an adult embodied as a child, a body with a different skin colour to yours, or just a body that doesn't look like you but moves like you. We were interested here also in whether the effect of the counselling as Freud would be enhanced when there is the strong feeling of perceptual body ownership over the Freud body. Hence we had another experimental condition which was the same as described above, except that when you were in the Freud body it moved asynchronously with your own movements. The purpose was to reduce the illusion of perceptual body ownership over the Freud body. Our idea was that in this condition the quality of the outcome in terms of the (self-)counselling reducing the negative feelings around the personal problem would be lessened.

Top left - a stereo view from the point of view of the experimental participant, seeing a virtual body representing himself with Sigmund Freud listening to him on the other side of the room. Top right - the person wearing the virtual reality equipment. Bottom left - the person embodied in the body of Sigmund Freud talking back to a representation of himself. Bottom right - the real person being scanned to capture his body image.

In a third condition of the experiment participants did not see Freud but a copy of themselves on the other side of the room, otherwise the setup was the same as the first, with synchronous virtual body movement.  In other words in this condition they really had a conversation with themselves.

In both synchronous conditions participants had a strong illusion of body ownership - whether the body they saw was their own, or that of Freud.

In all conditions there was an improvement in the feelings about the personal problem that had been under discussion. However, we found participants experienced the greatest improvement in their feelings about their personal problem in the condition where participants interacted with the Freud body that moved the same as they did, 

Our method allows a realisation in virtual reality of ‘talking to yourself’ but with the added twist that the ‘self’ that you talk with can represent anyone else who could be good at offering advise or counselling. From the perspective of that other person you can view your problem in another light, and perhaps reach towards a solution in a better way.

This work was funded under the European Research Council Project TRAVERSE.

07 November, 2014

What happens in your brain when your virtual body is threatened?

What happens in your brain when your virtual body is threatened?

Mar González-Franco, Tabitha C Peck, Antoni Rodríguez-Fornells, Mel Slater (2014)  A Threat to a Virtual Hand Elicits Motor Cortex Activation   Experimental Brain Research 232: 3. 875-887.

Figure 1. The experimental setup. Real: the participant wears a high resolution, wide field-of-view, stereo, head-tracked head-mounted display (NVIS SX111) and EEG cap (g.tec). Virtual top: the virtual reality showing the gender-matched virtual body spatially coincident with the participant’s actual body, and in the same posture. Virtual bottom: The two experimental conditions seen by the participant when looking towards the hand from a first person perspective: HAND - virtual hand stabbed by the knife; TABLE - virtual table stabbed by the knife (control condition).

When you wear a head-tracked wide field-of-view stereo head-mounted display and you look down towards your body you can see a life-sized virtual one instead. You look around and you see a reflection of that body in a mirror. In your whole life whenever you have looked towards your body you have seen it, likewise towards a mirror. Hence the simplest perceptual hypothesis for the brain to adopt is that this is your body.

In this study we looked at what happened when the virtual body was threatened. When someone anticipates that a knife might stab their hand that is resting on a table they would be likely to attempt to move the threatened hand out of the way. They would expect to feel considerable pain should the knife actually stab the hand. In this work we considered what happens when a person’s real body is visually substituted by a life-sized virtual body, and they see a threat or attack to a hand of this virtual body seen from first person perspective. Our experiment investigated brain activity in response to events that would cause pain to the observer were these events to occur in reality. Our contribution has been to  introduce a new technique for the study of pain observation, by using immersive virtual reality (IVR) for the scenario and stimulation, while recording brain activity with EEG.

Pain observation experiments typically present a series of pictures with hands or other extremities undergoing painful situations, and they compare the brain response of the participants to the activation produced by pictures where the same extremities do not undergo painful situations [1-4]. Many of these experiments present scissors and needles perforating the extremities as painful stimuli. A potential advantage of immersive virtual reality is that there is greater ecological validity, going beyond the presentation of two-dimensional, static stimuli. There is a life-sized, three dimensional virtual body seen in stereo, that visually substitutes the obscured real body of the participant and results show that this normally induces a whole body ownership illusion [5]. Our hypothesis was that harm to the virtual hand would be associated with positive changes in P450 in line with previous studies, and that this would be enhanced with illusory body ownership. We also investigated the mu band and readiness potential (RP).

While immersed in the virtual reality the 19 participants (10 female, right-handed) repeatedly experienced during 15 minutes two conditions in a within-group design: HAND where the knife stabbed the virtual right hand, and TABLE where the knife stabbed the table 15 cm away from the right hand (Figure 1). The experiment consisted of 70 trials repeating the HAND and TABLE conditions (30 HAND and 40 TABLE).

Both EEG and electromyography (EMG) were recorded using an gUSBamp  amplifier with a resolution of 30nV; the electrodes were set to cover the motor cortex area and surrounding: FC3, FC4, C3, C4, CP3, CP4 located according to the 10/20 standard EEG recording; the reference was set with an ear clip on the left ear lobe; the ground was positioned on the forehead; electrodes in the face measured ocular activity (EOG). Three EMG electrodes were placed in the flexor carpi ulinaris muscle of the right arm to measure whether participants moved their hand. All the electrodes were kept to impedances below 10 kΩ. The data was recorded with a sampling frequency of 512 Hz. After the exposure participants answered a questionnaire on a 1-5 Likert Scale where 1 was anchored to strong disagreement and 5 to strong agreement:

Ownership: I felt as if the hand I saw in the virtual world might be my hand.
Harm Hand: I had the feeling that I might be harmed when I saw the knife inside the hand.
Harm Table: I had the feeling that I might be harmed when I saw the knife outside the hand.
No Ownership: The hand I saw was the hand of another person.
Body Threat:  I saw the knife as a threat to my body.

Figure 2. EEG Recordings. Left: Grand averaged stimulus locked ERPs for six representative front, central and parietal electrode locations. A significant increase in the amplitude of the P450 is observed in the HAND condition mainly at C3 and CP3 locations. Baseline from [-200 ms to 0 ms], time 0 indicates the stimuli onset; a low pass filter 12Hz half-amplitude cutoff was applied. Right: (a) Time Frequency Evolution of the two conditions and the difference in the spectral activity. (b) Grand averaged 1-s short time power spectra calculated from EEG data (electrode C3) recorded. The baseline corresponds to the range [-1 to 0] seconds before the stimuli and the activity period corresponds to the range [0.7 to 1.7] seconds after the stimuli. Both the Baseline and TABLE frequency spectra show a peak in the mu-rhythm that is attenuated in the HAND condition. (c) Grand averaged Mu-rhythm (9-12Hz) Event Related Desynchronization for the C3 electrode. (d) Grand averaged Readiness Potential (C3-C4) subtraction between the brain activity in the two hemispheres shows movement preparation effects. Low pass filter 8Hz, half-amplitude cutoff.

Figure 3. Box plots showing the responses to the questionnaire. The thick lines are the medians, and the boxes are the interquartile ranges (IQR). Wilcoxon matched pairs sign-rank tests show differences between Ownership and No Ownership  (P < 0.0001); Harm Hand and Harm Table (P < 0.0002); Body Threat and Harm Table (P < 0.0003). Harm Hand and Body Threat ( P < 0.018).


• The results suggest that when a person is in an immersive virtual reality and has body ownership illusion towards a virtual body that apparently substitutes their own body, there are autonomic responses that correspond to what would be observed were the events to take place in reality. Overall automatic brain mechanisms –P450– were found in this variation of the classical pain observation experiment, which is consistent with previously reported results.

• The results cannot be explained as participants experiencing empathy towards another person since they witnessed attacks to their co-located virtual body and both subjective and objective data suggest that they experienced this as an attack on their own body.

• The results support our initial hypothesis that a threat to a virtual hand, towards which the participant has an illusion of ownership, would significantly produce a harm prevention effect (the Readiness Potential (C3-C4) and oscillatory movement-related components, the mu-ERD), such as trying to move it away from the source of the harm. The questionnaire also confirmed high levels of ownership over the virtual body.

• The correlation between the automatic brain mechanisms –P450– and the subjective illusion of ownership suggests  a potentially new measure of virtual embodiment.

1. Avenanti, A., et al.,. NeuroImage, 2006. 32(1);  
2. Bufalari, I., et al., Cerebral Cortex, 2007. 17(11); 
3. Fan, Y. and S. Han, Neuropsychologia, 2008. 46(1);  
4. Li, W. and S. Han, Neuroscience Letters, 2010. 469(3);
5. Slater, M., et al., PLoS ONE, 2010. 5(5).

Funded by European Union FP7 IntegratedProject VERE (#257695);  FI-DGR predoctorate grant from the Catalan Government co-funded by the European Social Found (EC-ESF); Spain MICIN (PSI2011-29219);  ERC project TRAVERSE (#227985).