In the Presence of Jaron Lanier

In the Presence of Jaron Lanier

The talk by Jaron Lanier Friday 11th November 2017 at UCL was attended by about 150 people. Jaron talked about themes from his new book Dawn of the New Everything: A Journey Through Virtual Reality. It is best to read the book of course, but here I will simply say that he introduced the Thesis - that virtual reality offers a fantastic way for the construction of new shared realities, where we can experience each others' dreams. He also introduced the Antithesis that VR offers a perfect means for behavioural modification - since it can act as a 'Skinner box' where a person's environment is completely controlled, and based on feedback in response to their actions and choices algorithms can move them surreptitiously towards new states of mind. The Synthesis ....

Introducing Jaron Lanier

Start of the talk

The talk was very inspiring and, judging by the questions, and the comments afterwards, was very much appreciated and enjoyed.

Here I want to say that it provided the opportunity for previous members of my group, now friends, to get together even if briefly, and renew old ties, and perhaps plan a future gathering. In the pictures below all but Zillah Watson, Aiden O'Neill (and Jaron Lanier) were in the Virtual Environments and Computer Graphics group at UCL at some time. I have indicated those that were postdocs, and PhD students by the year that they obtained their PhD.

Left to right: Daniela Romano (ex postdoc), Vinoba Vinayagamoorthy (2006), Joel Jordan (2010), Marco Gillies (ex postdoc)

Vinoba with Joel

Lee Pip Bull (2006) with Vinoba

Left to right: Zillah Watson, Anthony Steed (1996),
Aiden O'Neill (Penguin Books), Jaron Lanier, 
Mel Slater, Sylvia Xueni Pan (2010 and postdoc)

Ghost Presence

While attending WVRF17 I was lucky to experience two immersive VR movie experiences. The first was Through You by Saschka Unseld and Lily Baldwin and the second Defrost: The Virtual Series by Randal Kleiser and Tanna Frederick. They are both 360 degree stereo movies, rendered in Samsung Gear VR. They are both beautifully made and very exciting advances on VR cinematography.

However, here I want to concentrate on the form rather than the content since they are quite different in the kind of presence illusions they evoke. In Through You, you are a witness to a kind of dance that tells the story of a romance over time: what was, and what might have been. In the set up I experienced you physically stand near a wall covered with what felt like a velvet curtain, and were advised to be facing and touching that wall with both hands in order to maintain balance and orientation. Throughout the virtual performance I found that I did keep at least one hand touching the wall, which acted as a kind of anchor. However, the events were taking place anywhere in the scene, so I was often touching the wall even while turned around 180 degrees from my starting position.

In my theory of presence, the basis of the perceptual illusion of ‘being there’ is that the system supports natural sensorimotor contingencies for visual perception. So the extent to which I can use my body and perceive the virtual world in a normal way (head turning, bending down, looking around, reaching out and touching, and seeing wide field of view, with stereo vision, high resolution etc – for vision) the simplest hypothesis for the brain do adopt is “This is where I am”.

In both movies there were head based visual sensorimotor contingencies. The Gear VR does not support full 6 df head-tracking but only head orientation. But since in both movies the viewer is only required to look around while remaining in the same place, this is not very noticeable. So we can say that there were reasonable head-based visual sensorimotor contingencies.

In Through You it was a shock to look down towards myself and realise that I had no body. Our first ever paper in the field of VR (VRAIS, 1993) concentrated on the importance of having a self representation in the virtual environment, a finding that has been supported by more recent results. So even though as you look around, you see the world in 3D stereo, with reasonable field of view, latency and resolution, so that you get approximate natural sensorimotor contingencies, there is still something critically missing in the scene – You. I personally felt a kind of ghostly ‘place illusion’, I was there and not there, some kind of in-between state.

In Through You you are an observer, a witness. The environment does not respond to you in any way. The actors are completely unaware of your presence. One of my proposed bases of the ‘Plausibility Illusion’ is that the virtual world responds to you, i.e., that you can influence what happens, and another is that there are events that relate personally to you. Plausibility refers to the illusion that ‘This is really happening  - now’ (even though you know that this is not the case). This illusion that events are happening is meant in a very personal sense – that the events can impinge on you – that someone may be looking at you, that they can get into your space, that the events may be dangerous or beneficial to you personally. I guess it is something to do with our basic survival instinct, that we continually evaluate surrounding events for possible threat, and that the Plausibility Illusion is concerned with this feeling. Without this personal reality in relation to the ongoing events, you are an observer – very much like you are the observer of a film or stage play. Of course the events can influence your emotions, your appreciation of the aesthetics, you may relate the events to your own life – but the events themselves are not physically connected to you. We cannot call this Ghost Plausibility, since in this type of situation there is no Plausibility at all in the strict sense of the meaning of ‘Plausibility Illusion’ (Psi).

Nevertheless ‘Ghost Place Illusion’ (sensorimotor contingencies but no self representation) and the absence of Psi do give rise to a strange feeling – I will call it Ghost Presence. You are like a Ghost, being in and simultaneously not being in a scenario that unfolds completely independently of your will. From that perspective you can attain a special perspective on the events that you witness. It is somewhat like watching a conventional movie, but your Ghost Presence gives you a familiarity with the space and a closeness to the characters that you cannot get from a movie. It is a different experience, a different art form. This is also superbly illustrated by the prize winning After Solitary by Emblematic. Here you are placed in a virtual replication of a tiny prison cell in which the (real) protagonist spent five years in solitary confinement. You have a first hand experience of the cell, its size, its squalor. From that you can begin to imagine the horror of living 5 years alone (or even 5 hours) in such conditions. The prisoner is there describing what happened to him and his feelings. However, the prisoner is not addressing you personally, he doesn’t look at you or invoke your presence in any way. He is speaking to a general audience. His story is powerful, he is telling it to anyone and everyone but not you specifically. You have the Ghost Presence, where especially the experience of the space is extremely powerful and unforgettable. The monologue of the prisoner provides additional information that also has a powerful emotional impact.

In Defrost you awake after being frozen in liquid nitrogen for 30 years after suffering a stroke (that was incurable at that time). Now the stroke has been cured. A doctor is telling you all this, and adds that due to the fact of your 30 years of sleep you are unable to move, except for your head. When you look down towards yourself you see you are embodied and in a wheel chair. This is 360 degrees and stereo video playback. Since you can look around but in a fixed position the tracking of the Samsung Gear is sufficient to provide reasonable visual sensorimotor contingencies, and there is corresponding Place Illusion. Since you have a body it is not a ghostly PI. The explanation of being unable to move because of the long sleep helps with respect to Plausibility, and a whole series of characters come to talk with you personally including the doctors and (now grown up) children. Over several episodes a drama unfolds in which you play the major role. As well as being a brilliant idea it is very well designed and acted, and a great example of a new kind of first person narrative experience that is going to emerge from VR.

It is important to note that I am not comparing Defrost and Through You from the point of view of their creativity and execution or as experiences, but rather the illusions that they evoke (in me). Through You results in Ghost Presence, which is a very unique experience, being simultaneously there and not there, in the scene and not in it. The contradictions give rise to a kind of tension which is itself a new experience – one that we cannot get at all from any other media. As I said, it is a new kind of art form. Defrost shows how narratives that exploits the PI and Psi illusions  might develop – and especially the conjunction of PI and Psi in the illusion of body ownership.  Through You and Defrost offer different qualities of experience, each created with very high production values, it is not that one is ‘better’ than the other in any way.

When we have used questionnaires to assess presence (e.g., in) we have invariably included the question:

“When you think back about your experience, do you think of the [environment X] more as images that you saw, or more as somewhere that you visited?”

Now when I spend time in a computer graphics model-based VR, my answer to that question has typically a very high score (e.g., 6 or 7 on a 1-7 scale where 1 means ‘images that you saw’ and 7 means ‘somewhere that you visited’). I think back to some environments I have experienced recently, and it is the case I have the feeling to have been somewhere rather than just seen something.

However, when I think back to my experience of Through You and Defrost, my score is pretty low for both, 2/7 – I remember the images, rather than of being somewhere. For Through You this is understandable, because of the Ghost Presence illusion. For Defrost since there were pretty good visual sensorimotor contingencies, I should have the memory of having been somewhere not just seen images. So why is this?

I think this may be to do with being in a virtual world created by video rather than graphics. We are really used to seeing video, it is everywhere throughout our lives. We have never been in world that looks like video. We know that the characters in video have zero information about us. We know that video is something that ‘just plays’ (most of the time) irrespective of anything we do (other than fast forward/back or switch off). In other words video comes with a whole set of very powerful and ingrained expectations. Model based graphics VR does not – we see an environment and virtual characters, that look ‘sort of’ like the real thing, but are not. We don’t know what to expect. Typically there is strong interaction in these worlds whereas there is very minimal interaction with video. So it is possible that the very powerful expectations and associations with video act as a top down dampening of the illusions normally associated with VR. So although during an experience I might have some feeling of ‘being there’ and that the events are ‘really happening’, as soon as I’ve finished, what I remember is the video. I haven’t ‘been there’ but I’ve ‘seen it’.

This is not an Uncanny Valley effect, where because the video is almost but not quite perfect, we reject it. It is an expectations and associations based effect, where cognition to some extent is overriding perception – since we know in a very deep and experiential way what video is, and therefore if this looks like video (a) I can’t be here and (b) this isn’t really happening right now.

The fallacy of the large sample size argument

Another bureaucratic solution to problems of science is being increasingly heard. This is the idea that papers that report experiments with small sample sizes should be "desk rejected". This is a relatively new phenomenon. Recently I have had two referees on two different papers mention this point. I have seen tweets suggesting that anything less than n = 100 won't do. This is supposed to contribute to the solution of the 'crisis of reproducibility' - especially in psychology and related disciplines.

Such proposals for bureaucratic solutions with fixed norms do not take account of elementary statistics. The sample size needed for estimation or hypothesis testing is relative to the variance of the variables involved. 

Take the simplest example - where a normally distributed random variable X has unknown mean m and known variance s². The experiment will deliver a sample of n independent observations on X. The goal is to estimate m, with a 95% confidence interval, u to v. Critically we want to choose a sample size n such that v-u< e.

The 95% confidence interval is  xbar ± 1.96*s/sqrt(n) where xbar is the sample mean.

Therefore we require:

3.92s/sqrt(n) < e, leading to n > (3.92s/e)^2. 

The sample size needed is proportional to the variance. For example, suppose that s = e. Then a sample size of 16 is good enough.  Suppose that s = 0.80*e, then a sample size of around 10 would be good enough. A sample size of 100 would be needed if s would be 2.55 times e. It is clearly the variance in relation to the required error size that is important.

This really matters because running experiments is typically very expensive. If a sample size of 10 would do why use 100? It would only be to satisfy a slogan, not for any real contribution to more reliable statistical inference.

The supposed failure of reproducibility is built into the system and will not be overcome by adopting slogans. In classical statistical inference with 5% significance levels, according to the theory the null hypothesis will be wrongly rejected approximately 5% of the time anyway. No researcher can know if their experiment is one of the 5%! This is the point of repeat studies. Yet repeat studies are hard to publish (no novelty) and if indeed they do find results at odds with the original study then the researchers on the original study may have their integrity questioned. Yet who can know whether the second study is one of the 20% that will falsely not reject the null hypothesis (assuming a power of 80%)!

The Bayesian approach does not have these problems. An experiment will result in probabilities about hypotheses or probability distributions over parameters rather than fixed answers or conclusions. As more data are collected through repetition studies so these probabilities will be updated. Also it should be noted that a similar argument used in the example above will hold for finding a Bayesian credible interval. 

To paraphrase a well known saying: "It's the variance, stupid!" 

Pre-Registration of Analysis of Experiments is Dangerous for Science

The idea of pre-registration of experimental analysis is supposed to be one amongst a set of solutions to the crisis in the explosion of false results making it into the scientific literature - especially psychology and related sciences. It is argued that pre-registration of analyses will curb the over-analyses of data that is carried out in a search for ‘significant’ results.

Here I argue that such a move will not only not solve any crisis but that it will have detrimental effects on science. The idea arises as a false notion of how scientific research operates. In the accepted ideology the scientist has a ‘hypothesis’ and then formulates this as a ‘null hypothesis’ (H0) and an ‘alternative hypothesis’ (H1) (of course there could be sets of these for any particular experiment). H0 is set up to mean that the original scientific hypothesis is ‘false’. Specifically H0 is chosen to satisfy simple mathematical properties that will make the resulting statistical analysis quite easy to do, following well-known text book formulae. For example, the scientist may believe that experimental condition E will produce a higher value on some critical variable y than experimental condition C (other things being equal). In such a case H0 would typically be that μC = μE , and H1 that  μE > μC. Under the null hypothesis the distribution of the test statistic t for the difference of two means is known, but critically under certain statistical assumptions. Depending on the results of the computed test statistic t after the experiment, H0 is rejected (if t falls in some critical region) or not. H0 not being rejected is not evidence for it being 'true'. Classical statistics does not even permit the researcher to give odds or make probability statements about the support for H0  or H1.

Anyone who has ever done an experiment in psychological and related sciences knows that this ideology is just that, an ‘ideology’.  In our own work we carry out experiments in virtual reality where under different conditions we examine participant responses. For example, we want to see how conditions E and C influence participants responses on a set of response variables y1,...,yp, but also where there are covariates x1,...,xk. In the (rare) simplest case p = 1 and k = 0. The conditions E and C are typically ‘Experimental’ and ‘Control’ conditions. Theory will make predictions about how the response variables may be influenced by the experimental conditions. However, it will rarely will tell us the expected influence of the covariates (e.g., things like age, gender, education, etc).

Let’s start with the simplest case - a between groups experimental design with two conditions E and C, and one response variable, and no covariates. The null and alternate hypotheses can be as above. So this is very simple, and we would register this experimental design and the main test would be a t-test for the difference between the two sample means.

The experiment is carried out and the data is collected. Now the problems start. The t-test relies on a various assumptions such as normality of the response variables (under conditions E and C), equal variances and also independent observations. The latter can usually be assured by experimental protocols. You do the analysis and check the normality of the response variables, and they are not normally distributed under E or C and the variances are nowhere near each other. The t-test is inappropriate. A bar chart showing sample means and standard errors indeed shows that the mean response under E is greater than under C, but one of the standard errors is worryingly large. You plot the histograms and find that the response variable is bimodal under both conditions, so even the means are not useful to understand the data. You realise eventually that the bimodality is caused by sex of the participants, and therefore nothing now makes sense without including this. Moreover, the design happened to be balanced for sex.  If you take sex into account the variation in these data is very well explained by the statistical model, if you don’t then nothing is ‘significant’. So now your experimental design is 2x2 with (C,E) and Sex as the two factors. Here everything makes sense, we find the mean(Condition-E-Male) > mean(Condition-C-Male) and mean(Condition-E-Female) > mean(Condition-C-Female), both by far. Everything that needs to be compatible with normality is so. The earlier result is explained because there is overlap between the results of Male and Female.

So what do we do now? Do we register a new experiment and do it all over again? In practice this is often not possible: the PhD student has finished, the grant money has run out, circumstances have changed so that the previous equipment or software is just no longer available, etc.. You kill the paper? In fact it is a kind of mysticism to suppose that what you had registered, i.e., the thoughts you had in your head before the experiment are somehow going to influence what happened. What happened is as we described, the means tell their story in spite of what you may have thought beforehand. Of course doing a post hoc analysis is not supposed to be valid in the classical statistical framework, because you have already ‘looked at’ the data. So are the results to be discounted? The argument of ‘registration’ it is that you should register another experimental design.

So you register another design where now sex is explicitly an experimental factor giving a 2x2 design. But to play safe you also record another set of covariates (age, education, number of hours per week playing computer games, etc). You run the experiment again, and just as before find the difference in means and all is well. Moreover, between the 4 groups of the 2x2 design there are no significant differences in any of the covariates. However, while preparing the paper you are generating graphs, and you notice an almost perfect linear relationship between your response variable and the number of hours a week playing computer games. You then include this as a covariate in an ANCOVA, and find that all other effects are wiped out, and that the explanation may be to do with game playing and not gender and not even E and C. In fact in more depth you find that E and C do have the predicted differential effect but only within the group of females. You find also that game playing is linked to age, and also education, so that there is a complex relationship between all these variables that is impossible to describe with ANOVA or ANCOVA single equation type models and what is needed is a path analysis. You set up a path analysis according to what you suspect and it is an exceedingly good fit to the data, and in fact a very simple model. Unfortunately, this means that you have to throw the data away, and register another experiment, and start again.

And so it goes on.

Is this ‘fishing’? It might be described that way, but in fact it is the principled way to analyse data. The Registrationists would have us believe that you set up the hypotheses, state the tests, do the experiment, run the tests, report the results. It is an ‘input-output’ model of science where thought is eliminated. It is tantamount to saying “No human was involved in the analysis of this data”. That surely eliminates bias, fishing and error, but it is wrong, and can lead to completely false conclusions as the simple example above illustrates. In the example above we would have stopped immediately after the first experiment, reported the results (“No significant difference between the means of E and C”) and that would be it - assuming it were even possible to publish a paper with P > 0.05. The more complex relationships would never have been discovered, and since P > 0.05 no other researcher might be tempted to go into this area.

Of course “fishing” is bad. You do an experiment, find ‘nothing’ and then spend months analysing and reanalysing the data in order to find somewhere a P < 0.05. This is not investigation on the lines above, it is “Let’s try this” “Let’s try that”, it is not driven by the logic of what is actually found in the data; it is not investigative but trial and error.

Now the above was a very simple example. But now suppose there are multiple response variables (p > 1) and covariates (k > 0). There are so many more things that can go wrong. Residual errors of model fits may not be normal, so you have to transform some of the response variables; a subset of the covariates might be highly correlated so that using a principle component score in their stead may lead to a more elegant and simpler model; there may be a clustering effect where response variables are better understood by combining some of them; there may be important non-linearity in the relationships that cannot be dealt with in the ANCOVA model, and so on. Each one of these potentially requires the registration of another experiment, since such circumstances could not have been foreseen and were not included in the registered experiment.

The distinction is rightly made between ‘exploratory’ and ‘confirmatory’ experiments. In practice experiments are a mixture of these - what started out as ‘confirmatory’ can quickly become ‘exploratory’ in the light of data. Perhaps only in physics there are such clear cut ‘experiments’, for example, to confirm the predictions of the theory of relatively by observing light bending during eclipses. Once we deal with data in behavioural, psychological or medical sciences things are just messy, and the attempt to try to force these into the mould of physics is damaging.

Research is not carried out in a vacuum: there are strong economic and personal pressures on researchers - not least of which for some is the issue of ‘tenure’. At the most base level tenure may depend on how many and how often P < 0.05 can be found. Faced with life-changing pressures researchers may choose to run their experiment, then ‘fish’ for some result, and then register it, reporting it only some months after the actual experiment was done.

Registration is going to solve nothing at all. It encourages an input-output type of research and analysis. Register the design, do the experiment, run the results through SPSS, publish the paper (if there is a P < 0.05). In fact too many papers follow this formula - evidenced by the fact that the results section starts with F-tests and t-tests without even presenting first in a straightforward way in tables and graphs what the data shows. This further illustrates the fact that apart from the initial idea that led to the experiment, there is no intervention of thought in the rest of the process - presenting data in a useful way and discussing it before ever presenting a ‘test’ requires thought. In this process discovery is out the window, because this occurs when you get results that are not those that were predicted by the experiment. At best discovery becomes extremely expensive - since the ‘discovery’ could only be followed through by the registration of another experiment that would specifically address this.

Overall ‘registration’ of experiments is a not only not a good idea, it is a damaging one. It encourages wrong views of the process of scientific discovery and will not even prevent what it was designed for - ‘fishing’. It will have a negative impact, reducing everything to formulaic ways of proceeding, that the unscrupulous will find a way to get round, and those interested more in discovery than accumulating P values will find frustrating and alien. 

The Presence of VR in the News

This is a response to the comment posted by Jon Angel on my blog post 'In the Presence of Freud'.

My view is that all news however presented is distortion. This ranges from the selection of WHAT to present, HOW to present it, WHO is presenting it etc.. While there may or may not be an underlying 'objectivity' - as soon as someone transforms those events into a medium that is different from the multi sensory, infinitely multi faceted events, there is distortion. This is not a negative statement but just inevitable - when you transform something from one media to another and the transformation is not 1-1, then there cannot be anything else but distortion.

What matters in terms of distortion is the political / economic stance of the people who present the news in relation to those who absorb it. I.e., WHAT is selected depends very much on the political stance of the news maker. How it is interpreted and received depends on the stance of the 'consumer'. Take any news story and see how it is presented in the Guardian and how it is presented on Fox news. Are these even reporting the same set of events?

So I do not see anything special with respect to VR on this issue. It is another medium. It has the exclusive property that it technologically puts the viewer into the action. Note that there is a fundamental change since the 'viewer' is no longer just a viewer but a participant. It is totally different to watch a masacre on tv or read about it in the newspaper than to be in it, all around you. This will give consumer-participants a different understanding of the news events. This understanding is neither superior nor inferior to any other presentation - it is just different, it has different qualities reflecting the fact that VR is a different media. 

Is showing the news on tv more biased or distorting than showing it in a newspaper? No - they are different and they have different qualities of experience associated with them.

So overall - I disagree with the point of the article that VR is something special regarding its 'distortion' capabilities. It can be used to deliver different information, which is neither more or less 'objective' than any other method. 

Finally it depends on the goal of the news presenter. If the goal is an analytical understanding of the events and putting them in some overall context then VR - by itself - might not be  a useful option. If it is to give people an experience of how it might have been to have 'been there' then probably it delivers more appropriate information than just a written story. However, even that is not certain, since people turn written stories into vivid imaginal experiences. These imaginal subjective experiences based on reading are not less 'objective' than watching it on tv or being in VR - basically we perceive what we expect to perceive.

So the only thing I would argue in the end is that VR is a different medium and can deliver quite different experiences.

In the Presence of Freud

Counselling Yourself in Immersive Virtual Reality

Sofia Adelaide Osimo, Rodrigo Pizarro, Bernhard Spanlang, Mel Slater (2015)
Conversations between self and self as Sigmund Freud—A virtual body ownership paradigm for self counselling, Scientific Reports, 5, 13899; doi: 10.1038/srep13899

Most people silently talk to themselves in order to help resolve personal problems, and for many other reasons. But when you talk to yourself who are you talking to? Suppose that the ‘other self’ that you're talking to is actually represented as another person who listens to you and replies - but the reply is actually your own reply?


We have used immersive virtual reality to provide a way that you can have a conversation with yourself, but the other ‘you’ that you talk with can be represented as another person (or even another copy of yourself). The idea was to investigate whether this method would reduce the negative feelings associated with personal problems.

In our experiment people had a virtual body that looked like themselves, that they could see when looking directly towards their own body, and also in a mirror. The body moved in synchrony and in correspondence with their own movements.

In one condition of the experiment the participants saw a representation of Dr Sigmund Freud standing at the other side of the virtual room. As they explained their personal problem to him he was gesturing to indicate that he was listening. In the next phase the person was then embodied in the Freud body that also they saw directly by looking towards themselves and in a mirror. The Freud body moved the same as they did. While in the Freud body they saw and heard their own body representation on the other side of the room explain the problem to them. Then as Freud they could offer counselling.

They then switched back to their own body, and could see and hear the Freud body delivering his counselling. The voice of the Freud body was their own voice, but with a deeper pitch so as to disguise it. They could switch back and forward between their own body and the Freud body until they felt they had a resolution of the problem.

It has been found in many other studies that when a virtual body substitutes your own and moves the same as you do then you have the perceptual illusion that it is your body - irrespective of whether the body looks like you. For example, it could be an adult embodied as a child, a body with a different skin colour to yours, or just a body that doesn't look like you but moves like you. We were interested here also in whether the effect of the counselling as Freud would be enhanced when there is the strong feeling of perceptual body ownership over the Freud body. Hence we had another experimental condition which was the same as described above, except that when you were in the Freud body it moved asynchronously with your own movements. The purpose was to reduce the illusion of perceptual body ownership over the Freud body. Our idea was that in this condition the quality of the outcome in terms of the (self-)counselling reducing the negative feelings around the personal problem would be lessened.

Top left - a stereo view from the point of view of the experimental participant, seeing a virtual body representing himself with Sigmund Freud listening to him on the other side of the room. Top right - the person wearing the virtual reality equipment. Bottom left - the person embodied in the body of Sigmund Freud talking back to a representation of himself. Bottom right - the real person being scanned to capture his body image.

In a third condition of the experiment participants did not see Freud but a copy of themselves on the other side of the room, otherwise the setup was the same as the first, with synchronous virtual body movement.  In other words in this condition they really had a conversation with themselves.

In both synchronous conditions participants had a strong illusion of body ownership - whether the body they saw was their own, or that of Freud.

In all conditions there was an improvement in the feelings about the personal problem that had been under discussion. However, we found participants experienced the greatest improvement in their feelings about their personal problem in the condition where participants interacted with the Freud body that moved the same as they did, 

Our method allows a realisation in virtual reality of ‘talking to yourself’ but with the added twist that the ‘self’ that you talk with can represent anyone else who could be good at offering advise or counselling. From the perspective of that other person you can view your problem in another light, and perhaps reach towards a solution in a better way.

This work was funded under the European Research Council Project TRAVERSE.

What happens in your brain when your virtual body is threatened?

What happens in your brain when your virtual body is threatened?


Mar González-Franco, Tabitha C Peck, Antoni Rodríguez-Fornells, Mel Slater (2014)  A Threat to a Virtual Hand Elicits Motor Cortex Activation   Experimental Brain Research 232: 3. 875-887.

Figure 1. The experimental setup. Real: the participant wears a high resolution, wide field-of-view, stereo, head-tracked head-mounted display (NVIS SX111) and EEG cap (g.tec). Virtual top: the virtual reality showing the gender-matched virtual body spatially coincident with the participant’s actual body, and in the same posture. Virtual bottom: The two experimental conditions seen by the participant when looking towards the hand from a first person perspective: HAND - virtual hand stabbed by the knife; TABLE - virtual table stabbed by the knife (control condition).

When you wear a head-tracked wide field-of-view stereo head-mounted display and you look down towards your body you can see a life-sized virtual one instead. You look around and you see a reflection of that body in a mirror. In your whole life whenever you have looked towards your body you have seen it, likewise towards a mirror. Hence the simplest perceptual hypothesis for the brain to adopt is that this is your body.

In this study we looked at what happened when the virtual body was threatened. When someone anticipates that a knife might stab their hand that is resting on a table they would be likely to attempt to move the threatened hand out of the way. They would expect to feel considerable pain should the knife actually stab the hand. In this work we considered what happens when a person’s real body is visually substituted by a life-sized virtual body, and they see a threat or attack to a hand of this virtual body seen from first person perspective. Our experiment investigated brain activity in response to events that would cause pain to the observer were these events to occur in reality. Our contribution has been to  introduce a new technique for the study of pain observation, by using immersive virtual reality (IVR) for the scenario and stimulation, while recording brain activity with EEG.

Pain observation experiments typically present a series of pictures with hands or other extremities undergoing painful situations, and they compare the brain response of the participants to the activation produced by pictures where the same extremities do not undergo painful situations [1-4]. Many of these experiments present scissors and needles perforating the extremities as painful stimuli. A potential advantage of immersive virtual reality is that there is greater ecological validity, going beyond the presentation of two-dimensional, static stimuli. There is a life-sized, three dimensional virtual body seen in stereo, that visually substitutes the obscured real body of the participant and results show that this normally induces a whole body ownership illusion [5]. Our hypothesis was that harm to the virtual hand would be associated with positive changes in P450 in line with previous studies, and that this would be enhanced with illusory body ownership. We also investigated the mu band and readiness potential (RP).

While immersed in the virtual reality the 19 participants (10 female, right-handed) repeatedly experienced during 15 minutes two conditions in a within-group design: HAND where the knife stabbed the virtual right hand, and TABLE where the knife stabbed the table 15 cm away from the right hand (Figure 1). The experiment consisted of 70 trials repeating the HAND and TABLE conditions (30 HAND and 40 TABLE).

Both EEG and electromyography (EMG) were recorded using an gUSBamp  amplifier with a resolution of 30nV; the electrodes were set to cover the motor cortex area and surrounding: FC3, FC4, C3, C4, CP3, CP4 located according to the 10/20 standard EEG recording; the reference was set with an ear clip on the left ear lobe; the ground was positioned on the forehead; electrodes in the face measured ocular activity (EOG). Three EMG electrodes were placed in the flexor carpi ulinaris muscle of the right arm to measure whether participants moved their hand. All the electrodes were kept to impedances below 10 kΩ. The data was recorded with a sampling frequency of 512 Hz. After the exposure participants answered a questionnaire on a 1-5 Likert Scale where 1 was anchored to strong disagreement and 5 to strong agreement:

Ownership: I felt as if the hand I saw in the virtual world might be my hand.

Harm Hand: I had the feeling that I might be harmed when I saw the knife inside the hand.

Harm Table: I had the feeling that I might be harmed when I saw the knife outside the hand.

No Ownership: The hand I saw was the hand of another person.

Body Threat:  I saw the knife as a threat to my body.

Figure 2. EEG Recordings. Left: Grand averaged stimulus locked ERPs for six representative front, central and parietal electrode locations. A significant increase in the amplitude of the P450 is observed in the HAND condition mainly at C3 and CP3 locations. Baseline from [-200 ms to 0 ms], time 0 indicates the stimuli onset; a low pass filter 12Hz half-amplitude cutoff was applied. Right: (a) Time Frequency Evolution of the two conditions and the difference in the spectral activity. (b) Grand averaged 1-s short time power spectra calculated from EEG data (electrode C3) recorded. The baseline corresponds to the range [-1 to 0] seconds before the stimuli and the activity period corresponds to the range [0.7 to 1.7] seconds after the stimuli. Both the Baseline and TABLE frequency spectra show a peak in the mu-rhythm that is attenuated in the HAND condition. (c) Grand averaged Mu-rhythm (9-12Hz) Event Related Desynchronization for the C3 electrode. (d) Grand averaged Readiness Potential (C3-C4) subtraction between the brain activity in the two hemispheres shows movement preparation effects. Low pass filter 8Hz, half-amplitude cutoff.

Figure 3. Box plots showing the responses to the questionnaire. The thick lines are the medians, and the boxes are the interquartile ranges (IQR). Wilcoxon matched pairs sign-rank tests show differences between Ownership and No Ownership  (P < 0.0001); Harm Hand and Harm Table (P < 0.0002); Body Threat and Harm Table (P < 0.0003). Harm Hand and Body Threat ( P < 0.018).

Conclusions

• The results suggest that when a person is in an immersive virtual reality and has body ownership illusion towards a virtual body that apparently substitutes their own body, there are autonomic responses that correspond to what would be observed were the events to take place in reality. Overall automatic brain mechanisms –P450– were found in this variation of the classical pain observation experiment, which is consistent with previously reported results.

• The results cannot be explained as participants experiencing empathy towards another person since they witnessed attacks to their co-located virtual body and both subjective and objective data suggest that they experienced this as an attack on their own body.

• The results support our initial hypothesis that a threat to a virtual hand, towards which the participant has an illusion of ownership, would significantly produce a harm prevention effect (the Readiness Potential (C3-C4) and oscillatory movement-related components, the mu-ERD), such as trying to move it away from the source of the harm. The questionnaire also confirmed high levels of ownership over the virtual body.

• The correlation between the automatic brain mechanisms –P450– and the subjective illusion of ownership suggests  a potentially new measure of virtual embodiment.

1. Avenanti, A., et al.,. NeuroImage, 2006. 32(1);  

2. Bufalari, I., et al., Cerebral Cortex, 2007. 17(11); 

3. Fan, Y. and S. Han, Neuropsychologia, 2008. 46(1);  

4. Li, W. and S. Han, Neuroscience Letters, 2010. 469(3);

5. Slater, M., et al., PLoS ONE, 2010. 5(5).

Funded by European Union FP7 IntegratedProject VERE (#257695);  FI-DGR predoctorate grant from the Catalan Government co-funded by the European Social Found (EC-ESF); Spain MICIN (PSI2011-29219);  ERC project TRAVERSE (#227985).  

Video:  https://www.youtube.com/watch?v=029XNWctb4A 

Visual-Tactile and Visual-Motor Influences on Virtual Body Ownership

The vast amount of research on the rubber hand illusion uses visuotactile synchronous stimulation to induce the illusion. This means that sight of the rubber hand being touched is synchronous temporally and spatially with the tactile stimulation felt on the corresponding (hidden) real hand. It has also been shown that the illusion can be induced with visuomotor stimulation - meaning that (in this case) the virtual hand moves synchronously with the movements of the corresponding (hidden) real hand.

For most of our work on virtual full body ownership we have relied on visuomotor effects, where the virtual body moves synchronously with the real body. This is accomplished through real-time motion capture, so that participant movements are mapped to the corresponding movements of the virtual body. The virtual body is seen directly by looking towards it through the head-mounted display, and also in a virtual mirror.

Which of these two methods of stimulation is the most powerful in inducing the body ownership illusion - visuotactile or visuomotor? In a recent paper  (PDF) we describe an experiment that addresses this question.

Participants were in a reclined position and saw their full virtual body from a first person perspective through a head-tracked, wide field-of-view head-mounted display. As they moved their leg (A) the corresponding virtual leg would move synchronously or asynchronously (B). When the experimenter tapped a leg with the wand (C) the participant would see a virtual ball tapping the corresponding position on the virtual leg (synchronously or asynchronously). Hence we had a 2 by 2 experimental design (synchronous movement or asynchronous combined with synchronous tapping or asynchronous) where these were delivered alternately.  There were 60 participants in a between groups design - hence each group of 15 experienced just one of the 4 combinations of these two factors.

The experiment was organised so that every subject first experienced for a while the best possible setup - that is visuomotor and visuotactile synchrony. Then after some questions had been answered they experienced one of the 4 conditions. Then questions about body ownership and agency were again answered.

Based on the questionnaire responses visuomotor synchrony outweighed visuotactile in producing the illusion.

However, unusually, we also attempted to measure not only what generated the illusion but also what extinguished it. Here we used a method first proposed in the study of presence in virtual reality (the sensation of being in the place depicted by the virtual environment displays). This method is called ‘breaks in presence’. The assumption is that the normal state is for the illusion of presence to occur, but occasionally it breaks for various reasons (errors in rendering or tracking, physical entanglement with cables, bumping into a Cave screen, or just spontaneous switches in attention or perception). From an indication by participants about when each break occurs it is possible to estimate an overall probability of presence (the proportion of time the participant had this illusion). Here we adopted the same idea except that instead of presence we considered the illusion of body ownership, and participants reported when the illusion vanished.

We found that a break in body ownership could be caused equally by asynchronous visuomotor or visuotactile stimulation. Hence while synchronous visuomotor was paramount in generating the illusion, the number of breaks that occurred did not differ between visuomotor and visuotactile asynchrony.

We also recorded skin conductance and heart rate. This was in order to measure the response to a sudden event that took place. The picture above shows that participants had their real (and virtual) legs resting on a table. At the end of all the stimulation the table suddenly pulled away. Participants tended to react with an involuntary response to stop their legs from falling, and this showed up in both heart rate and skin conductance changes. Moreover these changes were positively correlated with a questions about how stressed they had felt at that moment. However, there were no differences in the physiological changes amongst the four conditions of the experiment - the event was equally arousing under all conditions. We believe that this is because seeing a virtual body that coincides spatially with your own body is already enough to produce a body ownership illusion. Additional synchronous multisensory stimulation only adds to this.

By looking at breaks in the body ownership illusion we were able to assess subjective ownership through time as well as at the end of all the stimulation. To obtain this information we used the same method as in the ‘breaks in presence’ work - that is we only asked participants to indicate when the illusion broke, and not when it started. For presence this procedure makes sense, because if we ask people to report when they become ‘present’ in the virtual place the very requirement to report this may disrupt it. Asking them to report when the illusion breaks is not the same, since, of course, the illusion has already broken. However, with hindsight it is probably possible to ask people to report when an illusion of body ownership kicks in without disrupting the illusion. Indeed we did this in an earlier paper (PDF) (for different reasons - we were interested in estimating time for the rubber hand illusion to start). Although based only on reporting breaks our statistical method can estimate the probability of being in the illusion of ownership state, in future work we will also try out the idea of asking participants to report when the illusion starts as well as when it ends.

Finally our method also includes an approach that may overcome some problems in subjective assessment of the body ownership illusion. Normally researchers ask participants in experiments under different conditions to report things like ‘How much did you have the feeling that the (virtual / manikin) body was your body?’. But participants naïve to this idea (as they should be) have no real clue what we are talking about. In everyday life we do not go around thinking "Oh my body feels like it belongs to me." As is the case with presence, the special qualia attached to a 'body ownership illusion' is to have that feeling of ownership even knowing that it is an illusion - that the virtual body is obviously not really your body. 

Now especially in control conditions (e.g., asynchronous) we are asking them to report on something that they do not know about - yet of course they will always give some answer to a questionnaire. This is especially problematic in within-group studies where we ask people to report the strength of the illusion in both an experimental (e.g. synchronous) condition and in a control (e.g., asynchronous condition). But these are not balanced in the sense that the order of the conditions does matter. Experiencing first an asynchronous condition and then a synchronous one is really very different from the other way around - since when the synchronous condition is experienced first participants know what you are talking about with respect to ‘body ownership’ and therefore can more appropriately evaluate the asynchronous condition. No amount of counter balancing can overcome this, and anyway it violates a fundamental assumption behind the statistical analysis (by ANOVA) of within-group designs - that all orders of stimuli delivery are equivalent.

Here what we did is give all participants the experience of the best setup that we could offer (within the constraints of that experimental design) to induce body ownership. Hence when later we ask questions about their responses to the experimental conditions they have already experienced the ‘best’ setup, so that they have an experience against which they can compare.

Questionnaires alone are never the best method of measurement. But they can be improved through asking people to compare their responses to different setups; but first demonstrating as a baseline the closest we can get to inducing the strongest response.

Kokkinara, E., and Slater, M. (2014). Measuring the effects through time of the influence of visuomotor and visuotactile synchronous stimulation on a virtual body ownership illusion. Perception 43, 43 – 58. (PDF).

YouTube Video

This research was conducted as part of the VR-HYPERSPACE project.