Mel Slater's Presence Blog

Thoughts about the concept of presence in virtual reality - a place to write freely without the constraints of academic publishing,and have some fun.

My Photo

I still find immersive virtual reality as thrilling now as when I first tried it 20 years ago.

07 November, 2014

What happens in your brain when your virtual body is threatened?

What happens in your brain when your virtual body is threatened?

Mar González-Franco, Tabitha C Peck, Antoni Rodríguez-Fornells, Mel Slater (2014)  A Threat to a Virtual Hand Elicits Motor Cortex Activation   Experimental Brain Research 232: 3. 875-887.

Figure 1. The experimental setup. Real: the participant wears a high resolution, wide field-of-view, stereo, head-tracked head-mounted display (NVIS SX111) and EEG cap (g.tec). Virtual top: the virtual reality showing the gender-matched virtual body spatially coincident with the participant’s actual body, and in the same posture. Virtual bottom: The two experimental conditions seen by the participant when looking towards the hand from a first person perspective: HAND - virtual hand stabbed by the knife; TABLE - virtual table stabbed by the knife (control condition).

When you wear a head-tracked wide field-of-view stereo head-mounted display and you look down towards your body you can see a life-sized virtual one instead. You look around and you see a reflection of that body in a mirror. In your whole life whenever you have looked towards your body you have seen it, likewise towards a mirror. Hence the simplest perceptual hypothesis for the brain to adopt is that this is your body.

In this study we looked at what happened when the virtual body was threatened. When someone anticipates that a knife might stab their hand that is resting on a table they would be likely to attempt to move the threatened hand out of the way. They would expect to feel considerable pain should the knife actually stab the hand. In this work we considered what happens when a person’s real body is visually substituted by a life-sized virtual body, and they see a threat or attack to a hand of this virtual body seen from first person perspective. Our experiment investigated brain activity in response to events that would cause pain to the observer were these events to occur in reality. Our contribution has been to  introduce a new technique for the study of pain observation, by using immersive virtual reality (IVR) for the scenario and stimulation, while recording brain activity with EEG.

Pain observation experiments typically present a series of pictures with hands or other extremities undergoing painful situations, and they compare the brain response of the participants to the activation produced by pictures where the same extremities do not undergo painful situations [1-4]. Many of these experiments present scissors and needles perforating the extremities as painful stimuli. A potential advantage of immersive virtual reality is that there is greater ecological validity, going beyond the presentation of two-dimensional, static stimuli. There is a life-sized, three dimensional virtual body seen in stereo, that visually substitutes the obscured real body of the participant and results show that this normally induces a whole body ownership illusion [5]. Our hypothesis was that harm to the virtual hand would be associated with positive changes in P450 in line with previous studies, and that this would be enhanced with illusory body ownership. We also investigated the mu band and readiness potential (RP).

While immersed in the virtual reality the 19 participants (10 female, right-handed) repeatedly experienced during 15 minutes two conditions in a within-group design: HAND where the knife stabbed the virtual right hand, and TABLE where the knife stabbed the table 15 cm away from the right hand (Figure 1). The experiment consisted of 70 trials repeating the HAND and TABLE conditions (30 HAND and 40 TABLE).

Both EEG and electromyography (EMG) were recorded using an gUSBamp  amplifier with a resolution of 30nV; the electrodes were set to cover the motor cortex area and surrounding: FC3, FC4, C3, C4, CP3, CP4 located according to the 10/20 standard EEG recording; the reference was set with an ear clip on the left ear lobe; the ground was positioned on the forehead; electrodes in the face measured ocular activity (EOG). Three EMG electrodes were placed in the flexor carpi ulinaris muscle of the right arm to measure whether participants moved their hand. All the electrodes were kept to impedances below 10 kΩ. The data was recorded with a sampling frequency of 512 Hz. After the exposure participants answered a questionnaire on a 1-5 Likert Scale where 1 was anchored to strong disagreement and 5 to strong agreement:

Ownership: I felt as if the hand I saw in the virtual world might be my hand.
Harm Hand: I had the feeling that I might be harmed when I saw the knife inside the hand.
Harm Table: I had the feeling that I might be harmed when I saw the knife outside the hand.
No Ownership: The hand I saw was the hand of another person.
Body Threat:  I saw the knife as a threat to my body.

Figure 2. EEG Recordings. Left: Grand averaged stimulus locked ERPs for six representative front, central and parietal electrode locations. A significant increase in the amplitude of the P450 is observed in the HAND condition mainly at C3 and CP3 locations. Baseline from [-200 ms to 0 ms], time 0 indicates the stimuli onset; a low pass filter 12Hz half-amplitude cutoff was applied. Right: (a) Time Frequency Evolution of the two conditions and the difference in the spectral activity. (b) Grand averaged 1-s short time power spectra calculated from EEG data (electrode C3) recorded. The baseline corresponds to the range [-1 to 0] seconds before the stimuli and the activity period corresponds to the range [0.7 to 1.7] seconds after the stimuli. Both the Baseline and TABLE frequency spectra show a peak in the mu-rhythm that is attenuated in the HAND condition. (c) Grand averaged Mu-rhythm (9-12Hz) Event Related Desynchronization for the C3 electrode. (d) Grand averaged Readiness Potential (C3-C4) subtraction between the brain activity in the two hemispheres shows movement preparation effects. Low pass filter 8Hz, half-amplitude cutoff.

Figure 3. Box plots showing the responses to the questionnaire. The thick lines are the medians, and the boxes are the interquartile ranges (IQR). Wilcoxon matched pairs sign-rank tests show differences between Ownership and No Ownership  (P < 0.0001); Harm Hand and Harm Table (P < 0.0002); Body Threat and Harm Table (P < 0.0003). Harm Hand and Body Threat ( P < 0.018).


• The results suggest that when a person is in an immersive virtual reality and has body ownership illusion towards a virtual body that apparently substitutes their own body, there are autonomic responses that correspond to what would be observed were the events to take place in reality. Overall automatic brain mechanisms –P450– were found in this variation of the classical pain observation experiment, which is consistent with previously reported results.

• The results cannot be explained as participants experiencing empathy towards another person since they witnessed attacks to their co-located virtual body and both subjective and objective data suggest that they experienced this as an attack on their own body.

• The results support our initial hypothesis that a threat to a virtual hand, towards which the participant has an illusion of ownership, would significantly produce a harm prevention effect (the Readiness Potential (C3-C4) and oscillatory movement-related components, the mu-ERD), such as trying to move it away from the source of the harm. The questionnaire also confirmed high levels of ownership over the virtual body.

• The correlation between the automatic brain mechanisms –P450– and the subjective illusion of ownership suggests  a potentially new measure of virtual embodiment.

1. Avenanti, A., et al.,. NeuroImage, 2006. 32(1);  
2. Bufalari, I., et al., Cerebral Cortex, 2007. 17(11); 
3. Fan, Y. and S. Han, Neuropsychologia, 2008. 46(1);  
4. Li, W. and S. Han, Neuroscience Letters, 2010. 469(3);
5. Slater, M., et al., PLoS ONE, 2010. 5(5).

Funded by European Union FP7 IntegratedProject VERE (#257695);  FI-DGR predoctorate grant from the Catalan Government co-funded by the European Social Found (EC-ESF); Spain MICIN (PSI2011-29219);  ERC project TRAVERSE (#227985).  


12 July, 2014

Visual-Tactile and Visual-Motor Influences on Virtual Body Ownership

The vast amount of research on the rubber hand illusion uses visuotactile synchronous stimulation to induce the illusion. This means that sight of the rubber hand being touched is synchronous temporally and spatially with the tactile stimulation felt on the corresponding (hidden) real hand. It has also been shown that the illusion can be induced with visuomotor stimulation - meaning that (in this case) the virtual hand moves synchronously with the movements of the corresponding (hidden) real hand.

For most of our work on virtual full body ownership we have relied on visuomotor effects, where the virtual body moves synchronously with the real body. This is accomplished through real-time motion capture, so that participant movements are mapped to the corresponding movements of the virtual body. The virtual body is seen directly by looking towards it through the head-mounted display, and also in a virtual mirror.

Which of these two methods of stimulation is the most powerful in inducing the body ownership illusion - visuotactile or visuomotor? In a recent paper  (PDF) we describe an experiment that addresses this question.

Participants were in a reclined position and saw their full virtual body from a first person perspective through a head-tracked, wide field-of-view head-mounted display. As they moved their leg (A) the corresponding virtual leg would move synchronously or asynchronously (B). When the experimenter tapped a leg with the wand (C) the participant would see a virtual ball tapping the corresponding position on the virtual leg (synchronously or asynchronously). Hence we had a 2 by 2 experimental design (synchronous movement or asynchronous combined with synchronous tapping or asynchronous) where these were delivered alternately.  There were 60 participants in a between groups design - hence each group of 15 experienced just one of the 4 combinations of these two factors.

The experiment was organised so that every subject first experienced for a while the best possible setup - that is visuomotor and visuotactile synchrony. Then after some questions had been answered they experienced one of the 4 conditions. Then questions about body ownership and agency were again answered.

Based on the questionnaire responses visuomotor synchrony outweighed visuotactile in producing the illusion.

However, unusually, we also attempted to measure not only what generated the illusion but also what extinguished it. Here we used a method first proposed in the study of presence in virtual reality (the sensation of being in the place depicted by the virtual environment displays). This method is called ‘breaks in presence’. The assumption is that the normal state is for the illusion of presence to occur, but occasionally it breaks for various reasons (errors in rendering or tracking, physical entanglement with cables, bumping into a Cave screen, or just spontaneous switches in attention or perception). From an indication by participants about when each break occurs it is possible to estimate an overall probability of presence (the proportion of time the participant had this illusion). Here we adopted the same idea except that instead of presence we considered the illusion of body ownership, and participants reported when the illusion vanished.

We found that a break in body ownership could be caused equally by asynchronous visuomotor or visuotactile stimulation. Hence while synchronous visuomotor was paramount in generating the illusion, the number of breaks that occurred did not differ between visuomotor and visuotactile asynchrony.

We also recorded skin conductance and heart rate. This was in order to measure the response to a sudden event that took place. The picture above shows that participants had their real (and virtual) legs resting on a table. At the end of all the stimulation the table suddenly pulled away. Participants tended to react with an involuntary response to stop their legs from falling, and this showed up in both heart rate and skin conductance changes. Moreover these changes were positively correlated with a questions about how stressed they had felt at that moment. However, there were no differences in the physiological changes amongst the four conditions of the experiment - the event was equally arousing under all conditions. We believe that this is because seeing a virtual body that coincides spatially with your own body is already enough to produce a body ownership illusion. Additional synchronous multisensory stimulation only adds to this.

By looking at breaks in the body ownership illusion we were able to assess subjective ownership through time as well as at the end of all the stimulation. To obtain this information we used the same method as in the ‘breaks in presence’ work - that is we only asked participants to indicate when the illusion broke, and not when it started. For presence this procedure makes sense, because if we ask people to report when they become ‘present’ in the virtual place the very requirement to report this may disrupt it. Asking them to report when the illusion breaks is not the same, since, of course, the illusion has already broken. However, with hindsight it is probably possible to ask people to report when an illusion of body ownership kicks in without disrupting the illusion. Indeed we did this in an earlier paper (PDF) (for different reasons - we were interested in estimating time for the rubber hand illusion to start). Although based only on reporting breaks our statistical method can estimate the probability of being in the illusion of ownership state, in future work we will also try out the idea of asking participants to report when the illusion starts as well as when it ends.

Finally our method also includes an approach that may overcome some problems in subjective assessment of the body ownership illusion. Normally researchers ask participants in experiments under different conditions to report things like ‘How much did you have the feeling that the (virtual / manikin) body was your body?’. But participants naïve to this idea (as they should be) have no real clue what we are talking about. In everyday life we do not go around thinking "Oh my body feels like it belongs to me." As is the case with presence, the special qualia attached to a 'body ownership illusion' is to have that feeling of ownership even knowing that it is an illusion - that the virtual body is obviously not really your body. 

Now especially in control conditions (e.g., asynchronous) we are asking them to report on something that they do not know about - yet of course they will always give some answer to a questionnaire. This is especially problematic in within-group studies where we ask people to report the strength of the illusion in both an experimental (e.g. synchronous) condition and in a control (e.g., asynchronous condition). But these are not balanced in the sense that the order of the conditions does matter. Experiencing first an asynchronous condition and then a synchronous one is really very different from the other way around - since when the synchronous condition is experienced first participants know what you are talking about with respect to ‘body ownership’ and therefore can more appropriately evaluate the asynchronous condition. No amount of counter balancing can overcome this, and anyway it violates a fundamental assumption behind the statistical analysis (by ANOVA) of within-group designs - that all orders of stimuli delivery are equivalent.

Here what we did is give all participants the experience of the best setup that we could offer (within the constraints of that experimental design) to induce body ownership. Hence when later we ask questions about their responses to the experimental conditions they have already experienced the ‘best’ setup, so that they have an experience against which they can compare.

Questionnaires alone are never the best method of measurement. But they can be improved through asking people to compare their responses to different setups; but first demonstrating as a baseline the closest we can get to inducing the strongest response.

Kokkinara, E., and Slater, M. (2014). Measuring the effects through time of the influence of visuomotor and visuotactile synchronous stimulation on a virtual body ownership illusion. Perception 43, 43 – 58. (PDF).

This research was conducted as part of the VR-HYPERSPACE project.

15 February, 2014

The Presence of Your Distant Virtual Body

One way to think about a body ownership illusion is that it arises when the brain attempts to solve a contradiction between different sensory modes, and chooses as a resolution the simplest hypothesis that appears to explain what's happening. For example, in the rubber hand illusion your real out-of-sight hand feels tactile stimulation, while a visible rubber hand on the table in front of you, and in a plausible position and orientation, is seen to be touched. The felt touch and seen touch are synchronous. So there is a contradiction between the felt touch in one place and the seen touch in another. The contradiction is resolved by going along with the hypothesis that the rubber hand is indeed your own hand.

There is a more severe contradiction discussed in the literature: you see your body some metres in front of you being tapped on the back, but you feel the tapping (of course) on your back (synchronously with the seen tapping). Here the contradiction produces a strange effect - somehow your body is over 'there' (the sensation of touch can shift to the body in front) but of course your visual ego-centre is 'here' (where you are really located). There can be a reported sensation of body ownership over that distant virtual body. But is this 'body ownership' in the sense of somehow feeling your body to be at the distant location, or is it just a question that you recognise the body as your body, and so in answer to a questionnaire would give a high score to a question about body ownership, but qualitatively this is not the same as when your virtual body is spatially coincident with your real body and seen from first person perspective?

We examined this issue with immersive virtual reality, and partially reproduced the experiment described above.  The conclusion we reached is that the way the brain attempts to resolve the contradiction between you being 'here' but feeling stimulation on the body in front over 'there', is to produce illusions of drift. One way to resolve the contradiction between the two locations is to make them coincide. So if the virtual body in front is illusorily perceived as drifting backwards towards your position, or if you have the illusion of drifting towards the virtual body in front, then the two bodies will become united into one.

In our experimental setup the virtual body in front was subjected to a threat (a spinning fan was lowered towards its head). So if you drift towards the body in front this would put you in danger, but if the body in front drifts back towards you that would put it out of danger. We found a strong positive correlation between the illusion of drifting forward and heart rate deceleration (increasing danger) but a negative correlation with the virtual body drifting backwards  (decreasing danger).

The paper can be read in Frontiers in Human Neuroscience

15 July, 2013

Presence Through the Eyes of a Child

 Using a head-mounted display and body tracking suit, entering into a virtual reality, you can experience yourself as a child of about 4 years old. You look into a mirror, or directly down towards your own body, but you see the child body instead. The brain appears to be remarkably flexible in quickly accepting the proposition that your body is different - especially when you move your body the virtual body is seen to be moving the same as you feel yourself to be moving.The virtual body has substituted your real body.
Alternatively you can be embodied in a virtual body of the same size as the child one, except that this is a shrunken down adult body. Otherwise everything is the same. In both conditions people tended to have a strong illusion that the virtual body was their body.
 The question we set out to answer with this arrangement is whether embodiment in the two different types of bodies would lead to differences in perception and also attitudes. You remember as a child that things seemed to be enormous, that if you see them today they don't look that way. Is it just a question of your size, or is something more at work? It has been shown that size illusions operate when you make people apparently small or big - like if you were the size of a Barbie doll, how would you see the world? You see it bigger. What we found though goes beyond that. In the two conditions (child or shrunk down adult) both overestimated sizes of objects, as expected. However, the child condition led to much greater size overestimation. It must therefore be not just the size but the form of the body that is having this effect.

We also gave people an implicit association test. This requires people to quickly categorise themselves according to child or adult attributes. Their adult attributes (like their age, what they do etc) were obtained a while before the experiment from a questionnaire. Those in the child condition nevertheless were found to identify themselves more with child like attributes than those in the adult condition.

A critical aspect of the findings was that the differences between the child and adult embodiment was due to the degree to which participants had the sensation that the virtual body was their body (their degree of 'body ownership' over the virtual body). We had another condition where everything in the setup was the same, except that the virtual body moved independently of the person's real body movements. In this condition the illusion of body ownership was very much reduced compared to the condition when the virtual body moved synchronously with the real body movements. In this asynchronous condition the difference between the child and adult conditions vanished. Both still overestimated sizes, but there was no difference between them, and the overestimation was about the same as that in the synchronous adult condition.

The body has a kind of semantics, meaning is attributed to a body type. In this case it was a child's body, something of which we've all experienced. Perhaps embodying people in such a child-like body automatically leads the brain to bring to the fore types of mental processing that go along with being a child. We have only shown this with respect to size perception, and implicit associations, but maybe there is more to this. Also we do not know how long the effects last - much work remains to be done.

Domna Banakou, Raphaela Groten, and Mel Slater (2013) Illusory ownership of a virtual child body causes overestimation of object sizes and implicit attitude changes, PNAS doi: 10.1073/pnas.1306779110


31 May, 2013

Racial Bias and Virtual Embodiment

You look at your body directly and also you see it in a mirror. When you move the body moves. The strange thing is that although you don't recognise it as your body, nevertheless it feels as if it is. This type of virtual body ownership illusion has been demonstrated several times. What we are interested in though is the consequences of this body ownership illusion. Specifically, does the type of body that you feel is your body influence your behaviours, perceptual judgements, even your cognition?
Body Semantics
A body type has an intrinsic meaning. For example, through stereotyping when you see a very old person you might automatically associate that person with certain deficits in cognition and strength - even though that might be totally untrue for that particular person. I call this 'body semantics' - the type of body carries with it certain predisposed attitudes, behaviours, psychology, physical abilities.  That's when you see someone else's body. What about your own?
The beauty of virtual reality is that we can change your body. In a recent paper we showed that if you take one group of people and you put them in a casually dressed body suggestive of being 'cool', progressive, relaxed then they play the drums with greater body movement than another group of people put in a suited formal looking body. Here we followed a similar idea, and placed light skinned people in a dark skinned virtual body, using a wide field-of-view, head-tracked head-mounted display, and a motion capture suit to track the person's real body movements. The participants saw their virtual body from first person perspective with respect to the viewpoint of that body, so that when they looked down towards themselves they saw the virtual body instead of their real one. When they moved the body moved the same (through the motion capture suit). When they looked in a virtual mirror they saw this other body instead. 

Implicit Association Test
A few days before they entered the virtual reality we applied a test called an Implicit Association Test (IAT) for racial bias. What this does is force you to make rapid associations between concepts and representations of Black or White people. The idea is that if your reaction times in pairing White faces with positive words and Black faces with negative words is faster than White faces with negative words and Black faces with positive words, then this indicates a racial bias. It does not mean that the person is racist - far from it - but perhaps reflects an implicit and automatic bias caused through socialisation via the media (perhaps 'anti-socialisation' would be more appropriate).
The experimental study had four different conditions (experienced by 4 different groups of 15 people): being embodied in a light skinned body, a dark skinned body, a purple skinned body, or no actual body but a dark skinned body in a mirror that did not move the same as the participant. The purpose of the purple skinned body was to check whether the effects were caused by just 'difference' or 'strangeness' or race. The purpose of the 'no body' was to check that the results were caused by the illusion of body ownership over the virtual body, and not simply seeing the different body.
During the virtual reality experience nothing much happened - some virtual characters walked past the participant - half of these dark and the other half light skinned. After the conclusion of the experience the participants again completed an IAT test.

What happened is that the IAT score declined only for those who had been in the dark skinned body. Somehow, becoming, however briefly, a member of the 'out group' in a very obvious way was enough to signal the brain that this was no longer your 'out group' but your in group. The fact that this operates so fast is remarkable, but in these body illusions I find everything remarkable - the fact that a few seconds of stimulation can make a rubber arm feel like your own, or a few seconds of apparently being in another body can make it feel like your own. By the way the group of Manos Tsakiris at Royal Holloway London, very recently also demonstrated a similar effect by using a black rubber hand in the rubber hand illusion.

Tabitha C. Peck,  Sofia Seinfeld, Salvatore M. Aglioti  & Mel Slater (2013) Putting yourself in the skin of a black avatar reduces implicit racial biasConsciousness and Cognition, 22(3), 779-787.

This work is funded under the FP7 Project VERE and the ERC Project TRAVERSE.

31 January, 2013

In the Presence of Violence

In the Presence of Violence

The Scene

You’re in a rather empty bar minding your own business, enjoying a quiet drink, and eventually a stranger walks in and starts to talk to you. He is pleasant and you find that you have some things in common, in particular you both support the same football team. The friendly chat lasts a while, and suddenly someone you hadn’t noticed, who’d been sitting by the bar, a third person, strides over and starts to accuse your new found acquaintance of “staring” at him. Of course he denies this, saying that he’d just been talking to you. This third person becomes increasingly aggressive in tone and uses threatening body language towards your acquaintance. Whatever the latter says is immediately turned around to make it sound like an escalation of the argument, and soon it becomes clear to you that the only goal of the aggressor is to cause a fight. Occasionally the victim, the one you’d been having the friendly chat with, looks towards you, but he doesn’t actually ask you for help. Obviously this is going to end in some pretty bad violence, in spite of the fact that your acquaintance is trying everything he can to de-escalate.

The question of interest here is – what do you do? Do you intervene? Do you quietly leave the bar? Do you stand there trying to think what to do but not actually do anything? Do you freeze? What should you do? What is the right thing to do? Would it make things worse or better if you intervened?

Now replay the above scene. The man walks into the bar, and he starts chatting with you. You find that you and he really have nothing much in common. Although he seems to be interested in football, it is apparent that he does not support the same team as you. Otherwise, everything is the same as in the first version. The aggressor clearly wants to start a fight and this is going to end in violence.

What do you do in these slightly different circumstances?

Bystander Intervention

This is an example of the so-called ‘bystander’ problem – how people respond when they come face to face with violent emergencies. I wrote about this in an earlier blog entry ‘The Illusion of Violence’. As pointed out earlier it is very difficult to study this type of situation experimentally – to investigate the factors that might lead to someone intervening or not. There have been many studies of this type of bystander situation, but experiments do not actually include violence, or even if they do it is not a face-to-face type of violent scene, but may be something that is shown on a video.

Our interest was to find out what people actually do when they are face-to-face with this type of violent confrontation. Yet we cannot carry out experiments in ‘real life’. However, research over the past 25 years has shown that people immersed in a virtual reality tend to behave realistically – carry out actions, have emotional responses, even have thoughts that would be appropriate for a situation occurring in reality. By being ‘immersed’ I mean that they are in a computer generated surrounding environment, that they see in 3D stereo, where everything is fully life-size, where they can perceive using their body in a natural way – for example, turning their head to look to the side, bending down to see underneath something, and so on. This is especially important interacting with virtual humans – these are life-sized, they talk with you, they look you in the eye, it would seem as if they could touch you – they seem to respond to you, they have a life-like presence in the same space as you. Now under these circumstances you’re talking with a (virtual) acquaintance, and the third (virtual) man suddenly appears and violently threatens your acquaintance, how do you respond?

The Bystander in Virtual Reality

We recently had a paper published that addresses this issue. The particular factor of interest that we looked at was the extent to which your responses are modulated by the sense of group identity between yourself and the victim. In order to be able to manipulate group identity in a natural way we used football (soccer) club affiliation. We recruited 40 male supporters of the Arsenal football team. They went through the experience described in the opening paragraphs. However, for 20 of them the eventual victim was clearly himself an Arsenal supporter, and for the other 20 he clearly was not an Arsenal supporter. So 20 of them, with respect to this situation, were ‘in-group’ with respect to the victim and the other 20 ‘out-group’ (or at least definitely not ‘in-group’). The aggressor was clearly ‘out-group’ since he made it very clear many times that he ‘hated’ Arsenal and thought that they were an extremely useless soccer team (I’ve put it more politely here than the way that he expressed it).

We wanted to see whether group affiliation could predict helping behaviour. We measured the latter by the number of physical or verbal interventions that the participants made once the argument had started. A physical intervention might be something like trying to step between the two characters, or gesticulating towards them.  Also for half of the participants we programmed the scenario so that the victim would sometimes look toward the participant, and for the other half not. So finally this experiment had 4 conditions: in-group, looking; in-group, not looking; out-group looking; out group, not looking – with 10 participants arbitrarily assigned to each group (in fact data for 2 of the participants were not usable, so we ended up with 38 not 40).

What we found was interesting – that the ‘in-group’ people intervened on the average more than the ‘out-group’ people. In a questionnaire after the experience we had asked how much the participants thought that the victim had been looking towards them for help. For those in the ‘in-group’ the stronger their belief that the victim was looking towards them for help the greater the number of interventions. However, for those in the ‘out-group’ condition there was no such relationship.

Now the finding is on the face of it fairly obvious. If a fight breaks out between two people in your presence you might be inclined to be more likely to help the victim if he had some affiliation with you (in this case supporting the same football team), other things being equal. However, the interesting aspect is that this appears to occur also in virtual reality, even though everyone knows that nothing ‘real’ is actually happening.

Another aspect of the results that is hard to convey in a journal paper is the actual reactions of the participants – that they were disturbed by the situation, reporting things like a racing heart, and also irrational worries like if they had intervened then the aggressor might have turned on them, and so on. As I’ve said before, some part of the brain does not understand about virtual reality – and simply takes what is happening at face value – and responds. Of course you ‘know’ that nothing real is happening, and therefore a slower cognitive response might then act to dampen down your responses compared to those that might occur if the events were actually taking place in reality. In virtual reality studies in fact typically we are trying to capture that first automatic response, the one that happens before you have ‘time to think’. This is the genuine response, and the one most likely to be similar in virtual and physical reality.

Another interesting result of this experiment is that it illustrates that the quality of the computer graphics is not vital. If you look at the video there will be something striking – when the virtual characters talk their mouths do not move! There is no lip sync. After the experiment we asked participants what things they thought took them out of the experience – how could the scenario have been improved? Very few people actually mentioned the lack of lip sync. I think that they became so involved in the situation that they somehow didn’t notice it.

One thing that they did tell us though was that ‘a fight like that would never happen in a bar like this’ – in other words the décor of the bar was wrong, it was not a bar that would be frequented by this type of football supporter. This aspect of plausibility is extremely important, and requires research on the domain to be simulated for any kind of experiment that is supposed to be depicting events that could happen in reality. (See ‘Illusion is Part of the Definition’).

Statistical Diatribe and Symbolic Regression

One other aspect of this paper is quite new. Research in this field follows the conventions of psychology (in this case social psychology) in terms of statistical analysis and reporting. In psychology there is convention of the 5% significance level, enforcement of the frequentist interpretation of statistics rather than Bayesian, and the tyranny of linearity. When you carry out a standard analysis such as regression or analysis of variance, even if using a generalised linear model,  somewhere in this is a very strong assumption of a linear (in fact affine) relationship between the response variable and the independent and explanatory variables (even if the variables themselves might be transformed e.g. to a log scale). But why should everything be linear? In fact it is safe to say that conventional statistics makes the linearity assumption mainly because the mathematics and computation is easier – and certainly the latter is a factor that should not make us stop and think too much today, compared to the nearly a century ago when many of these techniques were invented.

In this paper we had both the observed intervention data (number of physical and verbal interventions) and questionnaire data. I wanted to see what the relationship was between the number of interventions and the responses to the questionnaires. I used a method called ‘symbolic regression’ (which is a specific aspect of Genetic Programming). In particular I used a system called Formulize (or Eureqa) in order to analyse the relationship between the numbers of interventions and the subjective questionnaire responses. See the paper by Michael Schmidt and Hod Lipson in Science.  Supporting Text S3 of our paper briefly explains how this works. The important thing is that this discovered something that I don’t think would be possible with conventional statistics. For the number of physical interventions (N), the resulting equation was of the following form:

N = group*exp(LookAt + VictimLooked) + f(… other questionnaire variables…)

The f() represents some function which isn’t important in this particular discussion. The variable group = 0 means ‘out-group’ and group = 1 means ‘in-group’. LookAt = 1 for the case when the victim occasionally looked towards the participant during the argument and LookAt = 0 for the group where this did not happen. VictimLooked is the response to the statement: “After the argument started, the victim looked at me wanting help.” This (as all questions) was scored on a 1-7 scale where 1 meant least and 7 most agreement with the statement.

Now if we look at this we see that the whole first term on the right hand side of the equation vanishes for the ‘out-group’. Hence only for the ‘in-group’ were the ‘look at’ factor and the strength of the belief that the victim was looking for help important. For the ‘out-group’ these factors seemed to have no effect.

This equation captures 85% of the variation in the original data. The point is that it is much easier to look at this equation and try to understand what it signifies than looking at the original data (that it very well) represents. Reviewers in the psychological and social sciences have to accept that the world has changed since the 1920s when methods such as ANOVA were invented, and that this type of data exploration is a valid way to understand data. There is a world beyond formal 'hypothesis testing'. For example, now that we have this type of equation, what would be wrong with an experimental replication that ran the symbolic regression on the new data and then compared the form of the equation with the original data? Or, having found this equation it tells us that the more we foster the idea that the victim is looking to the participant to help the greater the number of interventions should be. We could set up an experiment to test that specific hypothesis. This is not a challenge to conventional methods, but a statement that there is more - and different techniques should not lead to suspicion. As another example, when we first analysed the 'number of interventions' data we did not use standard ANOVA, which is based on the assumption of a continuous response variable, and a normally distributed error structure. This is because 'count data' (the number of times something happens) is better modelled with a generalised linear model with a Poisson error structure (log-linear regression). This was treated as something 'suspect' by a first round of reviews, even though generalised linear models with Poisson error have been around for at least 60 years!

So What?

What are the conclusions from this type of experiment? First is that using VR in this way allows us to carry out lab based experiments where participants are confronted with a situation that has high ‘ecological validity’ – it is almost like real life. Such experiments don’t come out of nowhere, but they are guided by theory. Here it was the idea suggested by Mark Levine amongst others that group affiliation plays a major role in bystander behaviour – the apparent relationship between the bystander, the victim and the perpetrator. Having carried out the experiment we obtain data so that we can now look again at theory with this additional information, and perhaps formulate a revised theory, leading to another experiment. On the latter we have since carried out an experiment where we change the number of bystanders (not just one – the participant) and have examined the effect of that. The results of the new study are in preparation.

On the practical side we could suggest, for example, that if you are a victim, then yes do explicitly ask people around you for help. This might not be effective if those around do not share some group affiliation with you, but it should help if they do. Even if they do not share affiliation with you, this is something that itself is open to reframing. For example, fans of two rival football clubs might be bitter enemies, but if the situation is redefined so that they are both ‘football enthusiasts’ (compared to say rugby enthusiasts) then at that level they do have a joint affiliation. This was explored in where bystanders to a (non-violent) emergency behaved differently depending on whether they had been primed to think of themselves as fans of a specific football club, or general football supporters. So for a victim it might always be possible to appeal to higher level affiliations if it is possible to seek help from bystanders.

For the bystander him- or herself this experiment cannot say what they ‘should’ do. This depends entirely on circumstances and on the moral choice made by the bystander taking into account many factors – including most importantly their own safety and of others around. But for authorities this type of experiment might be very useful for the formation of policy. Even from this simple experiment, authorities could give advice that victims should explicitly ask for help if there are any potential bystanders around, even perhaps with advice about how to ‘reframe’ group affiliation (e.g., “Think about how my kids will feel”, might appeal to bystanders around who happen to have children!). Or another example, it is widely believed in the social psychology community that there is a ‘bystander effect’ such that the greater the number of bystanders the less the chance that anyone will intervene – because there is a diffusion of responsibility (“Why should I be the one to stop this?”). If that were the situation what should the victim do to break this? Should for, example, he or she choose someone at random in the crowd and appeal specifically to that person? Or should the victim somehow try to raise group consciousness towards prosocial behaviour (“You are all party to this attack by not helping me!”). We don’t know the answer, but with an experimental study we could gain some insight into this.

(WARNING -  this video includes bad language and
depicts a violent confrontation).

videoBystander Responses to a Violent Incident in an Immersive Virtual Environment

  • Mel Slater, 
  • Aitor Rovira,
  • Richard Southern,
  • David Swapp,
  • Jian J. Zhang,
  • Claire Campbell,
  • Mark Levine