Mel Slater's Presence Blog

Thoughts about research and radical new applications of virtual reality - a place to write freely without the constraints of academic publishing,and have some fun.

13 April, 2020

A Shocking Outcome in Virtual Reality

This blog post is on Behavioural and Social Sciences Nature Research.

29 March, 2020

Virtual reality for training

Virtual Reality for Training

Mel Slater

Simulation has been used for training for decades, most notably for flight simulation. These go back to the 1920s, the first being the Link Trainer. A flight simulator is essentially a complete aircraft cockpit, but with all of the visual, auditory and haptic information that signifies the state of the flight being produced by digital means. So the window views are generated by computer graphics, the sound is digitally controlled, and the simulator is in a huge box that is mounted on a platform that delivers the forces to which the simulated aircraft is subject. This is a kind of mixed reality system – the cockpit and all the controls are real, but the displays and forces acting on the physical structure are digitally produced.

Virtual Reality (VR) offers the possibility of training through mainly digital feedback, in the form of graphics, sound and haptic feedback where appropriate. Also it can apply to a very wide variety of circumstances without the need for highly expensive and non-portable platforms. For example, people in a company who have to present in front of clients can learn how to improve their performance by rehearsing in front of an entirely virtual audience, with the help of a trainer. A quite different application might be preparing for emergencies such as a fire in an office. All of these different types of application require essentially the same, and these days quite portable, hardware. 

Acquiring new skills, procedures, and knowledge, through action – eventually leading to expertise. It should be generalisable, so that when new and slightly different situations arise the practitioner can deal with these based on acquired knowledge of similar situations.
Training involves skills transfer where existing skills are utilised to learn new ones – for example, a mechanic may have very well learned how to forge a particular component, but then more rapidly learn what mistakes to avoid when making another different but similar enough component. It is important that training is in line with organisational needs, and follows the organisation’s ethos. 

Why is VR any use for this? Previously I have referred to three perceptual illusions which arise from the use of VR: Place Illusion (the illusion of being in the place depicted by the VR displays), Plausibility Illusion (that events happening there are really happening) and a Body Ownership Illusion (that your co-located virtual body is your body). Together, these mean that you will tend to act in VR much as you would in similar circumstances in reality. So when you stand over a precipice in VR, even though you 100% know that it is not real, and that there is nothing there, you can’t but help have feelings of anxiety. You respond as if it is real.  However, for VR to be useful in training we have to have evidence that people do respond realistically in VR. An example of the type of study that is necessary was given by Bhagavathula (2018) who compared pedestrian behaviour in reality and VR. It was found that pedestrians made similar decisions about crossing the road in both VR and reality, and there was no difference between VR and reality regarding the perceived risk, and estimations of distance of vehicles. However, there was a difference in estimation of speed of an approaching vehicle. On the one hand it is remarkable that the differences between reality and VR were so low, because an essential feature of VR is that everyone knows that even if an approaching car would hit them, nothing would happen. However, remember that VR operates out of perceptual illusions. If your sensory system is showing you an approaching car, then no matter what you might be thinking about it, the safe thing to do is to get out of the way! This is how Place Illusion and Plausibility operate.

So VR is excellent for training because (i) it can make the abstract something tangible. Instead of learning about a complex maneuver  by reading about it, or practicing it in an artificial way, one can actually do it in situ, in a virtual environment. Or for understanding the implications of some complex mathematics, one can use a visualization that involves mobilizing body movements in order to literally grasp it in a concrete way. 

(ii) VR enables doing, not simply observing. Doing engages the whole body in a multisensory way, and the more that the body is engaged the greater the chance for learning and retention. An example of maintenance training involves the learner actually doing so rather than only watching someone else do it or a video.  Or operators can train in facilities before they were ever built Operators can learn the process of making a delivery while maintaining safety for themselves and their customers

(iii) VR is highly suited for training in complex circumstances, where for practical, ethical or safety reasons training in the real site is not possible. Our previous example illustrated this – in that case people were able to train on an installation that did not yet exist. In these complex or dangerous circumstances people can train over and over again, without additional cost of providing materials, for example. Accidents or problems that occur during the virtual training are without physical consequences, but are, of course, ideal for learning. We earlier saw one example of training for fire hazards. But there are complex problems only involving interaction with other humans without complex machinery or installations involved. For example, how do medical doctors learn to deal with intransigent patients? One study showed that doctors faced with unreasonable demands from patients for antibiotics reacted much as they would in reality, and that therefore such environments could be used for ethical and social relationship training. 

 (iv) VR can offer multiple perspectives over the same scenario, which offers also a greater chance of understanding. In VR it is even possible to have a different perspective with respect to yourself as well as experiencing a scenario from different points of view and the training can be collaborative involving several remote participants, or collaboration for bringing experts together from all over the world

(v) VR is excellent for measurement – since everything that the trainees do can, in principle, be recorded and measured: Their overall behaviour, the bodily movements, their physiological and brain responses. The trainee could also re-enter the environment and observe their own recorded data being played out.

Having considered where VR might be good for training we need to also to pay attention to the pitfalls. For example, consider the learning of a sport like table tennis in VR. You could become an excellent VR table tennis player. But as anyone who seriously plays table tennis in reality knows, it is a highly complex skill involving multiple factors associated with every strike of the ball, probably mostly below the conscious awareness of the player. Unless the simulation of table tennis were perfect, it is likely that the skill from virtual table tennis would not translate to real table tennis. Even worse, an already skilled table tennis player might find their skills weakened after playing virtual table tennis because of negative transfer of training. This means that the skilled player may pick up habits that work well in VR but which do not work well in reality. The biggest danger of VR for training is such negative transfer, where people apparently learn something in VR, but where the real world is different enough, perhaps in very subtle ways, that the learning simply does not transfer or makes things worse.

Let’s look at some evidence. Basically if you think that VR is good for training then you will find evidence to support that. On the other hand if you think that it is not good, you will find evidence to support that too.

Winther et al (2020) evaluated VR training for pump maintenance. A comparison was made between VR, and traditional methods of video training and pairwise training where one person helps another to learn. The VR did not have any haptic feedback, whereas with the video and pairwise training trainees could work directly on the real equipment. The sample size was n=36 in a between-groups study with the 3 groups. On almost all measures of training outcome the VR method did not perform as well as the other two methods. This may not be too surprising since the VR method is the only one where trainees could not work directly on the actual machinery. 

Leder (2019) compared VR and PowerPoint instruction for safety training. It was found that VR had no advantage. However, problems with this study were that the VR (in a CAVE) was non-interactive, where trainees seemed to just essentially watch something, rather than be engaged. Also the sample size was small.

Sankaranarayanan et al (2018) compared VR training for a fire in an operating theatre with a control group. VR was found to be superior in learning to successfully put out the fire – 20% of the control group compared to 70% of the VR group were able to complete all the steps correctly one week after the exposure. Note that the sample sizes were quite small (10 in each group) and also it is not clear what the control group actually did. 

Murcia-López el al (2018) compared virtual and physical training for a bimanual assembly task. The VR training seemed not to be different in outcomes compared to the best performing physical condition. The sample size involved (n = 60, over three conditions) was reasonable. Retention of learned skills over time was not particularly good for any of the conditions. 

Gavish (2015) compared VR and AR for training of industrial assembly tasks. The comparisons were VR (with haptics) against a video only control, Augmented Reality working with the real device compared to a control. There were 10 engineers randomly assigned to each group, and the test was to do the real assembly. No difference was found between the VR and the VR-control condition, but AR resulted in less errors overall. However, as usual, look for the cautionary aspects – it is not entirely clear, but it seems that the VR was non-interactive. 

Borsci et al’s 2015 paper reported a meta study of mixed reality (MR) and VR studies for car service maintenance. They only found 8 papers with sufficient rigour to be included in the meta study, and mostly these were on mixed reality – even though hundreds of papers had been published. They concluded:

  • MR systems seemed to be more useful than VR
  • MR/VR resulted in less errors, and required less time for training than previous methods
  • Trainees found the VR/MR more interesting than other methods, with better generalizability
  • VR/MR methods were adaptable to individual expertise, but too much reliance on MR/VR is less effective than on-the-job training.

A number of limitations of the studies were identified:
·      The evaluation studies tend to ignore the organisational setting and organisational needs.
·      They use a limited set of evaluation criteria - too much focussed on time and errors, ignoring: Cybersickness, skill recall and decay, motivation, acceptance, trust, prior attitudes and cognitive skills.

The authors suggested a wider set of evaluation criteria:

·      The effect of and taking account of cognitive skills - including visuospatial abilities-
·      Levels of trust/acceptance of VR/MR tools 
·      Motivation in use by the trainees and trainers
·      Trainee attitudes towards the systems
·      Their previous experience
·      The impact of cybersickness
·      Physiological Reactions – e.g., attention shift, cognitive load, stress
·      Level of presence and engagement
·      Technical aspects and tools features – e.g. effect of designed features, expected and experienced system functioning.

Overall we can conclude that immersive technologies can provide an excellent method for training. It is concrete, it is multisensory involving and engaging the whole body, it can break out of the constraints of reality and give people perspectives that they cannot ever attain in reality, it can be lead to high quality measurement, it is infinitely repeatable, and because of this it is ultimately low cost. 

The cautionary message is that just because of these reasons do not assume that it is going to produce good results. In some circumstances a seemingly unimportant detail that is wrong can lead to negative transfer of training. Before advocating a particular solution for training it must be studied extensively.

Also consider whether the application can be equally well done with other methods (direct traditional teaching, role play, video). Ask yourself why, in this particular application, is VR necessary? Take into account cost, the time scale, the logistics and the feasibility of producing a good enough training scenario without negative transfer. 

Additional Resources

Here are some aspects of training that I have not considered, but which are discussed in this paper to which the section numbers refer:

      VR in surgical training (Section 2.4)
      Navigation rehearsal (Section 2.1.3) (the body is critical)
      Military (rehearsal of scenarios)
      VR in sports (negative transfer considerations critical).
      Model based VR vs 360 video (section 7.2).

Readers might also find the following useful:-

On negative transfer:

On comparative evaluation:

      Bhagavathula et al (2018). "The Reality of Virtual Reality: A Comparison of Pedestrian Behavior in Real and Virtual Environments", in: Proceedings of the Human Factors and Ergonomics Society Annual Meeting: SAGE Publications Sage CA: Los Angeles, CA), 2056-2060.
      Sankaranarayanan et al (2018). Immersive virtual reality-based training improves response in a simulated operating room fire scenario. Surgical endoscopy, 1-11.
      Murcia-Lopez, M., and Steed, A. (2018). A Comparison of Virtual and Physical Training Transfer of Bimanual Assembly Tasks. IEEE transactions on visualization and computer graphics 24, 1574-1583.
      Gavish, N., et al (2015). Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments 23, 778-798. 

Labels: , , ,

12 November, 2017

In the Presence of Jaron Lanier

In the Presence of Jaron Lanier

The talk by Jaron Lanier Friday 11th November 2017 at UCL was attended by about 150 people. Jaron talked about themes from his new book Dawn of the New Everything: A Journey Through Virtual Reality. It is best to read the book of course, but here I will simply say that he introduced the Thesis - that virtual reality offers a fantastic way for the construction of new shared realities, where we can experience each others' dreams. He also introduced the Antithesis that VR offers a perfect means for behavioural modification - since it can act as a 'Skinner box' where a person's environment is completely controlled, and based on feedback in response to their actions and choices algorithms can move them surreptitiously towards new states of mind. The Synthesis ....

Introducing Jaron Lanier

Start of the talk

The talk was very inspiring and, judging by the questions, and the comments afterwards, was very much appreciated and enjoyed.

Here I want to say that it provided the opportunity for previous members of my group, now friends, to get together even if briefly, and renew old ties, and perhaps plan a future gathering. In the pictures below all but Zillah Watson, Aiden O'Neill (and Jaron Lanier) were in the Virtual Environments and Computer Graphics group at UCL at some time. I have indicated those that were postdocs, and PhD students by the year that they obtained their PhD.

Left to right: Daniela Romano (ex postdoc), Vinoba Vinayagamoorthy (2006), Joel Jordan (2010), Marco Gillies (ex postdoc)

Vinoba with Joel

Lee Pip Bull (2006) with Vinoba

Left to right: Zillah Watson, Anthony Steed (1996),
Aiden O'Neill (Penguin Books), Jaron Lanier, 
Mel Slater, Sylvia Xueni Pan (2010 and postdoc)

26 May, 2017

Ghost Presence

While attending WVRF17 I was lucky to experience two immersive VR movie experiences. The first was Through You by Saschka Unseld and Lily Baldwin and the second Defrost: The Virtual Series by Randal Kleiser and Tanna Frederick. They are both 360 degree stereo movies, rendered in Samsung Gear VR. They are both beautifully made and very exciting advances on VR cinematography.

However, here I want to concentrate on the form rather than the content since they are quite different in the kind of presence illusions they evoke. In Through You, you are a witness to a kind of dance that tells the story of a romance over time: what was, and what might have been. In the set up I experienced you physically stand near a wall covered with what felt like a velvet curtain, and were advised to be facing and touching that wall with both hands in order to maintain balance and orientation. Throughout the virtual performance I found that I did keep at least one hand touching the wall, which acted as a kind of anchor. However, the events were taking place anywhere in the scene, so I was often touching the wall even while turned around 180 degrees from my starting position.

In my theory of presence, the basis of the perceptual illusion of ‘being there’ is that the system supports natural sensorimotor contingencies for visual perception. So the extent to which I can use my body and perceive the virtual world in a normal way (head turning, bending down, looking around, reaching out and touching, and seeing wide field of view, with stereo vision, high resolution etc – for vision) the simplest hypothesis for the brain do adopt is “This is where I am”.

In both movies there were head based visual sensorimotor contingencies. The Gear VR does not support full 6 df head-tracking but only head orientation. But since in both movies the viewer is only required to look around while remaining in the same place, this is not very noticeable. So we can say that there were reasonable head-based visual sensorimotor contingencies.

In Through You it was a shock to look down towards myself and realise that I had no body. Our first ever paper in the field of VR (VRAIS, 1993) concentrated on the importance of having a self representation in the virtual environment, a finding that has been supported by more recent results. So even though as you look around, you see the world in 3D stereo, with reasonable field of view, latency and resolution, so that you get approximate natural sensorimotor contingencies, there is still something critically missing in the scene – You. I personally felt a kind of ghostly ‘place illusion’, I was there and not there, some kind of in-between state.

In Through You you are an observer, a witness. The environment does not respond to you in any way. The actors are completely unaware of your presence. One of my proposed bases of the ‘Plausibility Illusion’ is that the virtual world responds to you, i.e., that you can influence what happens, and another is that there are events that relate personally to you. Plausibility refers to the illusion that ‘This is really happening  - now’ (even though you know that this is not the case). This illusion that events are happening is meant in a very personal sense – that the events can impinge on you – that someone may be looking at you, that they can get into your space, that the events may be dangerous or beneficial to you personally. I guess it is something to do with our basic survival instinct, that we continually evaluate surrounding events for possible threat, and that the Plausibility Illusion is concerned with this feeling. Without this personal reality in relation to the ongoing events, you are an observer – very much like you are the observer of a film or stage play. Of course the events can influence your emotions, your appreciation of the aesthetics, you may relate the events to your own life – but the events themselves are not physically connected to you. We cannot call this Ghost Plausibility, since in this type of situation there is no Plausibility at all in the strict sense of the meaning of ‘Plausibility Illusion’ (Psi).

Nevertheless ‘Ghost Place Illusion’ (sensorimotor contingencies but no self representation) and the absence of Psi do give rise to a strange feeling – I will call it Ghost Presence. You are like a Ghost, being in and simultaneously not being in a scenario that unfolds completely independently of your will. From that perspective you can attain a special perspective on the events that you witness. It is somewhat like watching a conventional movie, but your Ghost Presence gives you a familiarity with the space and a closeness to the characters that you cannot get from a movie. It is a different experience, a different art form. This is also superbly illustrated by the prize winning After Solitary by Emblematic. Here you are placed in a virtual replication of a tiny prison cell in which the (real) protagonist spent five years in solitary confinement. You have a first hand experience of the cell, its size, its squalor. From that you can begin to imagine the horror of living 5 years alone (or even 5 hours) in such conditions. The prisoner is there describing what happened to him and his feelings. However, the prisoner is not addressing you personally, he doesn’t look at you or invoke your presence in any way. He is speaking to a general audience. His story is powerful, he is telling it to anyone and everyone but not you specifically. You have the Ghost Presence, where especially the experience of the space is extremely powerful and unforgettable. The monologue of the prisoner provides additional information that also has a powerful emotional impact.

In Defrost you awake after being frozen in liquid nitrogen for 30 years after suffering a stroke (that was incurable at that time). Now the stroke has been cured. A doctor is telling you all this, and adds that due to the fact of your 30 years of sleep you are unable to move, except for your head. When you look down towards yourself you see you are embodied and in a wheel chair. This is 360 degrees and stereo video playback. Since you can look around but in a fixed position the tracking of the Samsung Gear is sufficient to provide reasonable visual sensorimotor contingencies, and there is corresponding Place Illusion. Since you have a body it is not a ghostly PI. The explanation of being unable to move because of the long sleep helps with respect to Plausibility, and a whole series of characters come to talk with you personally including the doctors and (now grown up) children. Over several episodes a drama unfolds in which you play the major role. As well as being a brilliant idea it is very well designed and acted, and a great example of a new kind of first person narrative experience that is going to emerge from VR.

It is important to note that I am not comparing Defrost and Through You from the point of view of their creativity and execution or as experiences, but rather the illusions that they evoke (in me). Through You results in Ghost Presence, which is a very unique experience, being simultaneously there and not there, in the scene and not in it. The contradictions give rise to a kind of tension which is itself a new experience – one that we cannot get at all from any other media. As I said, it is a new kind of art form. Defrost shows how narratives that exploits the PI and Psi illusions  might develop – and especially the conjunction of PI and Psi in the illusion of body ownership.  Through You and Defrost offer different qualities of experience, each created with very high production values, it is not that one is ‘better’ than the other in any way.

When we have used questionnaires to assess presence (e.g., in) we have invariably included the question:

“When you think back about your experience, do you think of the [environment X] more as images that you saw, or more as somewhere that you visited?”

Now when I spend time in a computer graphics model-based VR, my answer to that question has typically a very high score (e.g., 6 or 7 on a 1-7 scale where 1 means ‘images that you saw’ and 7 means ‘somewhere that you visited’). I think back to some environments I have experienced recently, and it is the case I have the feeling to have been somewhere rather than just seen something.

However, when I think back to my experience of Through You and Defrost, my score is pretty low for both, 2/7 – I remember the images, rather than of being somewhere. For Through You this is understandable, because of the Ghost Presence illusion. For Defrost since there were pretty good visual sensorimotor contingencies, I should have the memory of having been somewhere not just seen images. So why is this?

I think this may be to do with being in a virtual world created by video rather than graphics. We are really used to seeing video, it is everywhere throughout our lives. We have never been in world that looks like video. We know that the characters in video have zero information about us. We know that video is something that ‘just plays’ (most of the time) irrespective of anything we do (other than fast forward/back or switch off). In other words video comes with a whole set of very powerful and ingrained expectations. Model based graphics VR does not – we see an environment and virtual characters, that look ‘sort of’ like the real thing, but are not. We don’t know what to expect. Typically there is strong interaction in these worlds whereas there is very minimal interaction with video. So it is possible that the very powerful expectations and associations with video act as a top down dampening of the illusions normally associated with VR. So although during an experience I might have some feeling of ‘being there’ and that the events are ‘really happening’, as soon as I’ve finished, what I remember is the video. I haven’t ‘been there’ but I’ve ‘seen it’.

This is not an Uncanny Valley effect, where because the video is almost but not quite perfect, we reject it. It is an expectations and associations based effect, where cognition to some extent is overriding perception – since we know in a very deep and experiential way what video is, and therefore if this looks like video (a) I can’t be here and (b) this isn’t really happening right now.