Ghost Presence
While attending WVRF17 I was lucky to experience two immersive VR movie experiences. The first was Through You by Saschka Unseld and Lily Baldwin and the second Defrost: The Virtual Series by Randal Kleiser and Tanna Frederick. They are both 360 degree stereo movies, rendered in Samsung Gear VR. They are both beautifully made and very exciting advances on VR cinematography.
However, here I want to concentrate on the form rather than the content since they are quite different in the kind of presence illusions they evoke. In Through You, you are a witness to a kind of dance that tells the story of a romance over time: what was, and what might have been. In the set up I experienced you physically stand near a wall covered with what felt like a velvet curtain, and were advised to be facing and touching that wall with both hands in order to maintain balance and orientation. Throughout the virtual performance I found that I did keep at least one hand touching the wall, which acted as a kind of anchor. However, the events were taking place anywhere in the scene, so I was often touching the wall even while turned around 180 degrees from my starting position.
In my theory of presence, the basis of the perceptual illusion of ‘being there’ is that the system supports natural sensorimotor contingencies for visual perception. So the extent to which I can use my body and perceive the virtual world in a normal way (head turning, bending down, looking around, reaching out and touching, and seeing wide field of view, with stereo vision, high resolution etc – for vision) the simplest hypothesis for the brain do adopt is “This is where I am”.
In both movies there were head based visual sensorimotor contingencies. The Gear VR does not support full 6 df head-tracking but only head orientation. But since in both movies the viewer is only required to look around while remaining in the same place, this is not very noticeable. So we can say that there were reasonable head-based visual sensorimotor contingencies.
In Through You it was a shock to look down towards myself and realise that I had no body. Our first ever paper in the field of VR (VRAIS, 1993) concentrated on the importance of having a self representation in the virtual environment, a finding that has been supported by more recent results. So even though as you look around, you see the world in 3D stereo, with reasonable field of view, latency and resolution, so that you get approximate natural sensorimotor contingencies, there is still something critically missing in the scene – You. I personally felt a kind of ghostly ‘place illusion’, I was there and not there, some kind of in-between state.
In Through You you are an observer, a witness. The environment does not respond to you in any way. The actors are completely unaware of your presence. One of my proposed bases of the ‘Plausibility Illusion’ is that the virtual world responds to you, i.e., that you can influence what happens, and another is that there are events that relate personally to you. Plausibility refers to the illusion that ‘This is really happening - now’ (even though you know that this is not the case). This illusion that events are happening is meant in a very personal sense – that the events can impinge on you – that someone may be looking at you, that they can get into your space, that the events may be dangerous or beneficial to you personally. I guess it is something to do with our basic survival instinct, that we continually evaluate surrounding events for possible threat, and that the Plausibility Illusion is concerned with this feeling. Without this personal reality in relation to the ongoing events, you are an observer – very much like you are the observer of a film or stage play. Of course the events can influence your emotions, your appreciation of the aesthetics, you may relate the events to your own life – but the events themselves are not physically connected to you. We cannot call this Ghost Plausibility, since in this type of situation there is no Plausibility at all in the strict sense of the meaning of ‘Plausibility Illusion’ (Psi).
Nevertheless ‘Ghost Place Illusion’ (sensorimotor contingencies but no self representation) and the absence of Psi do give rise to a strange feeling – I will call it Ghost Presence. You are like a Ghost, being in and simultaneously not being in a scenario that unfolds completely independently of your will. From that perspective you can attain a special perspective on the events that you witness. It is somewhat like watching a conventional movie, but your Ghost Presence gives you a familiarity with the space and a closeness to the characters that you cannot get from a movie. It is a different experience, a different art form. This is also superbly illustrated by the prize winning After Solitary by Emblematic. Here you are placed in a virtual replication of a tiny prison cell in which the (real) protagonist spent five years in solitary confinement. You have a first hand experience of the cell, its size, its squalor. From that you can begin to imagine the horror of living 5 years alone (or even 5 hours) in such conditions. The prisoner is there describing what happened to him and his feelings. However, the prisoner is not addressing you personally, he doesn’t look at you or invoke your presence in any way. He is speaking to a general audience. His story is powerful, he is telling it to anyone and everyone but not you specifically. You have the Ghost Presence, where especially the experience of the space is extremely powerful and unforgettable. The monologue of the prisoner provides additional information that also has a powerful emotional impact.
In Defrost you awake after being frozen in liquid nitrogen for 30 years after suffering a stroke (that was incurable at that time). Now the stroke has been cured. A doctor is telling you all this, and adds that due to the fact of your 30 years of sleep you are unable to move, except for your head. When you look down towards yourself you see you are embodied and in a wheel chair. This is 360 degrees and stereo video playback. Since you can look around but in a fixed position the tracking of the Samsung Gear is sufficient to provide reasonable visual sensorimotor contingencies, and there is corresponding Place Illusion. Since you have a body it is not a ghostly PI. The explanation of being unable to move because of the long sleep helps with respect to Plausibility, and a whole series of characters come to talk with you personally including the doctors and (now grown up) children. Over several episodes a drama unfolds in which you play the major role. As well as being a brilliant idea it is very well designed and acted, and a great example of a new kind of first person narrative experience that is going to emerge from VR.
It is important to note that I am not comparing Defrost and Through You from the point of view of their creativity and execution or as experiences, but rather the illusions that they evoke (in me). Through You results in Ghost Presence, which is a very unique experience, being simultaneously there and not there, in the scene and not in it. The contradictions give rise to a kind of tension which is itself a new experience – one that we cannot get at all from any other media. As I said, it is a new kind of art form. Defrost shows how narratives that exploits the PI and Psi illusions might develop – and especially the conjunction of PI and Psi in the illusion of body ownership. Through You and Defrost offer different qualities of experience, each created with very high production values, it is not that one is ‘better’ than the other in any way.
When we have used questionnaires to assess presence (e.g., in) we have invariably included the question:
“When you think back about your experience, do you think of the [environment X] more as images that you saw, or more as somewhere that you visited?”
Now when I spend time in a computer graphics model-based VR, my answer to that question has typically a very high score (e.g., 6 or 7 on a 1-7 scale where 1 means ‘images that you saw’ and 7 means ‘somewhere that you visited’). I think back to some environments I have experienced recently, and it is the case I have the feeling to have been somewhere rather than just seen something.
However, when I think back to my experience of Through You and Defrost, my score is pretty low for both, 2/7 – I remember the images, rather than of being somewhere. For Through You this is understandable, because of the Ghost Presence illusion. For Defrost since there were pretty good visual sensorimotor contingencies, I should have the memory of having been somewhere not just seen images. So why is this?
I think this may be to do with being in a virtual world created by video rather than graphics. We are really used to seeing video, it is everywhere throughout our lives. We have never been in world that looks like video. We know that the characters in video have zero information about us. We know that video is something that ‘just plays’ (most of the time) irrespective of anything we do (other than fast forward/back or switch off). In other words video comes with a whole set of very powerful and ingrained expectations. Model based graphics VR does not – we see an environment and virtual characters, that look ‘sort of’ like the real thing, but are not. We don’t know what to expect. Typically there is strong interaction in these worlds whereas there is very minimal interaction with video. So it is possible that the very powerful expectations and associations with video act as a top down dampening of the illusions normally associated with VR. So although during an experience I might have some feeling of ‘being there’ and that the events are ‘really happening’, as soon as I’ve finished, what I remember is the video. I haven’t ‘been there’ but I’ve ‘seen it’.
This is not an Uncanny Valley effect, where because the video is almost but not quite perfect, we reject it. It is an expectations and associations based effect, where cognition to some extent is overriding perception – since we know in a very deep and experiential way what video is, and therefore if this looks like video (a) I can’t be here and (b) this isn’t really happening right now.
However, here I want to concentrate on the form rather than the content since they are quite different in the kind of presence illusions they evoke. In Through You, you are a witness to a kind of dance that tells the story of a romance over time: what was, and what might have been. In the set up I experienced you physically stand near a wall covered with what felt like a velvet curtain, and were advised to be facing and touching that wall with both hands in order to maintain balance and orientation. Throughout the virtual performance I found that I did keep at least one hand touching the wall, which acted as a kind of anchor. However, the events were taking place anywhere in the scene, so I was often touching the wall even while turned around 180 degrees from my starting position.
In my theory of presence, the basis of the perceptual illusion of ‘being there’ is that the system supports natural sensorimotor contingencies for visual perception. So the extent to which I can use my body and perceive the virtual world in a normal way (head turning, bending down, looking around, reaching out and touching, and seeing wide field of view, with stereo vision, high resolution etc – for vision) the simplest hypothesis for the brain do adopt is “This is where I am”.
In both movies there were head based visual sensorimotor contingencies. The Gear VR does not support full 6 df head-tracking but only head orientation. But since in both movies the viewer is only required to look around while remaining in the same place, this is not very noticeable. So we can say that there were reasonable head-based visual sensorimotor contingencies.
In Through You it was a shock to look down towards myself and realise that I had no body. Our first ever paper in the field of VR (VRAIS, 1993) concentrated on the importance of having a self representation in the virtual environment, a finding that has been supported by more recent results. So even though as you look around, you see the world in 3D stereo, with reasonable field of view, latency and resolution, so that you get approximate natural sensorimotor contingencies, there is still something critically missing in the scene – You. I personally felt a kind of ghostly ‘place illusion’, I was there and not there, some kind of in-between state.
In Through You you are an observer, a witness. The environment does not respond to you in any way. The actors are completely unaware of your presence. One of my proposed bases of the ‘Plausibility Illusion’ is that the virtual world responds to you, i.e., that you can influence what happens, and another is that there are events that relate personally to you. Plausibility refers to the illusion that ‘This is really happening - now’ (even though you know that this is not the case). This illusion that events are happening is meant in a very personal sense – that the events can impinge on you – that someone may be looking at you, that they can get into your space, that the events may be dangerous or beneficial to you personally. I guess it is something to do with our basic survival instinct, that we continually evaluate surrounding events for possible threat, and that the Plausibility Illusion is concerned with this feeling. Without this personal reality in relation to the ongoing events, you are an observer – very much like you are the observer of a film or stage play. Of course the events can influence your emotions, your appreciation of the aesthetics, you may relate the events to your own life – but the events themselves are not physically connected to you. We cannot call this Ghost Plausibility, since in this type of situation there is no Plausibility at all in the strict sense of the meaning of ‘Plausibility Illusion’ (Psi).
Nevertheless ‘Ghost Place Illusion’ (sensorimotor contingencies but no self representation) and the absence of Psi do give rise to a strange feeling – I will call it Ghost Presence. You are like a Ghost, being in and simultaneously not being in a scenario that unfolds completely independently of your will. From that perspective you can attain a special perspective on the events that you witness. It is somewhat like watching a conventional movie, but your Ghost Presence gives you a familiarity with the space and a closeness to the characters that you cannot get from a movie. It is a different experience, a different art form. This is also superbly illustrated by the prize winning After Solitary by Emblematic. Here you are placed in a virtual replication of a tiny prison cell in which the (real) protagonist spent five years in solitary confinement. You have a first hand experience of the cell, its size, its squalor. From that you can begin to imagine the horror of living 5 years alone (or even 5 hours) in such conditions. The prisoner is there describing what happened to him and his feelings. However, the prisoner is not addressing you personally, he doesn’t look at you or invoke your presence in any way. He is speaking to a general audience. His story is powerful, he is telling it to anyone and everyone but not you specifically. You have the Ghost Presence, where especially the experience of the space is extremely powerful and unforgettable. The monologue of the prisoner provides additional information that also has a powerful emotional impact.
In Defrost you awake after being frozen in liquid nitrogen for 30 years after suffering a stroke (that was incurable at that time). Now the stroke has been cured. A doctor is telling you all this, and adds that due to the fact of your 30 years of sleep you are unable to move, except for your head. When you look down towards yourself you see you are embodied and in a wheel chair. This is 360 degrees and stereo video playback. Since you can look around but in a fixed position the tracking of the Samsung Gear is sufficient to provide reasonable visual sensorimotor contingencies, and there is corresponding Place Illusion. Since you have a body it is not a ghostly PI. The explanation of being unable to move because of the long sleep helps with respect to Plausibility, and a whole series of characters come to talk with you personally including the doctors and (now grown up) children. Over several episodes a drama unfolds in which you play the major role. As well as being a brilliant idea it is very well designed and acted, and a great example of a new kind of first person narrative experience that is going to emerge from VR.
It is important to note that I am not comparing Defrost and Through You from the point of view of their creativity and execution or as experiences, but rather the illusions that they evoke (in me). Through You results in Ghost Presence, which is a very unique experience, being simultaneously there and not there, in the scene and not in it. The contradictions give rise to a kind of tension which is itself a new experience – one that we cannot get at all from any other media. As I said, it is a new kind of art form. Defrost shows how narratives that exploits the PI and Psi illusions might develop – and especially the conjunction of PI and Psi in the illusion of body ownership. Through You and Defrost offer different qualities of experience, each created with very high production values, it is not that one is ‘better’ than the other in any way.
When we have used questionnaires to assess presence (e.g., in) we have invariably included the question:
“When you think back about your experience, do you think of the [environment X] more as images that you saw, or more as somewhere that you visited?”
Now when I spend time in a computer graphics model-based VR, my answer to that question has typically a very high score (e.g., 6 or 7 on a 1-7 scale where 1 means ‘images that you saw’ and 7 means ‘somewhere that you visited’). I think back to some environments I have experienced recently, and it is the case I have the feeling to have been somewhere rather than just seen something.
However, when I think back to my experience of Through You and Defrost, my score is pretty low for both, 2/7 – I remember the images, rather than of being somewhere. For Through You this is understandable, because of the Ghost Presence illusion. For Defrost since there were pretty good visual sensorimotor contingencies, I should have the memory of having been somewhere not just seen images. So why is this?
I think this may be to do with being in a virtual world created by video rather than graphics. We are really used to seeing video, it is everywhere throughout our lives. We have never been in world that looks like video. We know that the characters in video have zero information about us. We know that video is something that ‘just plays’ (most of the time) irrespective of anything we do (other than fast forward/back or switch off). In other words video comes with a whole set of very powerful and ingrained expectations. Model based graphics VR does not – we see an environment and virtual characters, that look ‘sort of’ like the real thing, but are not. We don’t know what to expect. Typically there is strong interaction in these worlds whereas there is very minimal interaction with video. So it is possible that the very powerful expectations and associations with video act as a top down dampening of the illusions normally associated with VR. So although during an experience I might have some feeling of ‘being there’ and that the events are ‘really happening’, as soon as I’ve finished, what I remember is the video. I haven’t ‘been there’ but I’ve ‘seen it’.
This is not an Uncanny Valley effect, where because the video is almost but not quite perfect, we reject it. It is an expectations and associations based effect, where cognition to some extent is overriding perception – since we know in a very deep and experiential way what video is, and therefore if this looks like video (a) I can’t be here and (b) this isn’t really happening right now.
posted by Mel Slater | 12:33 pm
0 Comments:
Post a Comment
<< Home