Ground Truth in Zurich

And believe it or not, soon after returning from Cyprus I had to go to Zurich. This was to attend a meeting of the European ‘Future Emerging Technologies’ project called IMMERSENCE. The timing was fortunate since my wife also had to give a seminar in Zurich on the same day (yes, she is a scientist). The project mainly consists of experts in ‘haptics’. Haptics is concerned with touch and kinesthetics including force-feedback. Normally of course if you are in a virtual reality and you touch a virtual object you don’t feel it (because nothing is there). Remember that the stereo and head-tracking system may give you the illusion that an object is there right in front of you, but reach out to it, and you will feel nothing. This can (at least temporarily) break presence. Haptics typically employs devices that give back some feeling – normally the force feedback you get on touching something (i.e., when your hand collides with a solid object it does not pass through it) but less often the associated tactile sensation.

My role in the project is to investigate the extent to which presence is induced in a number of paradigmatic application scenarios. In this I work with the researcher Andreas Brogni in the group at Universitat Politècnica de Catalunya in Barcelona.

The idea we follow in this project is to compare responses in physical reality to those in virtual reality. For example, if in physical reality you pick up a cup, you will have applied forces, used your hand and arm in a particular way, felt various tactile sensations, resisted against the weight, and so on, all the way through to having various thoughts and feelings associated with lifting that cup. Now when you carry out the same task in a virtual reality, to what extent do you carry out the same actions, and have the same overall responses to your own actions and to the totality of the event ‘picking up the cup’. Presence in the virtual environment is the extent to which your responses are the same as those in the real environment.

Let’s consider this approach to presence a bit more. It is very practical. Ontological considerations can be avoided. But … many people have said that it does not take into account the beauty of virtual reality which is able to create experiences that are not real. For example, in virtual reality you can ‘fly’, there is no naturally occurring virtual gravity that will hold you at virtual ground level. It is all a matter of what the program allows you to do. For example, it is trivial to program the system such that if you look upwards and press a button on the wand (6 degree of freedom ‘mouse’) that you will typically be holding, then you can fly up in the virtual world. And why not? So can we not talk about presence in unreal situations, but only in relation to simulations of physical reality?

There are two answers to this. The first is that if in simulations that are bound to physical reality we learn how to increase the probability that the participant will respond to virtual events and objects as if they were real, then of course we can apply the same knowledge to non-physically grounded simulations, and increase the chance that people will respond to those types of situations and events as if they were real.

But the real answer is that this approach to presence does not demand at all that the virtual environment depict something that could be real – it is only that the responses to the virtual environment should be real (high presence). In other words if you learn to fly within a virtual reality and you respond to that experience as if what you were doing were real, then that is …. presence. Of course, in such situations the hard part is that there is no ground truth against which to compare – we do not usually fly within physical reality (the plane flies, we are just along for the ride). But what we do in virtual reality is expand the range of experience, to know something about what it would be like to fly - with realistic responses helped along by the exploitation of knowledge we have obtained by studying situations in which there is a ground truth.

Action in Cyprus

Then almost immediately after returning from California I had to go to Cyprus (Limassol) for a conference called ‘Virtual Reality Sofware and Technology’ (VRST 2006). What happened was that it turned out that I was one of the Chairs of this conference – apparently I had agreed to this but forgot. Then I thought – well if I’m Chair I will have to go, so we submitted a short paper.

The paper was about the impact of different types of rendering quality on ‘presence’ in virtual reality. What do I mean by rendering quality? – well in computer graphics you might display the scene just with lines (called ‘wire frame’) or with flat shaded polygons, or with textured polygons … or ultimately with objects that appear to be shaded correctly according to the light distribution in the scene. The latter is very hard to do – and until recently was impossible to do in real-time. But we had a way to do a limited version of this ‘global illumination’ – based on ray tracing - and the question was: what difference would it make?

You might think that the answer is obvious – the more realistic that a scene looks the more likely it is that people will feel themselves to ‘be there’. But actually the evidence to date doesn’t support that idea – there have been experiments that suggest that it makes no difference (google Zimmons rendering quality). The difference with our research was that we not only had a level of global illumination, but one where the shadows and reflections actually move. We had people looking down over a virtual pit – and they would see the shadows and reflections of their virtual body move in correspondence with their real body (well, to some extent, since we only can track their head and one hand). Although this experiment was completed a few months ago, to date I’ve only had time to analyse the questionnaire data – there are many questionnaires that attempt to measure presence and we used one of these. I call the type of presence that you can get from the use of a questionnaire ‘reported presence’. So our study showed that ‘reported presence’ seemed to be greater on average for those people in our experiment who experienced the pit room with the ray tracing rendering method (which has shadows and reflections) compared to those who experienced a method of rendering that did not include second order lighting effects (i.e., shadows and reflections). It seemed to make a difference. But is it only because there is dynamic feedback, or is it because it simply looks more realistic, or both of these? We have to wait for the next experiment.

But if there’s ‘reported presence’ is there also … unreported presence? Why ‘reported presence’ and not just ‘presence’? Well actually there is ‘unreported presence’! For example, more than 10 years ago when I was at Queen Mary College and we had one of the first types of Head-Mounted Display we had some London firefighters in for a demonstration. The environment we showed them was the pit room. There were two of them, and when they noticed the virtual pit, each of them visibly shook, and stepped back – they had a very strong behavioural response to what they were seeing. But … when we asked them afterwards about their experience (informal ‘reported presence’) in particular whether they had any sense of ‘being there’ the replies were … ‘Oh no mate, felt nothing, no nothing happened’. Their words contradicted what we saw with our own eyes.
Presence as far as I’m concerned is quite uninteresting if it is limited simply to what people will tell you about their experience after the event – no matter how well ‘validated’ the questionnaire; and you can’t ask them during the event because that in itself could destroy the experience. So presence is to do with how people respond to events and correspondingly how they are able to act within a virtual reality (or even in a mixed reality) – it is their response and activity in relation to virtual sensory data (how exactly it is produced is not important – but if you want to think of it as computer generated on computer controlled displays that is fine).

Obtaining their answers to some questions after they have completed the experience is ok, and can be useful. But what is more important is how they respond at many different levels: their physiological responses (e.g., heart rate and heart rate variability, temperature, electrodermal activity – skin sweat response – even measures of brain activity such as EEG). Then there is their automatic behaviour – like do they duck automatically if a virtual object flies towards their face? Then there is deliberate behaviour – like they decide to pick something up and look at it. Then there are thoughts ‘I wonder what’s going on over there?’ and emotions…. Basically the total response of the person to the situation and unfolding events. (And this is why the ‘pit room’ provides such a useful scenario for testing people’s responses within a virtual reality – since we know the types of response that people would normally have in reality).

So in virtual (or mixed reality) we substitute (or augment) real sensory data with virtual sensory data. The question is whether people respond to this as if it were real.

And … obviously there are different levels of this ‘presence’ – since as we saw in the firefighters example, they may exhibit high ‘presence’ in one field (automatic behavioural in that case) but low ‘presence’ in another (reported presence). High presence means that they are ‘high’ across the board.

Presence - the view from Marina del Rey

During the last week of October I was at the Institute of Creative Technology at University of Southern California in Los Angeles (Marina del Rey). The purpose was a meeting of invited people only to discuss presence in virtual reality, but where there was a mix of computer scientists, human factors people, engineers, artists and people involved in computer games. The meeting was interesting, and a great idea, but also I found it very frustrating - although there was a great view outside my hotel room.

This concept of ‘presence’ has been around for a long time – it started with ‘telepresence’ in the context of teleoperator systems (people controlling remote robots) and then shifted to ‘presence’ when virtual reality came into vogue in the late 1980s early 1990s. This is not going to be an academic article so I’m not going to provide any references – take a look on ‘google scholar’ and look up ‘presence’ ‘Durlach’ ‘Held’ ‘Sheridan’ ‘Loomis’ for some of the early discussions.

Basically presence has been thought of as ‘the sense of being there’ – some people when they go into a virtual reality experience a shift of their sense of place, so that they feel themselves to be in the place simulated by the VR rather than in the physical place in which they actually are.

This is the approach that I used also for many years, and it was useful for a while. But, how do you find out if someone has a ‘sense of being there’? The only way of course is to ask them, and that leads to the use of questionnaires and interviews. Anyway, over the years there have been a large number of papers devoted to this issue, there is an International Society for Presence Research (ISPR – don’t confuse it with the International Society for Paranormal Research), and several groups across the world who contribute to the ‘presence’ literature.

But…..

After all these years, a large part of the meeting at ICT was devoted to discussing ‘what presence really is?’ searching for the ultimate ‘definition’ of presence.

When you hear this kind of thing in discussion of a research subject, you know that the game is already lost.

How can there be ‘presence research’ when no two people seem to agree what they are talking about?

In spite of all the definitions of ‘presence’ - what presence ‘really is’ is equivalent to what ‘presence researchers’ do to measure it. Someone might define presence to be X, but if you look at what they do it may or may not have some relation with X. I was very frustrated during this meeting and the fruitless discussions about ‘definition’ – not too useful or exciting decades after people first started researching into this subject.

So I decided to start this ‘blog’. In this blog I’m going to try to explain some of my ideas about this phenomenon, but more importantly what I try to do to define it in practice – as I said what presence really is for any researcher is exactly equivalent to how they attempt to ‘measure’ it. So what I do is more important than what I say.