1  Introduction to Immersive Media

1.1 Defining Immersive Technologies and XR

Immersive technologies enable us to interact with digital content in natural, intuitive ways, often blurring the line between the physical and virtual worlds. At its core is the concept of “presence” - the feeling of being physically present in a non-physical world. This sense of presence is a key factor in creating compelling and effective immersive experiences.

The primary goal of immersive media is to create experiences where our bodies and senses interact with digital environments as naturally as they do with the physical world.

My fundamental approach to immersive media centers on a simple principle: the goal is to get the brain to work “as if” in a real environment.

This embodied interaction can be so compelling that users may instinctively react to and interact with virtual objects as if they were physical. This is the basis for the potential of VR in training, education and engaging entertainment, as well as rich and natural interaction and communication in all kinds of applications, including social virtual worlds and professional meeting environments.

VR user engaged in a virtual environment, showing delight.

Immersive media encompasses a spectrum of technologies known as the virtuality continuum, ranging from our familiar physical reality to fully immersive virtual environments. More about this spectrum and related concepts in 2.1 The Reality-Virtuality Continuum, but the key common terms are, briefly:

  1. Augmented Reality (AR): Augments the real world by overlaying digital information or objects onto it.

  2. Mixed Reality (MR): Mixing the real and virtual, in different ways. (More later).

  3. Virtual Reality (VR): Immerses users in a completely virtual environment, allowing for full-body interactions in a digital space.

XR is most commonly used as an umbrella term to cover all of these technologies. As they have significant overlap the same people are often interested in and working across these boundaries and we often want to talk about them as one type of technology - XR.

The power of immersive media lies in its ability to engage our senses and leverage our innate understanding of spatial relationships and physical interactions. By doing so, it creates more intuitive and engaging ways to interact with digital content, whether for entertainment, education, training, or professional applications.

One of the key advantages of immersive technologies is their ability to make complex digital interactions accessible to a wide range of users, including those who may not be familiar with traditional computer interfaces. For instance:

  • In VR, users can navigate 3D spaces by simply walking and moving your body, leaning in, bending down, rather than learning complex keyboard and mouse controls.
  • AR applications can overlay intuitive visual instructions onto real-world objects, making tasks like assembly or repair more straightforward for novices.
  • Hand-based interactions can allow users to manipulate digital objects as if they were physical, reaching out to touch or grab, reducing the learning curve for 3D modeling or data visualization tasks.

VR users moving around freely in a larger space.

This natural interaction paradigm opens up new possibilities for digital content creation and consumption, potentially democratizing access to complex digital tools and experiences. As immersive technologies continue to evolve, they promise to make digital interactions more intuitive, engaging, and accessible to people of all backgrounds and technical skill levels.

1.2 The Evolution of Immersive Technologies

The journey of immersive technologies, particularly Virtual Reality (VR), spans several decades, with roots tracing back to the 1960s. This evolution has been marked by technological breakthroughs, changing applications, and an expanding vision of what’s possible in virtual spaces.

1.2.1 Early Experiments

  • 1960s: Ivan Sutherland’s “Sword of Damocles” laid the groundwork for modern VR systems. This early prototype, while bulky and limited, demonstrated the potential for head-mounted displays in creating immersive experiences.

  • 1980s-1990s: VR saw periods of hype and subsequent disillusionment as the technology struggled to meet expectations. Despite limitations in graphics and processing power, this era saw the development of many foundational concepts in VR.

1.2.2 Early Applications

For many years, VR remained primarily confined to specialized contexts.

Back when I first took a VR course in the late 1990s, VR was primarily used in expensive and specialized contexts—research laboratories and high-end simulators.

The main motivations for using VR traditionally were:

  1. Training scenarios that are:
    • Expensive
    • Dangerous
    • Impossible to recreate in reality
  2. Flight simulators
  3. Prototyping
  4. Visualization

Example of a flight simulator, one of the early applications of VR technology.

These early applications laid the groundwork for more widespread use of VR technology. For instance, the EasyADL project demonstrated how VR could be used for prototyping and testing scenarios that couldn’t be replicated in reality due to sensor limitations.

In this project, I could create virtual sensors easily that would send information when the user grabbed something, when something left a cupboard, or when it was placed on the stove. I could easily create these virtual sensors to pass that information to the systems that were actually needed to implement the algorithms.

1.2.3 The Rise of Commercial VR

The modern era of commercial VR began to take shape in the mid-2010s, marking a significant shift that brought VR technology to a broader consumer audience. This period saw the development of more affordable and accessible VR hardware, setting the stage for widespread adoption.

1.2.4 Parallel Evolution of AR

While VR was developing, Augmented Reality (AR) was also making significant strides. The concept of overlaying digital information onto the real world began to gain traction, with early applications in heads-up displays for military aircraft and later in consumer mobile devices.

1.2.5 The Convergence of Technologies

As both VR and AR technologies advanced, the lines between them began to blur, giving rise to the concept of Mixed Reality (MR) and the broader field of Extended Reality (XR). This convergence has been driven by improvements in:

  • Display technology
  • Motion tracking
  • Computer vision
  • Processing power

These advancements have enabled more seamless blending of virtual and real-world elements, opening up new possibilities for immersive experiences. With recent hardware AR and MR is starting to catch up, but VR is still the more mature technology.

1.2.6 The Impact of Mobile Technology

The widespread adoption of smartphones and tablets has played a crucial role in the evolution of immersive technologies, particularly AR. The ubiquity of powerful, sensor-rich mobile devices has made AR experiences accessible to a wide audience, paving the way for applications in gaming, education, and various industries.

1.2.7 Looking Forward

The evolution of immersive technologies continues at a rapid pace, with ongoing developments in areas such as:

  • Haptic feedback
  • Eye tracking
  • Brain-computer interfaces
  • Volumetric displays

These advancements promise to further enhance the realism and interactivity of immersive experiences, potentially revolutionizing how we interact with digital content and each other in virtual spaces.

As we move forward, the challenge lies in harnessing these technologies to create meaningful, engaging, and accessible experiences that can benefit various aspects of human life, from entertainment and education to healthcare and professional training.

1.3 Understanding Presence and Immersion

Presence is a fundamental concept in immersive media, particularly in virtual reality (VR). It refers to the psychological state of feeling as if you are actually (“bodily”) present in a virtual environment, despite knowing that you are not physically there.

1.3.1 Defining Presence

Michael Abrash, now chief science officer at Meta since initially working with VR at Valve, emphasized the importance of presence.

Presence is: - Why we’re excited - Unique to VR - The key to VR’s success

A slide from an early presentation on VR from Michael Abrash, then at Valve, now Chief Science Officer at Meta, on the importance of presence in VR. (Abrash 2014)

Presence is more than just visual immersion; it’s about convincing the brain that the virtual environment is real.

As I’ve emphasized, the goal is to get the brain to work ‘as if’ in a real environment.

1.3.2 The Power of Presence

When presence is achieved, users may react to virtual stimuli as if they were real.

At one time, I was testing a VR environment for several hours, lifting containers and pouring virtual liquids from one to another. I’d been seated at a virtual table, completely absorbed in the task. When my hands grew tired, I instinctively tried to rest them on the virtual table surface.

My hands passed straight through, and I felt a genuine chill down my spine. This reaction showed how thoroughly my brain had accepted the virtual environment as real—even knowing it was virtual couldn’t prevent the instinctive response.

This anecdote illustrates how powerful the sense of presence can be, causing instinctive reactions to virtual objects.

Examples of users interacting with virtual objects as if they were real.

1.3.3 Factors Contributing to Presence

Several factors contribute to creating a sense of presence. The most important are:

  1. Interaction: The ability to interact with the virtual environment in natural, intuitive ways strengthens the sense of presence.

  2. Consistency: The virtual world must behave consistently and predictably to maintain the illusion of reality.

  3. Body Ownership: Seeing a virtual representation of your body that moves as you do can significantly enhance presence.

These can also contribute significantly, but are not as critical:

  1. Visual Fidelity: High-resolution displays and realistic graphics help convince the brain of the environment’s reality.

  2. Audio: Spatial audio that accurately represents the virtual space enhances immersion.

1.3.4 The McGurk effect

The McGurk effect is a perceptual phenomenon that demonstrates how our brains integrate visual and auditory information to create a coherent perception of speech. This effect is crucial for understanding how presence is created in VR environments.

In the McGurk effect:

  1. A person is shown a video of someone saying one sound (e.g., “ga-ga”)
  2. The audio is replaced with a different sound (e.g., “ba-ba”)
  3. The viewer often perceives a third, intermediate sound (e.g., “da-da”)

This illusion occurs because the brain attempts to reconcile the conflicting visual and auditory information, resulting in a perception that matches neither the visual nor the auditory input alone.

The McGurk Effect - Auditory Visual Illusion
The McGurk Effect - Auditory Visual Illusion
This classic psychological demonstration shows how our brain processes conflicting visual and auditory information. In the video, you'll see a person saying one sound while hearing a different sound, yet perceiving a third, intermediate sound. This illusion is particularly relevant to VR because it persists even when you're aware of it, demonstrating how our perception can be reliably manipulated through coordinated sensory inputs.
Watch on YouTube

What makes the McGurk effect particularly relevant to VR is its persistence even when the viewer is aware of the illusion.

This is a kind of illusion that is not dependent on you being aware of it. It works even though you know about it.

This demonstrates how our perceptual systems can be influenced by multisensory input, even overriding our conscious knowledge. In VR, this principle is leveraged to create a sense of presence by providing coherent multisensory experiences that our brains interpret as real, even when we know we’re in a virtual environment.

1.3.5 Understanding Different Types of Immersion

Immersion in virtual reality can be understood in two distinct but complementary ways: technological immersion and narrative immersion. Both contribute to the overall sense of presence in virtual environments but operate through different mechanisms.

1.3.5.1 Technological Immersion

Technological immersion refers to the degree to which our senses are enveloped by the virtual environment. This form of immersion is primarily achieved through hardware and software capabilities that block out the physical world and replace it with virtual stimuli. Key aspects include:

  1. Visual coverage (field of view)
  2. Audio spatialization
  3. Haptic feedback
  4. Motion tracking accuracy

The more complete this sensory replacement, the higher the level of technological immersion.

1.3.5.2 Narrative Immersion

While technological immersion focuses on sensory engagement, narrative immersion relates to our psychological engagement with the virtual environment’s story, context, and unfolding events. This type of immersion can occur even with relatively simple technology if the narrative elements are compelling.

As you explore these kinds of environments, they build up stories and narratives, which fit very well with how we generally remember things and understand the world.

The interplay between these two types of immersion can create powerful experiences in virtual environments. Later chapters will explore how narrative immersion can be particularly effective in visualization and educational applications, where engagement with content often matters more than technological sophistication.

1.3.6 The Future of Presence and Immersion

As VR technology continues to advance, we can expect even more convincing experiences of presence. Future developments may include:

  • Improved haptic feedback for more realistic touch sensations
  • Advanced eye-tracking for more natural visual experiences
  • Brain-computer interfaces for direct neural engagement

Understanding and enhancing presence and immersion remain central goals in the development of immersive technologies, driving innovation in hardware, software, and content creation.

1.4 Avatars and Virtual Embodiment

Virtual avatars and the concept of embodiment play crucial roles in immersive experiences, particularly in virtual reality (VR). They significantly impact how users perceive themselves and interact within virtual environments. This section explores the foundational psychological principles underlying virtual embodiment, while practical implementations and technical avatar creation systems are covered in Section 5.9.

1.4.1 The Rubber Hand Illusion

The concept of embodiment in virtual reality is closely related to the psychological phenomenon known as the rubber hand illusion. This illusion, first demonstrated by Botvinick and Cohen in 1998, shows how easily our brains can be tricked into accepting an artificial limb as part of our body.

In the classic rubber hand experiment: 1. A participant’s real hand is hidden from view. 2. A rubber hand is placed in a visible position. 3. Both the real hand and the rubber hand are stroked simultaneously with a brush. 4. After a short time, many participants begin to feel as if the rubber hand is their own.

Rubber Hand Illusion Experiment
Rubber Hand Illusion Experiment
This classic psychological experiment demonstrates how easily our brains can be tricked into accepting an artificial limb as part of our body. The video shows the experimental setup where participants' real hands are hidden while they observe a rubber hand being stroked simultaneously with their hidden hand. This illusion forms the foundation for understanding virtual embodiment in VR environments, showing how coordinated visual and tactile stimuli can manipulate our sense of body ownership.
Watch on YouTube

This illusion demonstrates the brain’s remarkable plasticity in constructing our sense of body ownership. It reveals that our perception of our body is not fixed, but can be manipulated through coordinated visual and tactile stimuli.

In VR, this effect is leveraged to create a sense of ownership over virtual body parts or entire avatars. When users see virtual hands moving in sync with their real hand movements, they quickly begin to feel as if those virtual hands are their own. This principle extends beyond just hands - entire virtual bodies can be embodied in VR, leading to powerful immersive experiences. ### The Illusion of Body Ownership

Research conducted at the Karolinska Institute’s Brain, Body and Self Laboratory (Group Ehrsson) has demonstrated how easily our brains can be tricked into accepting a virtual or artificial body as our own. This phenomenon is closely related to the sense of presence in VR environments.

A striking example comes from an experiment where a participant wearing a VR headset had a strong reaction when a knife was brought near their virtual body.

  • I viewed the manikins body as being my body. - Then researchers slid a knife across the dummy’s body. - You have the reaction, because she’s not, it’s not like she’s trying to stab me with it, but you do have the reaction to just sort of pull away a little bit because she, I mean, it really seems like she’s about to put something sharp against my stomach.

- Quote from video

This visceral response occurs even though the participant is consciously aware they are looking at a mannequin body through a VR headset.

Body Ownership Illusion in VR
Body Ownership Illusion in VR
This video demonstrates how VR can create powerful illusions of body ownership, showing participants' visceral reactions to virtual bodies being threatened or touched. The demonstration reveals how quickly our brains can accept virtual bodies as our own, even when we're consciously aware we're looking at a mannequin through a VR headset. This research forms the basis for understanding avatar embodiment in virtual environments.
Watch on YouTube

1.4.2 Flexibility of Body Perception

Our perception of our own body is surprisingly flexible and can be manipulated in VR environments. A study titled “Being Barbie: The Size of One’s Own Body Determines the Perceived Size of the World” demonstrated how inhabiting different sized virtual bodies can alter our perception of the world around us.

In this study, participants were asked to estimate how big a cube was. You can see that they were now using their doll bodies as reference points when showing with their own hands how big the cube was. The actual cube, of course, was not that big.

This flexibility extends beyond just size. Researchers have experimented with altering various aspects of virtual bodies, including:

  • Body shape
  • Sex
  • Race

Participants estimating object sizes while embodying different sized avatars.

1.4.3 Psychological Impact of Avatar Embodiment

The way we perceive our virtual bodies can have profound effects on our psychology and behavior. A study by Peck et al. (2013) explored how embodying an avatar with dark skin color could affect implicit racial bias.

If you get this experience of being in a virtual reality environment and having a darkly colored body, then that affects how much racial bias you have when evaluated before and after the experience.

Study on embodying avatars of different races.

The implications of this research extend far beyond just racial attitudes.

Depending on how you perceive yourself, you might even become better at math, for instance. This can really affect how you see the world, what you believe yourself to be able to do, and what judgments you are making.

1.4.3.1 The Proteus Effect

The Proteus Effect demonstrates how users may alter their behavior to conform to expectations set by their avatar’s appearance. For example, users with taller avatars may negotiate more aggressively in virtual environments.

1.4.4 Implications for VR Design

Understanding the rubber hand illusion and its extensions in VR has significant implications for VR design:

  1. Avatar Design: Avatars that move in sync with user movements enhance the sense of embodiment.
  2. Interaction Design: Designing interactions that reinforce the connection between user actions and avatar responses can strengthen the illusion of body ownership.
  3. Emotional Responses: Designers can leverage this illusion to create more emotionally impactful experiences, as users may react to virtual stimuli as if they were real.
  4. Therapeutic Applications: The body ownership illusion in VR has potential applications in therapy, such as treating phantom limb pain or body dysmorphia.

Ethical Considerations: The power to manipulate users’ sense of body ownership and self-perception carries significant ethical responsibilities. Research showing that virtual embodiment can influence implicit bias, cognitive performance, and self-concept demonstrates both the potential and the risks of these technologies. Questions of psychological safety, age-appropriate design, and the long-term impacts of virtual embodiment require careful consideration. We explore these ethical implications in depth in Chapter 9, examining how developers can create transformative experiences while respecting user wellbeing and autonomy.

By understanding and leveraging these principles, VR designers can create more immersive, engaging, and potentially transformative experiences. The rubber hand illusion and its VR extensions demonstrate the power of our brains to adapt to new body schemas, opening up exciting possibilities for virtual embodiment and presence in digital worlds.

1.4.5 Future Directions

As VR technology advances, we can expect more sophisticated avatar systems that provide even more convincing experiences of embodiment. This could lead to applications in:

  • Therapy and mental health treatment
  • Education and skill training
  • Social interaction and collaboration in virtual spaces

The study of avatars and virtual embodiment remains a rich area for research, with potential implications for our understanding of consciousness, self-perception, and social interaction.

For those interested in exploring this topic further, the Brain, Body and Self Laboratory publications page offers a wealth of research papers on related experiments and findings.

1.5 The Future of Immersive Technology

As immersive technologies continue to evolve at a rapid pace, the future of this field promises exciting developments and widespread impact across various sectors. This section explores potential advancements and implications for the future of immersive media.

1.5.2 Potential Applications

While detailed discussions of applications will be covered in later chapters, some key areas of interest for future immersive technology include:

  • Education and Training
  • Healthcare and Therapy
  • Design and Engineering
  • Entertainment and Gaming
  • Remote Work and Collaboration
  • Tourism and Cultural Experiences

1.5.3 Challenges and Considerations

As immersive technologies become more prevalent, several challenges will need to be addressed:

  1. Privacy and Data Security: Ensuring the protection of personal data in increasingly connected immersive environments.

  2. Ethical Implications: Addressing the psychological effects of prolonged immersion and the potential for manipulation in virtual environments.

  3. Accessibility: Ensuring that immersive technologies are inclusive and accessible to all, including those with disabilities.

  4. Content Creation: Developing tools and platforms that allow for efficient creation of high-quality immersive content.

1.5.4 Skills and Opportunities

The growing importance of immersive technologies is creating new career opportunities across various fields. As the technology evolves, there will be an increasing demand for professionals skilled in:

  • Immersive experience design
  • 3D modeling and animation
  • Spatial computing
  • XR development
  • Immersive storytelling

1.5.5 Conclusion

The future of immersive media is bright and full of potential. As these technologies continue to advance and integrate more seamlessly into our lives, they have the power to transform how we work, learn, entertain ourselves, and interact with the world around us. While challenges remain, the opportunities for innovation and positive impact are vast.

Note: Detailed discussions of hardware evolution and specific applications will be covered in subsequent chapters of this compendium.

1.6 Further Reading

Chapter 1 introduced the fundamental concepts of immersive media, including virtual reality (VR), augmented reality (AR), and mixed reality (MR). We explored the importance of presence and immersion in creating compelling XR experiences, and discussed the potential applications of these technologies across various fields. To deepen your understanding of these foundational concepts and their implications, consider the following resources:

1.6.1 Research Papers

  • Sjölie, D., & Badylak, S. (2019). Mind tricks for presence. In Proceedings of the 14th International Conference on the Foundations of Digital Games (FDG ’19). Association for Computing Machinery, New York, NY, USA, Article 47, 1–7.
    • This paper introduces concepts of synchronized reality and grounded simulation as starting points for designing mixed reality systems with optimal presence, providing case studies of commercial VR applications.
  • Sjölie, D. (2012). Presence and general principles of brain function. Interacting with Computers, 24(4), 193-202.
    • This paper explores the relationship between presence in virtual environments and general principles of brain function, providing foundational insights into the cognitive aspects of immersive experiences.
  • Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535), 3549-3557.
    • This paper introduces the concepts of place illusion and plausibility illusion as key components of presence in virtual environments, explaining how these factors can lead to realistic behavior in immersive VR.
  • Sanchez-Vives, M. V., & Slater, M. (2005). From presence to consciousness through virtual reality. Nature Reviews Neuroscience, 6(4), 332-339. This paper provides an overview of presence in virtual reality and its relationship to consciousness.
    • This review paper explores the concept of presence in immersive virtual environments, discussing how the sense of “being there” is signaled by people acting and responding realistically to virtual situations and events, and its relationship to consciousness.