2  XR Technologies and the Reality-Virtuality Continuum

2.1 The Reality-Virtuality Continuum

The reality-virtuality continuum is a conceptual framework that describes the full spectrum of experiences ranging from the completely real to the fully virtual. This continuum provides a basis for understanding different types of mixed reality experiences and how they blend elements of the physical and digital worlds.

2.1.1 Understanding the Continuum

At one end of the spectrum lies our familiar physical reality - the tangible world we interact with daily. On the opposite end, we find fully immersive virtual environments, such as fantastical realms like the Lord of the Rings universe. Between these extremes, there exists a range of mixed reality experiences:

Virtuality Continuum

I find it helpful to think about the spectrum in terms of mixing approaches. At the augmented reality end, you still feel essentially present in your physical reality, but you can add digital and virtual elements to that world. At the other end, you can feel like you’re on a virtual moon base—something completely removed from your actual physical reality—but where you can still add elements from the real world, such as your physical desk or a view of someone else in the room.

Key concepts mapped onto the spectrum are:

1. Real Reality (RR): Your everyday, familiar, physical environment.

2. Augmented Reality (AR): Closer to the physical reality end, AR enhances our perception of the real world by overlaying digital information or objects onto it. For example, a user might see virtual navigation arrows overlaid on real city streets. Users can interact with virtual elements while still feeling present in and aware of their physical surroundings.

3. Mixed Reality (MR): Can include everything between RR to VR, with all possible mixings of the real and the virtual. In the central part of the continuum, MR creates environments where physical and digital objects coexist and interact in real time. E.g., a virtual character that can interact with real-world objects in your living room, an ordinary table turning into an interactive touch display surface or the view out through a window changing to a view onto Mars.

4. Augmented Virtuality (AV): Augmenting your virtual world with elements from physical reality. E.g., bringing real people from your physical surroundings into your personal virtual office. Or making sure you can see and use your physical mouse and keyboard while in VR. Mostly an academic concept, not used much in the industry.

5. Virtual Reality (VR): At the far end of the continuum, VR immerses users in a completely synthetic environment. This could be as fantastical as exploring a virtual Hogwarts castle or as practical as a fully simulated surgical training environment. Allows for full-body interaction in a digital space. Unlimited by the physical space, although care is required when moving around (navigating) in VR.

The reality-virtuality continuum is not a rigid classification but rather a fluid spectrum. Many experiences blur the lines between these categories. For instance, a virtual moon base experience might incorporate a view of your real-world desk, creating a hybrid environment that combines elements from different points on the continuum.

Understanding this continuum is crucial for conceptualizing how different technologies can blend the physical and digital worlds. It opens up new possibilities for creating immersive experiences that can range from subtle augmentations of our physical reality to complete transportation into virtual worlds, with countless variations in between.

As we explore each category in more depth in the following sections, we’ll examine how different technologies and applications leverage various points along this continuum to create unique and engaging experiences.

2.2 Virtual Reality (VR) Systems

Virtual Reality (VR) systems represent the fully immersive end of the reality-virtuality continuum, providing users with complete digital environments that replace their physical surroundings. This section explores the components, capabilities, and evolution of modern VR systems.

2.2.1 Key Components of VR Systems

Modern VR systems typically consist of several key components:

1. Head-Mounted Display (HMD): The primary interface between the user and the virtual world.
2. Motion Controllers: Allow users to interact with the virtual environment.
3. Tracking System: Monitors the user’s movements and position in space.
4. Computer or Console: Generates the virtual environment and processes user interactions.

2.2.2 Brief History of VR

While VR has seen a significant resurgence in recent years, its roots trace back to the 1960s. For decades, VR remained primarily confined to specialized contexts.

The basic motivations for using VR remain fundamentally the same as when I first took a VR course in the late 1990s: training scenarios where it’s expensive, dangerous, or impossible to practice in reality. This core principle has remained consistent even as the technology has advanced dramatically.

Early applications included flight simulators, military training, and scientific visualization. However, limitations in technology and high costs restricted widespread adoption.

2.2.3 The Modern Era of VR (2016 onwards)

The modern era of commercial VR began in 2016 with the release of several key headsets:

1. HTC Vive: Developed by Valve and manufactured by HTC
2. Oculus Rift: The first version from Oculus, now owned by Facebook
3. Microsoft’s Windows Mixed Reality headsets: A cheaper alternative
4. Mobile VR solutions: Samsung Gear VR and Google Daydream
5. PlayStation VR: Sony’s offering for the PlayStation console

HTC Vive

Oculus Rift

Figure 2.1: The first consumer oriented VR headsets, from 2016, HTC Vive and Oculus Rift.

Image Attribution: - HTC Vive: “CES2016_HTCVive_Pre_Winters” by ETC-USC is licensed under CC BY 2.0 - Oculus Rift: Image by KniBaron from Bangkok, Thailand, licensed under CC BY 2.0

These headsets ranged in price from around 4,000 to 10,000 crowns. The mobile VR solutions, while innovative, have largely been discontinued due to the friction involved in using a smartphone as the display.

2.2.4 Recent Developments in XR Hardware

The XR landscape has continued to evolve rapidly:

1. Meta Quest 3(S): A fully mobile, standalone headset that offers 6DoF (six degrees of freedom) tracking without the need for external sensors or a connected PC. The Quest 3 is the better version with advanced optics and comfort, while Quest 3S is the newest, using the same chip and capable of running the same applications but otherwise optimized to be as cheap as possible.

2. Valve Index: Developed as a high-end PC-connected headset focusing on wide field of view, comfort, sounds and advanced controller design. The Valve Index is one of the older headsets still in use, as few newer headsets are developed specifically for PC-desktop-VR.

3. Bigscreen Beyond 2: A 107 g, custom-fit SteamVR headset with built-in eye tracking and optional comfort kits for longer sessions.(Bigscreen, Inc. 2024) Like the original Beyond, it relies on external base stations and controllers, but the new model shows how aggressively PCVR rigs can shrink when every gram is optimized around a single wearer’s face.

4. Pico 4: The most direct competitor to Meta Quest 3(S), with similar features. Also standalone. Includes enterprise versions with eye tracking etc.

5. Samsung Galaxy XR (“Project Moohan”): Samsung’s first Android XR headset blends 4K-per-eye micro-OLED displays, Snapdragon XR2+ Gen 2 compute, and comprehensive inside-out tracking while leaning on Google’s Android XR stack for software continuity across phones, tablets, and wearables.(West 2025)

6. Apple Vision Pro: A “prosumer”, early-adpoter, headset from Apple, providing high quality visuals and mixed reality with hand and eye tracking for a hefty price.

7. Varjo XR-4: An ultra-high-resolution headset aimed at professional applications, includes mixed reality with professional hand and eye tracking.

8. Meta Ray-Ban Display + Neural Band: Lightweight smart glasses with a monocular HUD paired with an sEMG wristband for precise text input, showcasing how glasses-class wearables are adopting richer interactions typically found in headsets.(Meta Platforms 2024)

2.2.5 The Form-Factor Spectrum (2025)

To understand how these devices relate to one another, it helps to visualize them along a spectrum that runs from fully tethered PCVR rigs to lightweight smart glasses. Each rung of the spectrum balances compute location, interaction methods, and ergonomics differently.

Segment	Representative Hardware	Primary Compute	Core Interactions	Typical Use Cases
PCVR	Valve Index, Bigscreen Beyond 2 (107 g custom-fit)(Bigscreen, Inc. 2024)	External PC with discrete GPU	Controllers with precise SteamVR tracking, optional finger sensing	Simulation labs, enthusiasts seeking fidelity
Performance Standalone	Meta Quest 3/3S, Pico 4	On-headset mobile SoC	Controllers, hand tracking, mixed reality passthrough	Consumer gaming, productivity MR
Premium Standalone MR	Apple Vision Pro, Samsung Galaxy XR	On-headset mobile SoC with co-processors	Eye and hand tracking, voice, spatial video passthrough	Prosumer productivity, enterprise collaboration
Split Compute / Tethered Ultra-light	Meta’s reported ultralight headset + compute puck	Waist or pocket compute puck	Eye tracking with gaze-and-pinch, sEMG bands	Long-session media viewing, travel-ready VR
Smart Glasses & HUD	Meta Ray-Ban Display + Neural Band, Samsung/Google HUD glasses	Paired phone/cloud	Wrist-based sEMG, voice, subtle gestures	Assisted reality, navigation, AI companions

The newest announcements are compressing weight and thermal budgets without giving up expressiveness. Meta’s reported ultralight puck-based headset aims to deliver sub-110 gram eyewear with gaze-and-pinch as the primary modality, while Bigscreen Beyond 2 demonstrates how far miniaturized optics and custom fitting can push PCVR hardware.(Heaney 2024; Bigscreen, Inc. 2024) Samsung’s Galaxy XR shows how manufacturers are pairing high-end micro-OLED optics with Android XR to compete head-on with Apple’s Vision Pro in the standalone MR tier, while Google positions Gemini-powered Android XR services as the connective AI tissue across both the headset and future eyewear.(West 2025; Google LLC 2024b, 2024a) On the glasses end of the spectrum, Meta’s Ray-Ban Display (69 g) pairs a monocular HUD with the Neural Band sEMG wristband for precise finger-driven text input, signaling that lightweight devices can still support fine-grained control.(Meta Platforms 2024)

Android XR as a cross-device layer

Google’s 2025 Android XR announcement frames a single software stack that spans headsets and glasses, with Gemini multimodal AI handling perception, assistant behavior, and cross-device continuity.(Google LLC 2024b, 2024a) Treat Android XR as connective tissue between the spectrum segments: developers can target one platform while deploying to puck-driven headsets, standalone devices, or future smart glasses, tapping Gemini for scene understanding, translation, and generative overlays.

2.2.5.1 Smart-Glasses Market Snapshot

• Display strategies: Meta opts for a high-brightness monocular HUD in Ray-Ban Display, while Samsung and Google preview a waveguide HUD slated for a 2026 launch to complement their Android XR headset push.(Meta Platforms 2024; Heaney 2025)
• Input diversity: sEMG wristbands, gaze-and-pinch, and always-listening voice agents are emerging to replace touchpad-only interaction.
• Battery and thermals: 6-hour eyewear runtimes and IP-rated accessories hint at expectations for all-day wear, but compute still regularly offloads to phones or pucks.
• Ecosystem bets: Meta ties hardware to Meta AI services, while Google positions Android XR as the open alternative, weaving Gemini agents into Samsung’s hardware roadmap and the broader Android developer base.(Meta Platforms 2024; Google LLC 2024b, 2024a; Heaney 2025)

The spectrum perspective helps frame design decisions in later chapters: interaction designers can map capabilities to expected hardware constraints, while developers can plan how features degrade gracefully across different compute locations.

Varjo VR-1

Prices start around 4,000 SEK for the Meta Quest 3S, with different configurations of consumer headsets up to around 10,000 SEK. Apple Vision Pro is not yet available in Sweden but costs 4,000 EUR elsewhere in Europe. The Varjo XR-3, aimed at professional users, is even more expensive.

💰 Professional-Grade Hardware

Varjo, a Finnish company, produces exceptionally impressive high-end virtual reality headsets. While these systems cost 50,000 SEK and above—sometimes reaching over 100,000 SEK—they offer exceptionally high resolution that enables users to see fine details in virtual environments that would be impossible with consumer-grade hardware.

There are more options out there, including HTC who where the first to release consumer VR and are still in the game but not as popular these days.

2.2.6 Key Technical Aspects

2.2.6.1 Field of View (FOV)

Field of View refers to the extent of the observable environment at any given time. It’s typically measured in degrees:

• Most consumer VR headsets offer an FOV between 90 and 110 degrees.

• Some high-end or experimental designs push this further but it has not really taken hold.

• Some high-end or experimental designs push this further but it has not really taken hold.

I know how much ya’ll love field-of-view and want more. I’m with you. I like it. I get it, I do. The tradeoffs are so bad. The tradeoffs on weight, form factor, compute, thermals… it’s all bad,

- Quote from Meta CTO Andrew Bosworth

Meta Explains Why It Sees Wide Field-of-View Headsets as a ‘bad tradeoff’

2.2.6.2 Resolution

Higher resolution displays provide clearer, more detailed images, enhancing realism and reducing the “screen door effect”. Modern high-end headsets offer resolutions exceeding 2000x2000 pixels per eye.

2.2.6.3 Tracking and Degrees of Freedom

• 3DoF (3 Degrees of Freedom): Only tracks rotational movement (looking left/right, up/down, and tilting head)
  • This was somewhat common in the cheapest headsets in the beginning of commercial VR, but is now completely outdated.
• 6DoF (6 Degrees of Freedom): Tracks both rotational and positional movement (including moving forward/backward, left/right, and up/down)

There are two main approaches to tracking in VR:

1. Inside-out tracking:
   • Cameras on the HMD look out into the world
   • More portable and easier to set up in new locations
   • This is the dominating approach with recent headsets like the Meta Quest 3(S) and the Apple Vision Pro.
2. Outside-in tracking:
   • Cameras or sensors in the room track the user
   • Requires more setup but can be more precise
   • Headsets using versions of this tracking include Valve Index and Bigscreen Beyond.

2.2.7 Conclusion

VR systems have evolved dramatically in recent years, becoming more accessible, powerful, and versatile. From high-end professional systems to consumer-friendly standalone headsets, VR technology is finding applications in diverse fields such as entertainment, education, training, and professional visualization. As these systems continue to advance, we can expect even more immersive and realistic virtual experiences in the future.

2.3 Augmented Reality (AR) Technologies

AR technology overlays digital content onto our view of the real world, enhancing our perception and interaction with our surroundings. This section explores the devices, software, and applications that make AR possible, focusing on current technologies and their applications.

2.3.1 Types of AR Systems

AR systems can be broadly categorized into three types:

1. Mobile AR: Uses smartphones or tablets as the AR device.
2. Head-Mounted Displays (HMDs): Wearable devices that provide a see-through display.
3. Projection-based AR: Projects digital information directly onto physical objects or surfaces.

2.3.2 Mobile Augmented Reality

Mobile AR is currently the most accessible and widely used form of AR technology. Major tech companies like Google and Apple have released development toolkits (ARCore and ARKit respectively) that enable AR experiences on smartphones. More recently companies such as Snapchat and Niantic has made significant plays at AR.

Mobile AR Demonstration - Real-time Object Placement

This demonstration showcases the core capabilities of mobile augmented reality using smartphones. Viewers will see how AR applications can detect surfaces and place virtual objects in real-world environments, illustrating the fundamental tracking and rendering technologies that make mobile AR possible. The video demonstrates the accessibility of AR technology that most viewers can experience on their own devices today.

Watch on YouTube

Most smartphones today have the capability to run mobile AR applications where you can view the camera feed and add virtual content to the real world. This technology has become remarkably accessible, with most users having these capabilities right in their pocket.

Mobile AR applications typically use the device’s camera to view the real world and then overlay digital content onto this view. This technology has found applications in various fields, including:

1. Gaming (e.g., Pokémon Go)
2. Navigation and wayfinding
3. Education and training
4. Marketing and advertising

2.3.3 Advanced AR: HoloLens and Beyond

Moving beyond mobile AR, we encounter more sophisticated AR devices like Microsoft’s HoloLens. These devices offer a more immersive and hands-free AR experience.

Microsoft HoloLens AR Demonstration - Spatial Computing

This video demonstrates Microsoft's HoloLens capabilities, showing advanced AR features including spatial mapping, gesture-based interaction, and see-through holographic displays. Viewers will observe how the HoloLens creates immersive mixed reality experiences with 3D holograms that appear to interact with the physical environment, highlighting both the potential and current limitations of head-mounted AR displays, including the restricted field of view that constrains where virtual content can appear.

Watch on YouTube

Key features of advanced AR headsets include:

1. See-through displays
2. Spatial mapping and understanding
3. Natural gesture-based interactions
4. Voice commands

However, current limitations include a restricted field of view for virtual content.

Current AR headsets face significant field of view limitations. In many demonstrations, you can see that virtual augmentations are only visible within a restricted rectangular area in the center of the user’s vision, rather than across their full field of view.

2.3.4 Industrial Augmented Reality

One of the most promising applications of AR technology is in industrial settings. Industrial applications represent one area where AR technology can be genuinely useful with current capabilities. This practical utility explains why Microsoft focused their later HoloLens releases on enterprise markets rather than consumer sales.

Industrial AR Application - Manufacturing and Assembly

This video showcases practical industrial applications of augmented reality in manufacturing and assembly environments. Viewers will see how AR technology provides real-time guidance and information overlay for workers, demonstrating assembly instructions, quality control procedures, and maintenance protocols. The demonstration illustrates why Microsoft focused HoloLens development on enterprise applications, showing how AR can improve efficiency, reduce errors, and enhance worker safety in industrial settings where the technology provides immediate practical value.

Watch on YouTube

Industrial AR applications include:

1. Assembly line assistance
2. Maintenance and repair guidance
3. Quality control
4. Training and skill development

These applications can significantly improve efficiency, reduce errors, and enhance worker safety in various industrial settings.

2.3.5 Recent Developments in AR

The AR hardware landscape has seen significant advancements in 2024, with major tech companies pushing the boundaries of what’s possible in wearable AR devices. However, we have also seen Microsoft discontinuing the Hololens line of products, as they seem to increasingly focus on software in the XR space.

2.3.5.1 Meta Orion

Meta’s Orion AR glasses prototype, showcased in September 2024, represents a significant leap forward in AR technology. Key features include:

• 70-degree field of view, substantially wider than competitors
• Advanced silicon-carbide lenses and micro LED projectors
• Resolution of 13 pixels per degree (with plans to increase to 26)
• Full sensor suite including:
  • Eye tracking
  • Hand tracking
  • Room tracking cameras
• Wireless neural wristband for input
• Separate wireless processor puck for computing
• Capability to display multiple apps simultaneously
• AI integration for enhanced experiences

Please note that while Orion shows impressive technical achievements, it remains a prototype focused on development and testing rather than consumer release.

2.3.5.2 Snap Spectacles (5th Generation)

Snap’s latest iteration of AR glasses, released in September 2024, offers:

• 47-degree field of view
• Gesture control capabilities
• AR effects and entertainment features
• Developer-focused distribution ($99/month developer kit)
• More robust but bulkier design compared to previous versions

While more limited in capabilities compared to Meta’s Orion, Snap’s approach focuses on practical, entertainment-oriented AR applications.

2.3.5.3 Future Outlook

As AR technology continues to mature, we’re seeing a trend toward: - Smaller, more efficient form factors - Enhanced display technologies - More sophisticated input methods - Stronger integration with AI and spatial computing

While true consumer AR glasses may still be several years away, recent developments from companies like Meta and Snap demonstrate significant progress toward making immersive AR a practical reality.

Meta’s Ray-Ban smart glasses, while not offering AR display capabilities, hint at another potential track toward mainstream adoption. By focusing on a fashionable form factor and integrating AI-powered voice commands and camera features, these glasses demonstrate how companies might bridge the gap between current technology limitations and consumer expectations. This “walk before you run” approach of creating socially acceptable smart glasses could help pave the way for eventual AR integration once display technology catches up with our ambitions for all-day wearable AR. The success or failure of such interim products may provide valuable insights into how AR glasses will need to evolve to achieve widespread adoption.

2.3.6 The Future of AR: Towards Ubiquitous Computing

Industry leaders like Mark Zuckerberg envision a future where AR becomes an integral part of our daily lives. The goal is to develop unobtrusive AR devices that look and feel like normal glasses or even contact lenses.

Mark Zuckerberg on the Future of Augmented Reality

In this presentation, Meta's CEO Mark Zuckerberg outlines his vision for the future of augmented reality and its potential to transform daily life. Viewers will hear about the long-term goal of creating unobtrusive AR devices that could replace many physical objects with digital alternatives, making tools and experiences more accessible and affordable. The discussion explores how AR glasses might eventually become as common as smartphones, fundamentally changing how we interact with digital information and potentially reducing our dependence on physical manufacturing.

Watch on YouTube

Zuckerberg highlights the potential impact.

Think about how many of the things that we have in our lives actually don’t need to be physical. They can be digital and think about how much better and more affordable and accessible they’re going to be when they are.

- Quote from Mark Zuckerberg

This vision suggests a future where many physical objects could be replaced by virtual counterparts, potentially leading to:

1. Reduced manufacturing and environmental impact
2. Increased accessibility to various tools and experiences
3. More affordable alternatives to expensive physical products

2.3.7 Conclusion

Augmented Reality technology is rapidly evolving, offering new ways to blend digital information with our physical world. From mobile AR applications to sophisticated headsets and industrial solutions, AR is finding its place in various sectors of our lives and work. As these technologies continue to advance, we can expect to see even more seamless integration of digital and physical realities, potentially transforming how we interact with information and our environment on a daily basis.

2.4 Mixed Reality (MR) and Hybrid Systems

Mixed Reality (MR) represents a significant portion of the reality-virtuality continuum, blending elements of both physical and digital worlds to create new environments where physical and virtual objects coexist and interact in real time. In recent years, MR capabilities have become increasingly central to mainstream XR devices, with major manufacturers integrating MR features into their flagship headsets. This shift represents a growing recognition that the ability to seamlessly blend virtual content with the physical world is crucial for the future of immersive computing.

Where early MR systems were specialized devices focused solely on augmented or mixed reality, modern XR headsets increasingly incorporate robust MR capabilities alongside their VR features. This trend towards integrated MR reflects a broader understanding that users need to be able to smoothly transition between fully virtual experiences and mixed reality interactions without changing devices.

Meta emphasizes this blending of realities in their latest headsets through features like full-color passthrough, which allows users to see their physical environment in high fidelity while interacting with virtual objects. Their vision for MR includes practical applications like virtual workspaces where multiple virtual screens can coexist with physical desks and keyboards, or fitness applications where virtual instructors can guide users through real-world workouts.

Apple’s Vision Pro takes this concept further with what they term “spatial computing,” where virtual content is deeply integrated with the physical space. Their approach includes features like the ability to scale and position virtual screens anywhere in the physical environment, eye and hand tracking for natural interaction with virtual elements, and the ability to adjust the blend between virtual and physical reality using a digital crown. Applications range from immersive FaceTime calls where participants appear life-size in your physical space to 3D movies that seem to extend your room into virtual environments.

This convergence of virtual and physical realities in mainstream devices signals a significant evolution in how we think about mixed reality - no longer as a separate technology, but as a fundamental feature of modern XR experiences.

2.4.1 Varjo XR-4: Enterprise Mixed Reality

Varjo has been at the cutting edge of mixed reality technology for a number of years, starting out with the XR-1 onto the latest XR-4. The Varjo XR headsets combine high-resolution virtual reality capabilities with advanced camera systems for seamless integration of real and virtual environments. Below is a video from a XR-1 demonstration. It has only improved since then.

Varjo XR-1 Mixed Reality Demonstration

This demonstration showcases Varjo's high-end XR-1 headset capabilities, featuring seamless blending of real and virtual environments through advanced camera systems and ultra-high resolution displays. Viewers will see practical applications including automotive design scenarios where virtual car interiors can be overlaid onto real vehicles, demonstrating the precision and quality possible with professional-grade mixed reality systems. The video illustrates how enterprise XR applications can achieve remarkable realism and practical utility in specialized industries.

Watch on YouTube

Key features of the Varjo XR-4 include:

1. High-resolution VR display
2. Low-latency, high-quality cameras
3. Seamless blending of real and virtual environments

One notable application of the Varjo XR headsets is in automotive design and testing.

The Varjo system demonstrates remarkable mixed reality capabilities—the cameras are fast and high-quality enough that users can safely drive while wearing the VR headset, seeing reality entirely through the display. The system can then overlay virtual elements, such as completely changing a car’s interior design in real-time. This level of seamless reality substitution is quite impressive.

2.4.2 Substitutional Reality

Substitutional reality is an innovative approach that blends live camera feeds with pre-recorded 360-degree video content, allowing for seamless transitions between real-time and pre-recorded experiences.

Substitutional Reality - Blending Live and Recorded 360° Video

This fascinating demonstration shows substitutional reality technology that seamlessly blends live camera feeds with pre-recorded 360-degree video content. Viewers will see how users can be imperceptibly switched between real-time and recorded experiences, enabling complex scenarios where people appear to leave and re-enter rooms from impossible directions. The technology showcases innovative approaches to manipulating temporal and spatial reality in mixed reality environments, opening possibilities for storytelling and experience design that blur the boundaries between live and recorded content.

Watch on YouTube

The system works as follows: 1. Users wear a VR headset equipped with a pass-through camera, initially showing them the real world. 2. The system can switch to a pre-recorded 360-degree video without the user noticing. 3. This enables complex scenarios, such as a person seemingly leaving the room and re-entering from a different direction.

2.4.3 The Void: Physical Props in Virtual Worlds

The Void represents a culmination of mixed reality concepts, creating large-scale experiences that combine physical environments with virtual reality.

The Void - Large-Scale Mixed Reality Entertainment

This video provides an overview of The Void's groundbreaking approach to mixed reality entertainment, which combines virtual reality headsets with elaborate physical environments. Viewers will see how The Void creates immersive experiences by building real physical spaces that correspond to virtual worlds, incorporating touchable props, surfaces, and environmental effects that align with virtual objects. The demonstration showcases techniques borrowed from magic and illusion to create convincing mixed reality experiences where users can physically interact with virtual environments on a room-scale level.

Watch on YouTube

Key aspects of The Void’s approach include: 1. Building physical environments that correspond to virtual landscapes 2. Incorporating touchable props and surfaces that align with virtual objects 3. Leveraging techniques from magic and illusion to direct attention and create convincing experiences

2.4.4 Non-photorealistic AR: Blurring the Lines

An intriguing approach to augmented reality involves processing the video feed of the real world to make it appear more like virtual content.

This approach involves processing the real-world video feed to make reality appear more like virtual content, creating a stylized aesthetic that makes it difficult to distinguish between real and virtual elements. This technique challenges traditional assumptions about how we present mixed reality experiences.

Non-Photorealistic AR - Stylized Reality Processing

This intriguing demonstration shows an innovative approach to augmented reality where the real-world video feed is processed to appear more like virtual content, creating a stylized visual aesthetic. Viewers will see how reality itself can be transformed to match virtual elements, making it difficult to distinguish between real and virtual objects in the mixed reality experience. This technique challenges traditional assumptions about AR presentation and opens new possibilities for creative expression and artistic applications in mixed reality environments where visual coherence takes precedence over photorealism.

Watch on YouTube

This approach challenges our assumptions about the nature of reality in mixed reality experiences and opens up new possibilities for creative expression and immersive storytelling.

2.4.5 Spatial Understanding

Modern MR systems have evolved to include sophisticated spatial understanding capabilities. These systems can: - Automatically detect and map physical spaces - Identify common objects like furniture and surfaces - Create persistent digital overlays that maintain their position in physical space - Enable shared experiences where multiple users can interact with the same virtual content anchored in physical space

This environmental understanding enables applications like: - Virtual workspace setups where digital screens persist between sessions - Intelligent object placement that respects physical surfaces and obstacles - Shared MR experiences where multiple users can collaborate in the same augmented space - Adaptive interfaces that adjust based on the available physical space

2.4.6 Digital Twins: Replicating Real Environments in VR

One way that spatial understanding can be used is to create complete digital replicas of real-world spaces, known as digital twins. While basic spatial understanding allows MR systems to detect and map spaces in real-time, digital twins take this concept further by creating precise virtual replicas that can be used for planning, simulation, and prototyping. This concept was demonstrated in a project where a VR application was matched to a physical apartment.

This project created a VR application matched to a precise digital twin of a real apartment. Users could walk around the virtual space, sit on the sofa, and interact with various elements using the Oculus Quest’s hand tracking to control virtual cockpit interfaces that corresponded to real-world furniture and spatial layouts.

Digital Twin VR Environment - Physical Space Replication

This demonstration showcases a VR application that precisely replicates a real apartment as a digital twin, allowing users to interact with virtual controls and interfaces while navigating a space that matches their physical environment. Viewers will see how the Oculus Quest's hand tracking capabilities enable interaction with virtual cockpit-style controls that correspond to real-world furniture and space layouts. This approach demonstrates how digital twins can be used for prototyping AR designs within VR environments, creating unique hybrid experiences that blend VR immersion with real-world spatial awareness.

Watch on YouTube

Among other things, this approach allows for prototyping AR designs within a VR environment, offering a unique blend of VR and AR concepts.Digital twins demonstrate how sophisticated spatial understanding can enable not just real-time interaction with virtual content, but long-term planning and modification of spaces through persistent digital replicas.

2.4.7 Conclusion

Mixed Reality and hybrid systems represent the cutting edge of immersive technologies, blending the physical and digital worlds in increasingly sophisticated ways. From high-end systems like the Varjo XR-4 and the Apple Vision Pro to creative approaches like substitutional reality and non-photorealistic AR, these technologies are pushing the boundaries of what’s possible in immersive experiences. The mainstreaming of MR features in consumer devices has accelerated development across the field, from sophisticated spatial understanding to digital twins, while specialized applications continue to explore new possibilities in specific domains.

As MR technologies continue to evolve, we can expect to see even more innovative applications that challenge our perceptions of reality and open up new possibilities for interaction, entertainment, education, and professional applications. The future of mixed reality promises to create increasingly seamless and intuitive ways of blending our physical and digital worlds, driven by both broad consumer adoption and specialized enterprise innovation.

2.5 Emerging XR Technologies

As the field of extended reality (XR) continues to evolve, new technologies are constantly emerging that push the boundaries of what’s possible in immersive experiences. This section explores some of the cutting-edge developments in XR, focusing on haptics, advanced tracking systems, and other innovative approaches to enhancing immersion and interaction.

2.5.1 Advanced Haptic Technologies

Haptic feedback represents one of the most promising emerging areas in XR technology, with researchers developing innovative ways to provide tactile sensations in virtual environments. These advancements include haptic gloves, advanced force feedback systems, and novel approaches like ultrasonic haptics that create tactile sensations in mid-air.

For a comprehensive exploration of haptic technologies, implementation techniques, and design best practices, see Section 5.7.

Additionally, emerging techniques like Galvanic Vestibular Stimulation, which address motion sickness through vestibular system manipulation, are covered in Section 5.4.7.

2.5.2 Advanced Tracking and Interaction Systems

2.5.2.1 Project Northstar: Pushing the Boundaries of AR

Project Northstar, developed by Ultraleap (formerly LEAP Motion), represents an exploratory venture into the future of augmented reality. While not a commercially available product, it offers open-source instructions for enthusiasts to build their own prototypes.

Project Northstar - Advanced Hand Tracking AR

This demonstration showcases Project Northstar, an open-source AR headset prototype developed by Ultraleap that pushes the boundaries of hand tracking and interaction in augmented reality. Viewers will see advanced hand tracking technology that enables natural interaction with virtual objects, including hand augmentations and novel gesture-based controls. The project explores innovative ways to interact with completely virtual objects using physical hand movements in real space, demonstrating how advanced tracking can create compelling AR experiences where virtual content responds naturally to hand gestures and movements.

Watch on YouTube

Key features of Project Northstar include:

1. Advanced hand tracking
2. Augmentations to hands for enhanced interaction
3. Exploration of novel ways to interact with virtual objects in physical space

Ultraleap’s particular strength lies in hand tracking technology and hand augmentations, exploring innovative interaction methods. This enables users to interact with completely virtual objects using natural body movements within their physical reality, creating compelling hybrid experiences.

For comprehensive coverage of hand tracking and gesture recognition technologies, implementation techniques, and best practices, see Section 5.8.

2.5.2.2 Redirected Walking and the Unlimited Corridor

Redirected walking is a technique that allows users to explore seemingly infinite virtual spaces within a limited physical area. This is achieved by subtly manipulating the user’s perception of movement.

Redirected Walking - The Unlimited Corridor

This impressive demonstration shows redirected walking technology that allows users to explore seemingly infinite virtual spaces within a limited physical area by subtly manipulating their perception of movement. Viewers will see how a cleverly designed physical wall structure can fool users into believing they are walking along an infinitely long, straight corridor when they are actually walking in a curved path. This technique demonstrates advanced spatial manipulation in VR, enabling large-scale virtual exploration in small physical spaces by exploiting human perceptual limitations and spatial orientation.

Watch on YouTube

This represents one of the more impressive recent developments in redirected walking. The cleverly designed wall structure can trick users into believing they’re walking along an infinitely long, straight corridor when they’re actually following a curved path.

2.5.3 Emerging Display Technologies

2.5.3.1 Light Field Displays

Light field displays represent an emerging display technology that promises unprecedented levels of realism and interactivity in immersive experiences.

Light Field Display Technology - Spherical 3D Playback

This demonstration showcases cutting-edge light field display technology that enables viewers to see properly rendered 3D views of scenes from multiple perspectives without special glasses. Viewers will observe how light field displays can create natural depth perception and motion parallax, allowing users to look around virtual scenes by moving their head position. The technology demonstrates how spherical light fields can be viewed from inside a volume, providing convincing 3D visualization that shifts perspective naturally as users move left, right, forward, back, up, or down, representing a significant advancement toward truly immersive glasses-free 3D displays.

Watch on YouTube

When viewing a spherical light field from inside the volume, you can see properly rendered 3D views of the scene in every direction. The perspectives shift naturally as you move left and right, forward and back, or up and down, creating convincing depth perception without special glasses.

While still in development, I believe light field displays offer the potential for glasses-free 3D viewing with natural depth perception and motion parallax.

For comprehensive coverage of light field capture technology, neural rendering techniques, video capture systems, and detailed technical implementations, see Section 7.4.

2.5.3.2 Immersive Light Field Video

Recent developments have introduced end-to-end systems for capturing and displaying high-quality, immersive light field video content, extending static light field displays to include temporal elements.

Light Field Video Capture System - Immersive Content Creation

This video demonstrates an end-to-end system for capturing high-quality, immersive light field video content, extending static light field displays to include temporal elements. Viewers will see the complex camera array setup and processing pipeline required to capture light field video, which enables truly immersive video experiences where users can move and look around within captured scenes. The technology represents a significant advancement in immersive content creation, allowing for the development of video content that provides natural depth perception and viewing freedom, adding a new dimension to traditional video experiences through spatial exploration capabilities.

Watch on YouTube

This technology allows for immersive video experiences where users can move and look around within the captured scene, adding a new dimension to video content.

2.5.4 AI-Enhanced Interaction in XR

Recent advances in artificial intelligence are transforming how users interact with XR environments, enabling more natural and intuitive interactions through natural language processing and contextual understanding. However, the computational demands of running sophisticated AI models in real-time XR applications currently limit widespread deployment.

For a comprehensive exploration of AI’s role in XR technologies, including AI-enhanced interactions, content generation, character interaction, and development tools, see Chapter 8.

2.5.5 Conclusion

These emerging XR technologies represent the cutting edge of immersive experiences. From advanced haptics and tracking systems to innovative display technologies, these developments are pushing the boundaries of what’s possible in virtual and augmented reality. As these technologies continue to evolve and become more accessible, we can expect to see even more compelling and immersive XR experiences in the future.

For those interested in staying up-to-date with the latest developments in XR technologies, conferences like SIGGRAPH and IEEE VR often showcase cutting-edge research and prototypes in this rapidly evolving field.

2.6 Further Reading

Chapter 2 delved into the various technologies that make up the extended reality (XR) spectrum, from virtual reality (VR) to augmented reality (AR) and mixed reality (MR). We examined the hardware and software components of these systems and explored the concept of the reality-virtuality continuum. To further your understanding of XR technologies and their place on this continuum, explore these resources:

2.6.1 Research Papers

• Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12), 1321-1329.
  • Introduces the reality-virtuality continuum, a fundamental concept in understanding XR technologies.

Bigscreen, Inc. 2024. “Bigscreen Beyond 2—Ultralight Custom PC VR.” Product Launch Blog. https://www.bigscreenvr.com/blog/bigscreen-beyond-2.

Google LLC. 2024a. “Gemini Brings Multimodal AI to Android XR Experiences.” The Keyword. https://blog.google/products/android/gemini-android-xr/.

———. 2024b. “Building for the Android XR Platform.” Google I/O Developer Keynote. https://developers.googleblog.com/en/building-for-the-android-xr-platform/.

Heaney, David. 2024. UploadVR. https://www.uploadvr.com/meta-ultralight-headset-with-puck-less-than-1000-wall-street-journal/.

———. 2025. “Samsung and Google Plan HUD Smart Glasses to Complement Galaxy XR.” UploadVR. https://www.uploadvr.com/samsung-google-hud-smart-glasses/.

Meta Platforms. 2024. “Introducing Ray-Ban Meta Smart Glasses with Heads-up Display and Neural Wristband.” Press Release. https://about.fb.com/news/2024/10/ray-ban-meta-display-neural-band/.

West, Nick. 2025. “Exclusive: Samsung Galaxy XR Headset Specs and Images Leak Ahead of Launch.” Android Headlines. https://www.androidheadlines.com/samsung-galaxy-xr-headset-specs-images-exclusive.