Why Visual Realism Matters:
Creation of the “realistic” sense of presence through immersive experiences. This presence enables you to perceive another entity as being physically in the same space with you. Visual realism will bring about a new generation of visual experiences and art, similar to the cultural evolution in past times.

Challenges in Creating Next-Generation Displays:
The human visual system is highly integrated and requires multiple factors to create immersion. Stereoscopic displays are necessary for 3D images. The system needs to support focusing the eyes at different distances. The display must cover a wider field of view than traditional models. Retina-level resolution across the entire field of view requires an enormous number of pixels. Screens must match the brightness and dynamic range of the physical world, demanding higher brightness than current HD TVs. Realistic motion tracking with low latency is essential for accurate movement and spatial awareness. A new graphics pipeline is needed to maximize performance without draining the battery or overheating the headset. All these components need to be integrated into a comfortable and wearable device.

Visual Turing Test:
Zuckerberg introduces the concept of the Visual Turing Test to explain the benchmark for their visual fidelity efforts. The term is analogous to the standard Turing Test in the field of artificial intelligence. Michael Abrash will explain the details of the Visual Turing Test in the next section.

Overcoming Challenges to Creating Realistic VR Displays:
Visual Turing test: A subjective test to assess whether VR displays can mimic the real world to an indistinguishable level. Current VR technology: While VR creates a strong sense of presence, it falls short of achieving the visual realism required for the Turing test.

Challenges in VR Display Technology:
Resolution: VR requires much higher resolution than traditional 2D displays to deliver a realistic visual experience. Other challenges: Beyond resolution, various factors contribute to VR display limitations, including Vergence-Accommodation Conflict, Chromatic Aberration, Ocular Parallax, Pupil Swim, and the need for compact and lightweight headsets with long battery life.

Complexities of Distortion and Focus:
Distortion: VR lenses distort the virtual image, reducing realism unless corrected in software. Dynamic distortion: Distortion changes as the eye moves, making software correction intricate. Discomfort: Distortion and headset weight can lead to temporary discomfort and fatigue during extended VR use. Focus: The ability to focus at varying distances accurately is a key challenge in VR displays.

Key Areas of Focus:
To improve VR display quality, the focus is on resolution, color range, brightness, and contrast.

Resolution:
VR headsets have wider fields of view, leading to a lower pixel density compared to traditional 2D displays. To achieve 2020 vision across the full human field of view, a resolution of more than 8K is required. The human visual system perceives things in high resolution only in the focused area, while the periphery is in lower resolution.

Color Range, Brightness, and Contrast:
Current VR headsets fall short in color range, brightness, and contrast compared to other devices like laptops, TVs, and mobile phones. VR needs to reach these levels to provide the fine detail and accurate representation of 2D displays.

Retinal Resolution Headset:
The goal is to create a retinal resolution headset with approximately 60 pixels per degree in the display, which is several times more than current VR headsets.

Custom Prototypes:
Researchers developed a prototype called Butterscotch, capable of displaying the 2020 vision line on an eye chart in VR. These prototypes showcase the potential for incredibly sharp images in VR but are not ready for commercial use.

Latest Retinal Resolution Prototype:
The latest prototype achieves a resolution of 55 pixels per degree, which is two and a half times the resolution of the Quest 2. No current display panels can support retinal resolution for the full field of view of VR headsets.

Introduction:
This transcript excerpt discusses the challenges and progress in achieving visual realism in VR, focusing on advancements in resolution, depth of focus, distortion, and dynamic range.

Resolution:
Butterscotch team developed a prototype with a higher resolution that provided a significant improvement in the VR experience. Current display panels are expected to continue improving, eventually reaching retinal resolution levels.

Depth of Focus:
Varifocal technology was introduced to adjust the focal depth dynamically, matching the distance of objects being viewed. A 2017 prototype with mechanical varifocal displays showed positive results in user preference and comfort. Electronic varifocal based on liquid crystal lenses became part of the Half Dome prototypes, reducing size and weight.

Distortion:
Current VR optics introduce distortion, which can be compensated for in software but requires dynamic correction due to eye movement. A distortion simulator using virtual objects and eye tracking was developed to rapidly explore different optical designs and distortion correction algorithms.

Eye Tracking:
Eye tracking plays a crucial role in determining focus, correcting optical distortions, and optimizing rendering resources. It allows for more efficient rendering by focusing on the parts of the scene being looked at in the highest detail, similar to the human visual system.

High Dynamic Range (HDR):
HDR is crucial for achieving realistic brightness and contrast, with preferred peak brightness levels at 10,000 nits. Current VR headsets like Quest 2 offer a maximum of about 100 nits, creating a significant gap compared to the real world. The Starburst prototype combines an incredibly bright lamp with LCD panels, although impractical, it is used for testing and studies to understand the HDR experience.

Conclusion:
The combination of advancements in retinal resolution, varifocal technology, distortion reduction, HDR, and eye tracking are crucial for achieving visual realism in VR. Further iterations and integration of these technologies are necessary, with research also exploring new solutions for future breakthroughs. The goal is to create headsets that can accurately represent the beauty and complexity of physical environments, leading to more immersive and realistic VR experiences.

A New VR Prototype: Introducing Holocake 2 and Mirror Lake:
Holocake 2: An innovative approach to VR headsets, using holographically generated lenses and polarized reflection to achieve a thinner and lighter design. Mirror Lake: A concept design that integrates various advanced visual technologies, including varifocal and eye tracking, into a ski-goggle-like form factor.

Key Innovations in Holocake 2:
Holographic Lenses: Holograms are used instead of physical lenses, offering a much flatter design while maintaining optical performance. Polarized Reflection: Light is bounced between reflective surfaces to reduce the effective distance between the display and the eye, resulting in a more compact package.

Challenges and Solutions:
Finding the Right Light Source: Holocake requires specialized lasers, which present challenges in terms of performance, size, and price. Engineering efforts are underway to achieve a consumer-viable laser solution.

Mirror Lake: Bringing Together Advanced Visual Technologies:
Varifocal Technology: Allows the lenses to adjust their focal length, providing a more natural and realistic viewing experience. Prescription Correction: Built-in prescription correction eliminates the need for eyeglasses or contact lenses while using the headset.

Eye Tracking: Incorporates eye tracking to enhance visual realism and enable more intuitive interactions.

Conclusion:
The future of VR holds immense potential, driven by constant innovation and the integration of advanced visual technologies. Holocake 2 and Mirror Lake represent significant steps towards achieving compact, lightweight, and visually immersive VR experiences. The challenges, particularly in finding a suitable laser source, are being actively addressed by engineering teams. The journey towards photo-realistic VR headsets is underway, promising a more intuitive and immersive way of interacting with the digital world.