Immersive Communication:
Lex Fridman and Mark Zuckerberg are hundreds of miles apart but can interact as photorealistic avatars in 3D with spatial audio, creating a sense of presence. This technology allows for a more meaningful and immersive way of communication on the internet. The avatars capture facial expressions and nuances that humans use to convey emotions, enhancing the experience.

The Metaverse Environment:
The participants are shrouded in a dark space with ultra-realistic avatars, making it feel like they are in the same room. The level of realism in the avatars’ visuals and expressions is highly impressive, creating an incredibly immersive experience.

Bridging Physical Distance:
Zuckerberg comments on the feeling of being in the same room, despite the physical distance. The experience surpasses traditional video conferencing by providing a sense of presence and intimacy.

Real-World Interactions:
The avatars can interact with each other in a way that mirrors real-world interactions, such as getting close or displaying emotions. The participants can engage in activities together, such as jiu jitsu, expanding the possibilities of virtual interactions.

Photorealism Technology:
This level of photorealism requires advanced technology and algorithms to capture and render the avatars in real time. The technology is still in development, but it has the potential to revolutionize the way people interact and connect in the digital space.

Immersive Experiences with Kodak Avatars:
Meta’s Kodak Avatars project aims to create photorealistic avatars that capture the subtle expressions and flaws of the human face. Instead of transmitting a full video, these avatars use a compact encoded version that maintains bandwidth efficiency.

The Power of Imperfections:
The subtle imperfections and variations in skin tone, wrinkles, and facial features contribute to the immersive experience. Capturing these details enhances the sense of presence and makes virtual interactions more engaging.

Eyes as a Key Element:
Eyes play a crucial role in communication, conveying emotions and expressions. The Kodak Avatar system captures eye movements and expressions, making virtual interactions more natural and realistic.

Accessibility and Future Vision:
Zuckerberg envisions making Kodak Avatars more accessible through quick scans using mobile phones. The goal is to streamline the scanning process to two to three minutes while maintaining the same high quality.

Beyond Video Calls:
The metaverse offers unique experiences that go beyond video calls, enabling users to participate in shared activities and games together. Mixed reality and augmented reality will allow for the integration of Kodak Avatars into the physical environment, creating immersive and interactive experiences.

Uncanny Valley Crossed:
Zuckerberg expresses awe at the photorealistic quality of his avatar, noting how it feels to be in the same room with someone through this interface. It has crossed the uncanny valley, seamlessly mimicking human expressions.

Challenges with Emote Accuracy:
Zuckerberg identifies the challenge of tuning emote accuracy, considering different people’s expressions vary in intensity. He questions the extent to which users should control their avatar’s emotiveness, as some might prefer a more expressive version than their real faces.

Personal Preferences:
Zuckerberg shares his experience of being criticized for his relatively stiff facial expressions. He suggests that he might want his avatar to express emotions more explicitly than he can physically, demonstrating the potential for digital self-representation.

Nuances of Eye Expressions:
Fridman points out the importance of subtle expressions, particularly in the eyes, for conveying emotions like happiness or boredom. Zuckerberg acknowledges the significance of these subtle movements and expresses curiosity about how others perceive his avatar’s expressions.

Immersion and 2D Experience:
Zuckerberg raises concerns about translating the immersive experience of VR and mixed reality into 2D videos. He speculates that the photorealistic nature of these avatars may make the 2D experience as impressive for viewers as it is for participants. However, he acknowledges that other immersive technologies have not always translated well to 2D screens.

Real-Time Heartbeat:
Zuckerberg reveals that his heart is beating fast during the conversation, indicating the level of excitement and anticipation he feels about this technology.

Lex Fridman discusses the remarkable intimacy achieved through remote conversations and acknowledges the groundbreaking nature of remote podcasts. Mark Zuckerberg envisions progressive rollout to enhance accessibility and gradually incorporate scanned avatars into various apps.

Work Style and Gaming Environments:
Zuckerberg anticipates the usefulness of avatars in work style applications and productivity. He suggests that games may require a different style, fitting the game’s aesthetic rather than realism.

Feedback on Workrooms and Quest 3:
Positive feedback received on Workrooms for the immersive experience and the sense of physical presence in remote meetings. Feedback also indicates some users’ discomfort with the expressive avatars’ lack of realism in that environment. The upcoming Quest 3 is expected to be the first mainstream mixed reality product.

Future Applications and Timeline:
Zuckerberg predicts the excitement of remote hang sessions with friends using avatars. He projects the rollout over several years, with ongoing tuning, adding scans, and expanding into more features. In the near future, various experiences using avatars are anticipated to emerge.

Zuckerberg’s excitement for Quest 3:
Expresses enthusiasm for the upcoming Quest 3 headset, describing it as “awesome.” Hopes to see more people, including celebrities and non-celebrities, experience the device.

Quest 3’s mixed reality capabilities:
Quest 3 will be the first mainstream mixed reality device, combining virtual and real-world elements. Zuckerberg believes Quest 3’s mixed reality is superior to that of Quest Pro, despite being more affordable ($500).

Factors contributing to Quest 3’s success:
Software improvements and addressing bottlenecks from previous products. Higher resolution cameras and sensors for enhanced mixed reality experiences. Collaboration with Qualcomm for a custom chipset, providing improved performance.

Mixed reality’s benefits and applications:
Offers a safer and more secure experience compared to fully immersive VR. Enables users to integrate digital content into their physical surroundings, such as adding windows to a room. Allows for more diverse experiences, including non-immersive VR games and fitness activities.

Challenges in mixed reality technology:
Requires multiple cameras to capture 3D space and complex AI processing to align the digital content with the user’s perspective. Ongoing efforts to improve and refine the technology in the coming years.

Quest 3’s potential impact:
Zuckerberg anticipates Quest 3 to be the first mixed reality device to reach millions of users. With a larger user base, it will attract developers to create diverse experiences, further fueling the ecosystem.

Additional features of Quest 3:
Hand tracking without controllers, utilizing cameras for gesture recognition. Improved display and chip, resulting in better graphics and a thinner, more comfortable design.

Physical-Digital Convergence:
Mark Zuckerberg describes a future where technology merges physical and digital worlds seamlessly. He envisions photorealistic holographic experiences with latency-free interactions, creating a more immersive digital experience in the metaverse.

Kodak Avatars and Mixed Reality:
Zuckerberg highlights the significance of Kodak’s ultra-realistic avatars, emphasizing their impact on integrating digital objects into physical environments. He predicts the normalization of holograms in our lives, with potential dominance over physical objects in the coming years.

Challenging the Notion of the Real World:
Zuckerberg discusses his perspective on the changing definition of the real world in the era of digital technology. He suggests that the modern real world encompasses both physical and digital components, rather than them being separate entities.

Advantages of Holographic Experiences:
Zuckerberg believes that holograms offer advantages over physical objects. He notes the ease of manipulation, instant accessibility, and the ability to transform holograms into desirable forms with simple gestures like snapping fingers.

Embodiment of AIs:
Zuckerberg envisions the embodiment of AIs in the mixed reality. He describes future meetings where attendees can appear as holograms, including AIs like Bob the engineer, who can be present as a realistic avatar.

Acceptance and Empathy:
Zuckerberg emphasizes the importance of acceptance in the adoption of photorealistic avatars. He believes these avatars promote empathy by enabling users to better connect and understand others, especially in contexts that require emotional or empathetic understanding.

Zuckerberg’s vision for the future of avatars:
In the future, Zuckerberg believes that people will use avatars to express themselves in ways that are not tied to their physical reality, allowing for a more personalized and expressive digital experience.

Combining photorealism and expression:
Zuckerberg highlights the possibility of combining photorealistic avatars with expressive, cartoonish elements, such as using a photorealistic avatar with a cartoony background or an expressive avatar next to a photorealistic one.

Long-term implications:
Zuckerberg questions whether photorealistic avatars will become the norm or if there will be ongoing use for both photorealistic and expressive avatars, suggesting that the balance between the two will be shaped over time.

The allure of photorealism:
Lex Fridman emphasizes the appeal of photorealistic avatars for experiences like remote social gatherings, where the goal is to recreate the feeling of being together in the same physical space.

Photorealism and identity:
Fridman raises philosophical questions about identity in the context of photorealistic avatars, such as whether people will strive to present their best selves through their avatars, leading to a distorted representation of their true identity.

Zuckerberg’s Perspective on Identity in the Metaverse:
Mark Zuckerberg questions if people are the entity driving their digital avatars or if they are the avatars themselves. He compares the fluctuating nature of physical bodies to the changes in people’s digital identities, raising psychological questions about self-perception. Most individuals tend to identify with their best version rather than their worst, but daily experiences may not align with either.

The Metaverse and the deceased:
Lex Fridman discusses the desire to communicate with loved ones who are no longer alive in the metaverse. Zuckerberg acknowledges the emotional power of such an experience but emphasizes the need to find a balance to avoid unhealthy effects.

Best Practices from Social Media:
Zuckerberg draws on experiences in managing death and identity on social media platforms. Existing protocols memorialize profiles, but certain private content is protected. He suggests that best practices from the digital world will be carried over into the metaverse.

Intersection of AI and the Metaverse:
Zuckerberg envisions a world where many people are empowered by AI tools, avoiding a single dominant superintelligence. AI versions of individuals can help creators and influencers interact with their communities beyond the limitations of time. Such AI versions can be distinguished from physical versions, offering a valuable alternative for interaction.

Challenges and Experimentation:
Zuckerberg acknowledges challenges in developing AI versions of individuals. While not yet ready for public release, Meta is experimenting with AI that people can interact with in various ways. This will eventually become a powerful capability for individuals in the metaverse.

Zuckerberg’s Vision:
Mark Zuckerberg envisions a diverse range of AI personalities, each serving specific purposes and enhancing user experiences across Meta’s platforms.

Benefits of AI Personalities:
Empowers individuals to accomplish their personal and business goals efficiently and effectively. Personal assistants for everyday tasks, offering factual information and assistance. Business-oriented AIs for seamless customer interactions, improving efficiency for both businesses and customers. Entertainment-focused AIs, such as joke-telling AIs, to enhance social interactions and group chats.

Expanding AI Capabilities in the Gaming World:
AIs to improve NPC interactions in games, making them more engaging and intelligent. Dungeon master AIs for text-based adventure games, embodying cultural figures to create unique experiences. Collaborations with celebrities like Snoop Dogg to create immersive and personalized AI personalities.

Technical Challenges:
Creating AI personalities that accurately reflect real-world creators requires careful consideration and technical expertise. Ensuring that AIs adhere to the creator’s preferences and accurately represent their knowledge and perspectives. Training AI models to be predictable and reliable in their communication, enabling creators to set parameters and boundaries.

Anticipated Timeline:
New character AIs expected to be released soon after Meta’s Connect event. Experimentation with existing characters, assistants, and games to begin this year, followed by small business integrations.

Impact on Personal Interactions:
Potential for profound personal experiences, such as engaging in conversations with one’s own AI assistant. Exploring new frontiers of self-expression and identity through virtual avatars. Blending physical and digital realities to create transformative experiences.

The Metaverse: A Physical and Digital Blend
Mark Zuckerberg envisions a future where the physical and digital worlds seamlessly blend, creating a more immersive and coherent experience. This vision addresses the limitations of current computing devices, which are often static and don’t fully capture the human experience.

Ethics and Boundaries in the Metaverse
The metaverse presents unique ethical and philosophical questions regarding what is allowed and what isn’t. Zuckerberg acknowledges the need to establish content policies and regulations while also ensuring freedom and minimizing harm. He raises the issue of anonymity in the digital world and its potential to reduce the incentive for good citizenship.

Breaking the Laws of Physics: Telepresence and Remote Work
The metaverse has the potential to break down physical barriers, allowing people to feel like they are together even when they are physically distant. Zuckerberg believes that this will revolutionize remote work, enabling people to work from anywhere while still experiencing the benefits of physical presence. He sees this as a solution to the ongoing debate about remote work versus in-person work, allowing people to choose their physical location without sacrificing collaboration and teamwork.

Building the Metaverse: A Gradual Evolution
Zuckerberg acknowledges the complexity of predicting the full implications of the metaverse and the need to mitigate potential downsides. He emphasizes the importance of taking a gradual approach, building the metaverse one year at a time and observing how developers and users interact with it. This iterative approach allows for adaptation, learning, and refinement based on real-world experiences and feedback.

Mark Zuckerberg’s Thoughts on Lama 2 Release and Future Plans:
Zuckerberg believes open-sourcing Lama 2 was the right decision, as its benefits outweigh the risks. The reception of Lama 2 has been incredibly positive, with more downloads and usage than anticipated. The current priority is integrating Lama 2 into consumer products, making it accessible to hundreds of millions of people. Future foundation models are in development, but no specific details or release dates are available.

AI Personas and Their Role in Shaping AI’s Future:
Personalized AI systems and AI personas provide valuable insights into user preferences and failure cases. These systems highlight areas where AI responses are insufficient or inappropriate. By using and interacting with these AI personas, society can collectively learn about personalization, generic responses, and boundaries in AI.

Exploring Diverse Use Cases for AI:
Meta’s initial AI launch aims to offer a variety of use cases for AI systems. Zuckerberg emphasizes that it is uncertain which use cases will resonate with users. Text-based adventure games, comedic AI assistants, and interactions with historical figures are among the diverse use cases being tested.

The Power of Presence:
Mark Zuckerberg emphasizes the significance of creating a sense of presence in the digital realm, allowing people to feel like they are in the same physical space as others, despite physical distance. The metaverse aims to bridge the gap between the physical and digital worlds, offering a rich and immersive experience that combines the best of both.

The Current State of Communication:
Zuckerberg acknowledges that current communication methods, such as phone calls or video chats, lack the intimacy and nonverbal cues that are essential for meaningful human interaction.

The Promise of the Metaverse:
The metaverse holds the promise of overcoming these limitations by enabling users to feel present with others, regardless of their physical location. This has the potential to revolutionize communication and social interaction, fostering deeper connections and a sense of community.

Technological Challenges:
Zuckerberg highlights the significant technological challenges involved in creating a truly immersive and realistic metaverse experience. Building new computing platforms and software stacks is necessary to achieve the desired level of presence and intimacy.

The Evolution of Social Media:
Zuckerberg draws parallels between the evolution of social media and the development of the metaverse. Just as social networks transitioned from text to photos and then to videos, the metaverse represents the next frontier of expression and content sharing.

Immersion and Vividness:
The ability to be immersed and feel present with others is seen as the next natural step in the evolution of social media. This immersive experience offers a richer and more vivid way of connecting with the world around us.

Collaboration and Gratitude:
Zuckerberg expresses his appreciation for the contributions of his colleagues and the Meta team in building the metaverse. He emphasizes the shared mission of creating a future where people can easily connect and feel present with others, regardless of distance.