Original Motivation and Enduring Excitement:
Dmitri Dolgov, an expert in autonomous vehicle development, feels privileged to be part of a once-in-a-lifetime opportunity to transform the transportation industry and positively impact society. The combination of making a substantial impact, pushing the boundaries of technology, and collaborating with talented individuals keeps him enthusiastic about the field.

Technology Behind Waymo’s Driver:
Waymo’s fully autonomous driver requires capabilities similar to human drivers: understanding location, surroundings, and interactions, making predictions, and executing safe driving decisions. It involves advanced technology in sensing, computation, machine learning, AI, and infrastructure, all of which are constantly evolving.

Key Learnings and Technology Evolution:
Over the past decade, Waymo has accumulated extensive autonomous driving experience in California, providing valuable insights for technology development and product refinement. The Trusted Tester program in San Francisco allows residents to experience the latest version of Waymo’s technology and provide feedback. Waymo’s fifth-generation driver incorporates the latest hardware and software, offering valuable insights from riders for further improvement.

Challenges and Complexity of Autonomous Driving:
Building a fully autonomous driver involves replicating human capabilities like understanding surroundings, predicting interactions, and making safe driving decisions. Waymo’s approach focuses on developing a comprehensive system that encompasses perception, prediction, planning, and control. The complexity lies in integrating various technologies and ensuring safe and reliable operation in diverse environments.

Hardware Suite:
Waymo uses three sensing modalities: cameras, lidars, and radars. The sensing modalities offer high-resolution, long-range, and 360-degree coverage around the vehicle. The onboard compute system emphasizes fast, real-time inference in high-capacity ML models.

Autonomy AI and Onboard Software:
Waymo leverages state-of-the-art ML in all parts of the stack, from perception to behavior prediction to planning. The company continually iterates on the architecture of its entire system to leverage the most advanced ML capabilities.

AI and ML Challenges:
Accuracy and efficiency are ongoing challenges in AI and ML for autonomous vehicles. Waymo works to leverage structured representations and address sparse data issues. Collaborating with the research community is essential for pushing the boundaries of AI and ML.

Scaling Across Geographies:
The fundamental capabilities required for driving in different environments are similar. Waymo’s experience in Arizona provides a strong foundation for scaling to new locations. Technical challenges in scaling include handling diverse weather conditions, road conditions, and traffic patterns. Data collection and simulation play crucial roles in adapting to new environments. Waymo’s commitment to safety and continuous improvement is essential for successful scaling.

Diverse Testing Locations:
Waymo has tested its autonomous vehicle system in over 25 cities, gaining valuable experience from various environments.

Phoenix as the First Deployment Site:
Phoenix was chosen for the first ride-hailing service deployment due to its unique characteristics and infrastructure.

Multi-Year Research and Engineering Efforts:
Waymo’s system has undergone years of research and engineering, leading to the fifth generation now operating in San Francisco.

Benefits of Driverless Operation in Phoenix:
The experience of operating driverless vehicles in Phoenix has been incredibly valuable for Waymo. It involved taking the technology to full-scale operation, not just one-off demos or pilots. The challenges encountered and lessons learned have been instrumental in Waymo’s progress.

Phoenix as a Proving Ground:
Phoenix provided a real-world testing environment for Waymo to evaluate its system’s capabilities thoroughly. The complexity of evaluating autonomous vehicles was recognized as being on par with the challenge of building the technology.

Learning to Develop and Iterate Quickly:
Waymo learned to develop and iterate on its system’s performance efficiently in Phoenix. The city allowed Waymo to identify areas for improvement and optimize various metrics. This iterative process helped prevent stagnation and ensured progress towards better performance.

Deployment as an Ongoing Process:
Waymo invested in building a robust system for evaluating and deploying its autonomous vehicles. This allowed researchers and engineers to focus on advancing the technology while deployment occurred in the background.

Operating a Fleet of Autonomous Vehicles 24-7:
Waymo gained valuable experience in operating a fleet of fully autonomous vehicles around the clock. Operating a fleet of rider-only autonomous vehicles requires additional considerations compared to fleets with human drivers.

Experience Transfer to Future Deployments:
The experience gained from operating autonomous vehicles translates well to future deployments in different domains. Waymo recently launched its Trusted Tester program in San Francisco, demonstrating the transferability of its experience to new domains.

Key Challenges in Launching Autonomous Ride-Hailing Services:
Launching a full autonomous ride-hailing service involves numerous challenges, including operational procedures, system and infrastructure requirements. Waymo’s unique experience in building and operating such a service is valuable for future deployments in different geographies.

Operating in Different Business Domains:
Waymo is expanding its operations to include long-haul trucking and local delivery programs. Designing AI capabilities that can be leveraged across these business domains is a key challenge. Waymo’s approach to this challenge involves considering it as a system design problem and enabling organizational capabilities to support this approach.

Unifying Technology Development:
Waymo’s approach focuses on building a driver system capable of fully autonomous driving across multiple applications and commercial lines. The company has two main business lines: Waymo One for ride-hailing and Waymo Via for trucking and local deliveries.

Generalization Across Domains:
The amount of overlap in technology and infrastructure varies depending on the combination of products and deployments. Local deliveries involve a subset of challenges compared to ride-hailing, while trucking has some shared capabilities but also requires specialization.

Core Driving Capabilities:
The fundamentals of autonomous driving, such as hardware, autonomy AI, and infrastructure, are shared across domains. This includes sensing, hardware platforms, compute, perception, semantics, prediction, planning, data science, data management, and simulation.

Benefits of Shared Infrastructure:
Sharing technology and infrastructure allows Waymo to focus on principled solutions and simplify architecture. It enables positive feedback loops between domains and leads to a more robust and generalizable core stack. This approach also promotes investment in core capabilities adaptable to different parts of the business.

Core System Benefits:
Investing in fundamental problem-solving without fragmented pieces enables solutions across various deployments and environments. The core system serves both current business applications well and provides a foundation for future expansion.

Future Opportunities:
The shared core technology and infrastructure will support pursuing additional vehicle platforms, scaling to new environments and cities, and supporting multiple product applications.

Data-Driven Development at Waymo:
Evaluation and data-driven development have been integral to Waymo’s strategy since its inception. The company’s data strategy has undergone significant changes over time, reflecting advancements in ML models and the need to address challenges posed by the tail of the distribution.

Changes in ML Models:
In the early days, ML work primarily focused on perception and supervised ML with human labeling. As advanced ML models found applications across other parts of the stack, the data strategy evolved to incorporate auto-labeling and generative approaches to data management.

Expanding into the Tail of the Distribution:
ML models operate on distributions, and cases in the tail of the distribution or out of the distribution present challenges. Waymo’s strategy has prioritized gathering data in the tail of the distribution to improve the performance of its models.

Evaluating Performance and Data Mining:
Evaluating system performance and data mining are crucial for identifying interesting examples representative of the tail part of the distribution. Techniques like data augmentation and simulation are used to leverage these examples and allow models to handle tail cases better.

Automating the Process:
Waymo has invested in frameworks and ML infrastructure to automate the entire process of data mining, training, and closing the loop on the data mining training cycle.

Data Feedback Loop:
Prioritizing the development of a robust data feedback loop is crucial for continuous improvement. Automating and minimizing human intervention in the feedback loop is essential to optimize performance.

Long Tail Circumstances:
Self-driving encounters unique long tail circumstances that differ from traditional driving scenarios. Examples include construction zones, jaywalking pedestrians, and animal crossings. Rare events like a drunk cyclist with a stop sign or animals on the road are also encountered. The simulator is utilized to expand these examples into similar scenarios for training and evaluation.

Balancing Head and Tail Optimization:
The trade-off between optimizing for average cases and long tail scenarios is a significant challenge. Overemphasis on the tail can lead to overly cautious behavior, while neglecting the tail can compromise safety. Techniques like modeling data, data augmentation, evaluation, and system-level capabilities are employed to address this trade-off.

Development for Unseen Scenarios:
Designing a system robust to unseen scenarios is critical due to the open-ended nature of long tail problems. System-level mechanisms like redundancy, anomaly detection, and outlier detection are essential for handling such scenarios. Perception models and algorithms are trained to reason about unfamiliar objects or situations.

Technology for Open-Ended Prediction and Planning:
Open-ended prediction and planning problems require a broader approach to technology. Techniques such as world modeling, behavior prediction, and reinforcement learning can be employed to address these challenges.

Overview:
This chapter discusses the challenges beyond perception in autonomous driving, such as behavior prediction, semantic understanding, and decision-making. It also delves into data strategies, including off-board perception and data augmentation, to tackle these challenges.

Challenges:
Perception is just the start of the challenge in autonomous driving. Behavior prediction, semantic understanding, and decision-making are complex due to the high dimensionality of the problem. The entire context of the scene becomes crucial in understanding intent, making predictions, and taking decisions.

Data Strategies:
Off-board perception and leveraging computational power help reduce reliance on human labels. Data augmentation techniques, including synthetic data generation, can enhance the training dataset. Auto-labeling simplifies labeling for behavior prediction by observing future events. Hierarchical graph neural nets model interactions between static and dynamic parts of the environment. Vector nets are used for behavior prediction by leveraging advanced techniques.

Imitation Learning:
Imitation learning in a complex environment like driving is powerful but insufficient. Augmentation with structured representations and simulator exploration helps address out-of-distribution issues.

Sensors:
Different sensing modalities, such as LiDARs, cameras, and radars, complement each other for autonomous driving. LiDARs are active sensors that work well in low-light conditions and provide precise 3D information.

LiDARs Complement Other Sensors:
Different wavelengths of radar allow them to perform better in specific environmental conditions, making them complementary to each other. Waymo’s strategy involves better sensors, enhanced data, and a more robust system. Avoiding limitations by not utilizing the power of various sensing modalities, including LiDAR, is unnecessary.

Designing for Performance and Manufacturing:
Waymo has developed five generations of hardware, gaining expertise in building LiDARs and understanding crucial factors for their entire stack. Their fifth-generation LiDARs are more advanced than industry standards and their fourth-generation system. Scaling is another reason for building their own LiDARs, as designing for manufacturability at scale is essential. Building self-driving hardware differs from building cars at scale, requiring expertise in manufacturing complex electro-optical systems.

Optimizing the Hardware Stack:
Waymo’s fifth-generation hardware, including LiDARs, is designed for manufacturability at scale, providing the necessary capabilities, reliability, and cost-effectiveness. Designing the entire hardware stack allows for optimizing sensing modalities for the software stack and the entire system, maximizing performance and value.

Industry’s Progress:
The autonomous vehicle industry has come a long way since its inception, moving from a science fiction concept to a deployable product. Waymo, a pioneer in the field, now operates a fleet of fully autonomous vehicles that are open to the public.

Current Achievements:
Waymo’s autonomous vehicles are fully functional, driving around and transporting passengers within their service territory. This milestone serves as a significant proof point for the viability and practicality of autonomous vehicle technology.

Challenges Ahead:
Despite the progress, there is still a long way to go before autonomous vehicles become widely adopted and seamlessly integrated into society. The industry needs to address technical challenges, regulatory hurdles, and public acceptance to ensure the successful adoption of autonomous vehicles.

Waymo’s Current State and Future Goals:
Waymo is currently in a very exciting phase, with proof points such as self-driving cars operating without human drivers. This solid footing provides a foundation for future advancements. The experience of operating self-driving cars in real-world conditions has been invaluable for Waymo’s development.

Challenges and Breakthroughs:
Waymo identifies many exciting challenges and potential breakthroughs in various areas. High-capacity ML models and their application in the autonomous vehicle domain are highlighted as promising areas of innovation.

Balancing Reality and Innovation:
Waymo emphasizes the tangible reality of self-driving cars while acknowledging the dynamic nature and technological momentum in the industry.

Expansion and Accessibility:
Waymo plans to expand its Waymo One and Waymo Via services to more places and people, making autonomous transportation more accessible.

Future Plans:
In the coming months, Waymo will focus on expanding its trusted tester program in San Francisco and testing and operations on the trucking side.

Conclusion:
Waymo is optimistic about the future of autonomous vehicles, with plans to continue expanding its services and innovating in key areas.