Introduction:
Deep learning, a modern rebranding of artificial neural networks, is experiencing a surge in interest and growth. Google’s research approach includes basic research, infrastructure building, product collaboration, and emerging area exploration.

Machine Learning Field Explosion:
Machine learning archive papers are growing faster than Moore’s law, indicating the rapid advancement of ideas. Deep learning, with its simple trainable units, has gained significant attention.

Early Excitement and Limitations of Neural Networks:
Neural nets faced challenges in the past due to limited computational power. The current availability of massive compute has enabled impressive results with artificial neural networks.

Capabilities of Neural Networks:
Computer vision: Neural nets can categorize images, generate captions, and transcribe audio. Language translation: Neural nets can translate text from one language to another. Progress in Computer Vision: ImageNet contest showcases remarkable improvement in object recognition accuracy.

U.S. National Academy of Engineering Grand Challenges:
The talk focuses on applying deep learning to address the U.S. National Academy of Engineering’s grand challenges for the 21st century.

Machine Learning’s Role in Addressing Global Challenges:
Machine learning can significantly contribute to solving global challenges, such as improving urban infrastructure, promoting healthier living, and mitigating climate change.

Self-Driving Cars: A Revolution in Transportation:
Self-driving cars are being actively tested and developed, with Waymo leading the way in real-world trials. These cars use sensor data to understand their surroundings, predict object movements, and make safe driving decisions.

Vision and Robotics:
Vision is crucial for robots to navigate and interact with their environment. Researchers are developing hardware labs of robotic arms that can experiment and pool their collective experience to improve their ability to grasp objects and perform various tasks.

Grasping as a Fundamental Task:
Grasping is a fundamental skill for robots, as it enables them to manipulate objects and perform complex actions. Researchers are focusing on improving robots’ ability to grasp objects effectively.

Robotics:
Researchers explored supervised learning with parallel robotic arms, enabling self-supervised grasping tasks. Grasp success rates improved from mid-60s to 96% for unseen objects through algorithm enhancements. Learning approaches are being explored to give robots new skills by observing videos, demonstrating promising results in emulating human actions.

Healthcare:
Machine learning has tremendous potential in healthcare for improved decision-making. In medical imaging, computer vision techniques aid in diagnostic tasks like diabetic retinopathy screening. An ML model achieved par or better performance compared to US board-certified ophthalmologists in grading retinal images. Advanced models can identify patterns in retinal images that ophthalmologists may not be aware of, providing new insights into patient health. Machine learning models can predict future outcomes for patients based on electronic medical record data, aiding in decision-making and treatment planning. Applications of these ML models include Smart Reply in Gmail, machine translation, and predicting various aspects of patient care using de-identified medical records.

Predicting Patient Mortality:
Traditional methods use 20 hand-designed features to predict mortality risk. Using all data in the medical record can improve prediction accuracy. Earlier assessment of patients’ medical risk is possible with this approach.

Accelerating Quantum Chemistry Simulations:
Quantum chemists use computationally intensive simulations to predict molecule properties. Training a neural network with simulation data can achieve indistinguishable accuracy. The neural network is 300,000 times faster than the original simulator.

Exploiting Simulators for Scientific Discovery:
Replacing simulation components with neural networks can lead to significant speedups. Screening large datasets becomes feasible with faster simulations.

Engineering Tools for Scientific Discovery:
The grand challenge aims to develop tools that facilitate scientific discovery. The speaker suggests exploring the use of neural networks to accelerate simulations.

TensorFlow:
TensorFlow is an open-source machine learning framework developed by Google for expressing machine learning research ideas and deploying them in real-world environments. It has gained significant interest and adoption, with over 16 million downloads and a vibrant community contributing to its development. TensorFlow is flexible and runs on various platforms, including mobile phones, desktops, and data centers.

Democratizing Machine Learning:
There is a shortage of machine learning experts compared to the vast number of organizations that could benefit from machine learning applications. Traditional machine learning problem-solving involves human experts making numerous decisions about modeling, data preparation, and experimentation. To make machine learning more accessible, researchers are exploring ways to automate some of these tasks and provide data-plus-computation solutions.

Neural Architecture Search:
Neural architecture search involves using a model-generating model to automatically design neural network architectures. The model-generating model creates descriptions of network structures, which are then trained on a specific problem. The accuracy of these trained models is used as feedback to guide the model-generating model towards better architectures. This iterative process leads to the development of increasingly accurate models over time.

Results and Insights:
Neural architecture search can produce models with unique connectivity structures that are not typically designed by human experts. These “organic style skip connections” allow information to flow more directly from the input to the output, improving model performance. Experiments on ImageNet, a popular image classification dataset, demonstrate the effectiveness of neural architecture search in achieving state-of-the-art results.

AutoML’s Achievements:
AutoML has revolutionized machine learning and computer vision, delivering state-of-the-art results through collaborative efforts of top researchers. Its impact is evident across various applications, from high-accuracy models to cost-efficient models suitable for resource-constrained environments.

AutoML in Google’s Cloud Products:
Google has integrated AutoML into its cloud products, enabling industries without machine learning expertise to utilize it. Users can label image datasets, upload them, and obtain highly accurate trained models for various tasks, such as part recognition in factories.

Exploring Alternatives to Reinforcement Learning:
Researchers are investigating evolutionary algorithms as an alternative to reinforcement learning in AutoML. This approach aims to determine the effectiveness of evolutionary algorithms in comparison to reinforcement learning.

Learning Symbolic Optimization Rules:
AutoML can learn and combine different optimization update rules symbolically, resulting in new and improved optimization functions. It can also discover non-linearity functions beyond those traditionally explored in neural network optimization.

Optimizing Models for Latency and Accuracy:
AutoML can be adapted to consider both latency and accuracy when searching for optimal neural architectures. This enables the creation of models that meet specific latency requirements while maintaining high accuracy.

Data Augmentation Policies:
AutoML utilizes data augmentation techniques to expand the training dataset and improve model performance. It applies transformations such as resizing, flipping, and cropping images while preserving their labels.

Exploring Multiple Architectures Simultaneously:
To reduce computational costs, AutoML can explore multiple neural architectures simultaneously, sharing parameters among them. This approach allows for efficient exploration of a broader range of architectures.

TPU Background:
Deep learning algorithms are tolerant of reduced precision, requiring only one decimal digit of precision, unlike traditional HPC codes. Neural algorithms utilize a handful of specific kinds of matrix operations and linear algebra operations. Low-precision linear algebra operations can be used to express various machine learning algorithms.

TPU Motivation:
Increased speech recognition usage could potentially double the number of computers needed for that feature alone. A hardware accelerator was required for low-latency inference needs.

TPUv1:
First TPU was for inference only, not training. Designed to fit a trained model in memory and perform low-precision number multiplication. Used in production for four years, including search queries, translations, speech, and image recognition. Published in ISCA 2017.

TPUv2:
Designed for both training and inference, targeting large problems. Features four chips with about 180 teraflops of compute, connected together. Matrix multiply unit provides 65,000 scalar results in 128×128 matrix per cycle. Designed to be connected into pods, such as 64 chips arranged in a 16×16 mesh.

TPUv3:
Announced in May, with improvements to the core chip and water cooling for higher power. Can be connected into a larger pod configuration, about four times bigger than TPUv2. Delivers eight times the performance of the earlier pod, achieving 100 petaflops of compute. Enables large-scale machine learning problems and quick experimentation results.

Programming and Accessibility:
Programmed via TensorFlow, allowing the same program to run with minor modifications on different platforms. Scales via synchronous data parallelism, making it easy for machine learning researchers to use pods without modification. DAWNBench benchmark shows that TPUs hold the top three spots for training time and training costs. 1,000 devices are made available for free to top researchers committed to open machine learning research.

Introduction to Learned Index Data Structures:
Traditional index data structures, such as B-trees, are used in databases and key-value stores to efficiently find data based on a key. Machine learning offers an opportunity to create learned index data structures that can replace traditional ones and provide improved performance and space efficiency.

How Learned Index Data Structures Work:
Learned index data structures use a small neural network to make predictions about the location of a key in a sorted list of keys. The neural network takes in a key as input and outputs a prediction of where that key is likely to fall in the sorted list. This prediction is used to perform the same function as a B-tree, which is to identify the disk page where the key must exist if it exists at all.

Benefits of Learned Index Data Structures:
Learned index data structures can be significantly faster than traditional B-trees, with one example being 60% faster. They can also be much smaller in size, with the same example being 1/20th the space of a B-tree. The accuracy and size of the learned index can be traded off to achieve the desired performance and space requirements.

Potential Applications of Learned Index Data Structures:
Learned index data structures have the potential to revolutionize how data is indexed and accessed in computer systems. They could lead to smaller and faster databases, key-value stores, and other data storage systems. The ability to adapt to the actual usage patterns of the system could further improve performance and efficiency.

Opportunities for Learning-Based Systems:
The speaker emphasizes the potential of using learning instead of heuristics in various domains, including instruction scheduling, register allocation, loop nest parallelization, networking, operating systems, ASIC design, and more. These systems could be self-adaptive and high-performance, eliminating the need for handwritten heuristics.

Challenges of AI in Society:
Machine learning systems are becoming more prevalent and influential in society, raising concerns about safety, bias, interpretability, and privacy. Medical diagnostic systems require interpretable predictions for clinicians to trust and understand the reasoning behind decisions. Machine learning systems can be vulnerable to privacy issues, especially with sensitive data.

Google’s Principles for AI:
Google has established principles to guide the responsible development and use of AI, addressing fairness, safety, interpretability, and privacy. These principles are not always straightforward to apply, and ongoing research is needed to develop methodologies for ensuring the safety of learned systems.

Fairness in Machine Learning:
Machine learning systems trained on biased data can perpetuate and amplify existing biases. Research is being conducted to explore how to mitigate bias and promote fairness in machine learning predictions.

Conclusion:
Machine learning hardware is a rapidly growing field with significant potential for practical applications. Machine learning is revolutionizing various disciplines by providing new approaches to solving complex problems. The speaker invites questions from the audience.