Background:
Tommaso Poggio, director of CBMM, organized a symposium on “Brains, Minds, and Machines” for MIT’s 150th anniversary in 2011. The final day of the symposium focused on the “marketplace for intelligence,” with participation from major companies like IBM, Microsoft, Google, and startups.

DeepMind’s Early Vision:
One of the startups invited was DeepMind, co-founded by Demis Hassabis, a former postdoc at CBMM. Demis shared DeepMind’s unofficial business plan: to create the first AI in the virtual world of games.

DeepMind’s Achievements in Games:
Demis returned to CBMM several times to update on DeepMind’s progress, most recently after AlphaGo’s victory over Lee Sedol in Seoul. DeepMind has surpassed all expectations in the virtual world of games and may now declare victory in this domain.

The Challenge of Intelligence:
Tommaso Poggio emphasizes that intelligence is the greatest problem in science today and tomorrow. Solving it will require multiple breakthroughs and Nobel Prizes, not just a single solution.

Neuroscience and Cognitive Science as Key Disciplines:
Both DeepMind and CBMM believe that neuroscience and cognitive science are essential for understanding human intelligence and developing intelligent machines. The journey to achieving this goal may be longer than many people expect.

Framing AI:
AI approaches can be categorized into two types: expert systems and learning systems. Expert systems rely on hard-coded knowledge and logic, but they struggle with the unexpected and are limited in scope. Learning systems can learn solutions to problems from first principles, potentially generalizing to new tasks and surpassing human capabilities.

Inspiration from the Brain:
The brain is the best learning system we have, and understanding it can inspire new algorithms, representations, and architectures for AI systems. CBMM and DeepMind are both working on developing self-learning systems inspired by neuroscience.

Inspiration and Validation:
The brain can inspire and validate algorithms developed using mathematical, physical, and neuroscience approaches. Reinforcement learning, a technique pushed forward in the 80s and 90s, has been implemented in the brain, suggesting its plausibility as part of an overall AI solution. Validation from the brain provides guidance and confidence in engineering efforts, helping researchers decide where to allocate more effort.

AlphaGo’s Impact:
AlphaGo’s success in the game of Go overturned traditional thinking and surprised AI experts. It demonstrated the potential of deep learning for complex tasks and highlighted the importance of self-play and reinforcement learning.

AlphaZero: A More General Approach:
AlphaZero is the latest incarnation of DeepMind’s AlphaGo program, aiming for generality across domains. It eliminates the need for human games for bootstrapping, relying solely on self-play and reinforcement learning. AlphaZero Zero started from random play and surpassed the original AlphaGo’s strength through self-play.

Removing Go-specific Knowledge:
AlphaGo Zero represents a significant step towards generality by removing the need for Go-specific knowledge. This is important for real-world applications where large amounts of human data may not be readily available. The goal is to build a maximally general system with minimal assumptions that can perform well across various domains.

AlphaZero Overview:
AlphaZero is a groundbreaking AI system developed by DeepMind, capable of playing any two-player perfect information game to world champion or higher standard. AlphaZero builds upon the success of AlphaGo, but generalizes its approach to any game with perfect information, such as chess, Go, and shogi.

AlphaZero’s Training Process:
AlphaZero utilizes self-play reinforcement learning, where it plays against itself to improve its strategies. The system starts with a neural network with no knowledge of the game and learns solely through self-play. AlphaZero generates synthetic training data through extensive self-play games, creating a large training corpus.

Neural Network Architecture:
AlphaZero employs a single neural network that combines the policy network and value network into one. The policy network narrows down the likely moves in a given position, while the value network evaluates the current position and predicts the probability of winning.

Monte Carlo Tree Search:
AlphaZero uses Monte Carlo tree search to efficiently explore the game tree and make decisions. The policy network guides the search to focus on the most promising moves, reducing the search width and depth. The value network helps evaluate positions during the search, allowing for early truncation of branches.

Training Regime:
AlphaZero iteratively trains the neural network based on the synthetic training data generated from self-play games. Every 100,000 self-play games, a new neural network is trained using the combined old and new training data. The new neural network is evaluated against the old one, and if it wins more than 55% of the time, it replaces the old network.

Computational Requirements:
AlphaZero’s training process requires significant computational resources. In the case of chess, AlphaZero was trained using 40 million games, played in about three seconds each, on 5,000 TPUs. The training time can be adjusted by varying the number of games played per batch and the amount of time allocated for each game.

Conclusion:
AlphaZero represents a remarkable advancement in artificial intelligence, demonstrating the power of self-play learning to achieve superhuman performance in complex two-player games. Its success highlights the potential of general-purpose AI systems that can learn and adapt to various domains without extensive human knowledge or handcrafted rules.

AlphaZero’s Dominance in Chess:
AlphaZero, an AI developed by DeepMind, convincingly defeated Stockfish 8, the World Chess Champion, in a 1,000-game match. With 155 wins, six losses, and the rest draws, AlphaZero demonstrated its superior strength. Starting from random, AlphaZero surpassed Stockfish’s performance in just four hours, indicating its exceptional learning efficiency.

Success in Other Games:
AlphaZero’s capabilities extend beyond chess. It defeated the world’s best Shogi program in a couple of hours. It also surpassed AlphaGo, the Go-playing AI, in eight hours of training using a new and more efficient architecture.

Potential for Mastery in Two-Player Perfect Information Games:
AlphaZero’s success suggests its potential to master any two-player perfect information game. This aligns with the childhood dream of Demis Hassabis, AlphaZero’s creator, who envisioned a master games player capable of solving various domains.

Search Efficiency and Comparison to Human Grandmasters:
AlphaZero exhibits a different level of search efficiency compared to traditional chess engines and human grandmasters. AlphaZero examines tens of thousands of moves before making a decision, striking a balance between brute force search and sample efficiency. Human grandmasters, on the other hand, consider a few hundred moves, demonstrating remarkable efficiency.

AlphaZero’s Strengths Even Without Search:
Without any search, AlphaZero’s evaluation function alone allows it to play at an international master standard. This indicates AlphaZero’s deep understanding of the game, even without relying on extensive search.

AlphaZero’s Unique Playing Style:
AlphaZero’s playing style differs significantly from traditional chess engines. It prioritizes mobility over materiality, leading to dynamic and aesthetically pleasing games. This style has rekindled the passion for chess among enthusiasts and inspired new strategies.

Zugzwang and the Hermetic Sealing of Stockfish:
AlphaZero’s victory over Stockfish showcases its mastery of the game’s intricacies. In one notable position, AlphaZero sacrifices pieces to gain mobility, effectively sealing off Stockfish’s queen and rendering it immobile. This hermetic sealing demonstrates AlphaZero’s ability to plan intricate strategies and foresee long-term consequences.

AlphaZero’s Advantages in Chess:
AlphaZero does not have hard-coded rules about piece values, allowing it to make unconventional sacrifices and value pieces based on context. It can take into account complex factors like pawn structure, king safety, and outpost control, which are difficult to balance with traditional handcrafted rules.

AlphaZero’s Impact on Chess:
AlphaZero discovered seven new themes in chess that had not been seen before in professional play. These new ideas have been documented in a book called “Game Changer,” which explores their significance and implications for chess strategy.

Challenges in Real-Time Strategy Games:
Real-time strategy games like StarCraft II present new challenges for AI, including partial observability, a vast action space, long-term dependency, and a dynamic game environment.

AlphaStar’s Architecture:
AlphaStar utilizes three feed-forward networks to process spatial observations, economic data, and unit information. A deep LSTM serves as the core of the system, handling temporal dependencies and producing a function head that outputs actions.

AlphaStar’s Training:
AlphaStar League is a population-based training method that involves creating a league of diverse competitors and using self-play and reinforcement learning to improve the AI’s performance. The Nash of the league is used to select the best strategies for competing against top human players.

Intrinsic Motivation in AlphaStar:
Intrinsic motivation is introduced to increase diversity in AlphaStar’s strategies. The AI is rewarded for achieving specific goals, such as building certain units or defeating specific opponents, encouraging it to explore different strategies.

Evolving Strategies in AlphaStar:
AlphaStar’s strategies evolve over time as different approaches become dominant in the league. This dynamic behavior reflects the ever-changing nature of StarCraft II and the AI’s ability to adapt and discover new tactics.

AlphaStar’s Performance Against Top Professionals:
After developing AlphaStar, DeepMind invited two top StarCraft II professionals, TLO and Mana, for a series of closed-door matches. AlphaStar won both matches with a score of 10-0, demonstrating its exceptional strength. The replays of these matches were released and received significant attention online.

AlphaStar’s Progress and Future Challenges:
AlphaStar’s performance represents a significant milestone in DeepMind’s journey in mastering games. The company believes that it has solved many of the fundamental challenges in games, including Atari games, Go, chess, and now StarCraft II. However, there are still other types of games, such as diplomacy, that require additional research and development.

Insights from AlphaStar’s Development:
The development of AlphaStar provided valuable insights into the nature of these systems. Early versions of AlphaGo, for example, exhibited a tendency to make moves that were technically correct but lacked creativity. AlphaStar, on the other hand, demonstrated more flexible and creative play, similar to human players.

AlphaGo’s Loss and Debugging:
AlphaGo lost game four against Lisa Doll due to a confusing move that exposed a weakness in its evaluation system. Fixing this weakness was challenging because AlphaGo is a self-learning system, and traditional patching methods were not applicable. This raises questions about debugging self-learning systems and the emergence of new paradigms for computer science.

Knowledge Search Space and Understanding:
Understanding the extent of AlphaGo’s knowledge coverage and the implicit knowledge it has acquired is a challenge. There is a need for mathematical methods to describe and quantify the knowledge search space. Understanding how AlphaGo makes decisions and the nature of its implicit knowledge is important for advancing the field.

Creativity in AlphaGo:
AlphaGo exhibited aspects of creativity, such as coming up with new moves that even human players had not considered. This can be categorized as extrapolation, but full out-of-the-box innovation and invention of new games are still beyond AlphaGo’s capabilities.

Challenges and Future Directions:
Many interesting challenges remain in the field of AI, including unsupervised learning, memory one-shot learning, imagination-based planning, abstract concept learning, transfer learning, and language understanding. These challenges need to be addressed to achieve full Artificial General Intelligence (AGI).

Games as a Training Ground for AI:
Games and simulations are valuable training grounds for AI, allowing for testing and development of algorithms. The ultimate goal is to develop general solutions that can be applied to real-world problems.

Real-World Applications of AI:
Powerful and mature AI algorithms are already proving useful in various real-world domains, such as healthcare, energy, personalized education, and virtual assistance. The potential applications of AI are vast and hold great promise for addressing various challenges.

AI for Scientific Discovery:
AI can be a powerful tool for scientific discovery, helping scientists understand the universe better. Current AI systems can be applied to various scientific problems, particularly those involving massive combinatorial search and clear objective functions. Key characteristics of problems suitable for AlphaZero-like approaches include massive combinatorial search nature, clear objective functions, and the ability to generate and evaluate many solutions efficiently.

DeepMind’s Expanding Research Team:
DeepMind is investing in a science team focused on utilizing AI for solving complex problems. The team aims to grow from 30-40 members to approximately 100, inviting interested individuals to apply. Various fields of application include genomics, theorem proving, quantum chemistry, and more.

Successful Applications of AI in Diverse Domains:
AI has been employed successfully in various fields, including exoplanet discovery, nuclear fusion control, healthcare diagnostics, and chemical synthesis. Collaborative efforts with Google have led to the discovery of new planets. Fusion reactor plasma control and diagnosing macular degeneration are among the notable applications.

AlphaFold: Tackling the Protein Folding Problem:
AlphaFold, DeepMind’s program, aims to solve the protein folding problem, a fundamental challenge in biology. Proteins are essential building blocks of life, and their 3D structure determines their function. Predicting the 3D structure from a 1D amino acid sequence is a crucial task.

Proteins as Molecular Machines:
Proteins are remarkable molecular machines with intricate functions. Videos visualizing proteins in action provide insights into their biomechanical processes. Examples include an enzyme in mitochondria producing ATP and a calcium pump regulating muscle contraction.

Significance of Solving Protein Folding:
Understanding protein folding enhances our knowledge of disease mechanisms and drug discovery. Crystallography, the traditional method for protein structure determination, is time-consuming and not applicable to all proteins. Solving protein folding would revolutionize disease understanding, drug discovery, and synthetic protein design.

The AlphaFold System:
AlphaFold is a neural network-based system trained to predict protein structures from their amino acid sequences. It takes the amino acid sequence as input and outputs predictions of angles and a distogram (pairwise distance matrix) of the protein. The system is trained on a database of 30,000 known protein structures with their corresponding sequences.

Structure Optimization:
Once the neural network is trained, a new protein sequence is input, and the system outputs angle and distogram distributions. Structure optimization is performed using a quasi-Newton method to find a candidate structure that minimizes a total score. The total score includes the angle and distogram scores, as well as a chemistry score to prevent atoms from overlapping.

Testing AlphaFold:
AlphaFold was tested in CASP-13, a biannual protein folding competition involving over 100 research groups worldwide. In CASP-13, AlphaFold was given amino acid sequences of proteins whose structures were not yet published. Teams had three weeks to submit predicted structures for each sequence.

CASP-13 Results:
AlphaFold achieved unprecedented accuracy in predicting protein structures in CASP-13. It outperformed all other competing methods by a significant margin. AlphaFold’s predictions were close to experimental structures, even for proteins with complex structures.

Groundbreaking Protein Folding Results:
DeepMind’s team, led by John Jumper, achieved remarkable success in a protein folding competition, winning 25 out of 43 proteins in the hardest category. Their predictions closely matched the ground truth, significantly outperforming other teams.

AI’s Impact on Science:
Hassabis emphasizes the potential of AI to address information overload and system complexity, especially in scientific research. He envisions AI-assisted science that accelerates breakthroughs and enables scientists to derive useful knowledge from vast amounts of data.

Challenges and Limitations:
Despite the impressive results, Hassabis acknowledges that the protein folding problem is still far from solved for practical use in biology. The goal is to achieve an error tolerance of one angstrom, which remains a significant challenge for current AI systems.

Responsible and Ethical Development of AI:
Hassabis stresses the importance of responsible and ethical development of AI, ensuring that it benefits society and is deployed safely. He calls for more research and discussion on the impact of AI, including issues like robustness, bias, and safety.

Neuroscience Inspiration and Understanding the Mind:
Hassabis sees neuroscience as a valuable inspiration for AI development, potentially leading to a better understanding of the human mind. He believes that distilling AI and intelligence into algorithmic constructs can facilitate comparisons with the human mind, shedding light on mysteries like creativity, dreaming, and consciousness.