Introduction of Demis Hassabis:
Demis Hassabis is a renowned individual with diverse expertise, including being a chess prodigy, studying computer science at Cambridge, founding successful computer game companies, obtaining a PhD in neuroscience from UCL, and holding a brief postdoc position under Tomaso Poggio.

DeepMind’s AI Development Approach:
Demis will share an overview of DeepMind’s approach to AI development, delving into the philosophy behind their methodologies.

The Culmination of AlphaGo:
The second half of Demis’s talk will focus on AlphaGo, a groundbreaking achievement in AI that showcases the company’s work.

Future Plans for AlphaGo:
Demis will discuss their aspirations for the future of AlphaGo and its potential applications.

DeepMind’s Background:
DeepMind was founded in 2010 and later joined forces with Google in 2014. It has grown into a large team of over 200 research scientists and engineers.

Demis Hassabis’s Vision:
He views DeepMind as an “Apollo program for AI,” aiming to fundamentally solve intelligence and subsequently use it to address various challenges. He believes AI is one of the most critical and powerful technologies mankind can create.

General Purpose Learning Algorithms:
DeepMind’s focus is on developing general purpose learning algorithms. These algorithms can learn automatically from raw inputs and experiences without any pre-programming.

Artificial General Intelligence (AGI):
DeepMind’s AGI is flexible, adaptive, and potentially inventive. AGI is designed to handle unexpected situations and adapt to unseen scenarios.

Narrow AI vs. AGI:
Narrow AI is handcrafted for specific purposes and lacks flexibility. AGI, in contrast, can operate across a wide range of tasks and handle unexpected situations.

Deep Blue as an Example of Narrow AI:
Deep Blue, a famous chess-playing computer, exhibited narrow AI. It was limited to playing chess and could not perform other tasks.

Reinforcement Learning:
DeepMind’s approach to AI and intelligence is through reinforcement learning. Agents interact with an environment, receiving observations and rewards. The goal is to build an accurate model of the environment and select actions that lead towards a desired goal.

Deep Reinforcement Learning:
DeepMind’s approach to AI combines deep learning with reinforcement learning, allowing RL to tackle challenging problems at scale. Reinforcement learning involves an agent interacting with an environment, receiving rewards for positive actions and penalties for negative ones, and learning to maximize rewards. DeepMind’s DQN algorithm, a deep reinforcement learning system, can play Atari games without prior knowledge of rules or game structure, achieving human-level performance on most games.

Grounded Cognition:
DeepMind believes a true thinking machine must be grounded in a rich sensory-motor reality, whether through physical robots or virtual worlds. Virtual worlds can be used as testing grounds for AI algorithms, providing unlimited training data and independent benchmarks. Games, designed for human players, offer a challenging and diverse environment for AI development.

Systems Neuroscience:
DeepMind incorporates systems neuroscience to understand the brain’s algorithms, representations, and architectures, focusing on the computational level rather than low-level synaptic details. Areas of focus include memory, attention, concepts, planning, navigation, and imagination, aiming to go beyond the success achieved with Atari games.

Introduction of Neural Turing Machines (NTM):
The concept of creating an artificial hippocampus, a brain structure involved in memory and navigation, as a potential solution for incorporating memory into neural networks. NTMs, a novel neural architecture, aims to address the need for large-scale, controllable memory capabilities in neural networks.

Key Features of NTMs:
NTMs comprise a recurrent neural network, analogous to a CPU, and an extensive memory store, akin to a K-N memory system. This architecture allows for learning and manipulation of memory elements through gradient descent.

Symbolic Reasoning with NTMs:
NTMs open up new avenues for symbolic reasoning, considered a holy grail in AI. Tackling classic AI problems, such as the Shrewsbury class of problems, which involve manipulating blocks and answering questions about the scene.

Mini Shroudelou: A Simplified Blocks World Problem:
Due to the complexity of the full Shrewsbury class of problems, researchers introduced a simplified 2D version called mini Shroudelou. In mini Shroudelou, a neural network agent learns to manipulate colored blocks in a vertical tower to match a target configuration, demonstrating problem-solving capabilities.

Training and Testing of NTMs:
NTMs are trained using reinforcement learning, allowing them to improve over time by trial and error. Once trained, NTMs are tested on unseen scenarios, evaluating their ability to solve problems optimally.

Solving Logic Puzzles and Graph Problems:
NTMs have shown promising results in solving logic puzzles, demonstrating their capacity for symbolic reasoning. Ongoing research explores the application of NTMs to graph problems, a general class of problems with broad applications.

Upcoming Publications on NTMs:
The team is preparing a significant publication later in the year, building upon their previous work on NTMs. This publication is expected to provide further insights and advancements in the field.

Neural Turing Machine and Sri Lalu Tasks:
DeepMind is experimenting with incorporating a simplified version of language into Sri Lalu tasks, where a neural Turing machine reads and remembers constraints given in code to solve puzzles.

Moving to 3D Environments:
DeepMind has repurposed the Quake 3 engine, calling it Labyrinth, to tackle navigation and 3D vision problems in a virtual environment. An agent learns to navigate, collect rewards, and find exits solely through pixel inputs.

Towards a Rat-Level AI:
DeepMind aims to create an AI agent capable of performing various tasks that rats can do, drawing inspiration from experimental ideas and tests from rat literature.

AlphaGo and the Challenge of Go:
AlphaGo combines neural networks with planning approaches to master the complex game of Go. Go’s profound complexity stems from its simple rules, vast possible board configurations, and the need for both intuition and calculation.

Go’s Challenges for Computers:
The huge branching factor (200 moves per position) and the lack of an evaluation function make brute force search and handcrafting rules for evaluation difficult. Go is a constructive game with no materiality concept, requiring prediction and susceptibility to local changes. Intuition and calculation are crucial for Go mastery, pushing the limits of human capabilities.

Divine Moves and Intuitive Play in Go:
Top Go players often describe brilliant moves as feeling “right” rather than being calculated. Go has a history and tradition of intuition and a concept called “divine moves.” Famous games and moves are passed down, inspiring players to achieve their own divine moves.

Training AlphaGo with Deep Neural Networks:
AlphaGo used two deep neural networks: a policy network and a value network. The policy network was trained on human expert data to predict human moves. Reinforcement learning was used to improve the policy network’s win rate through self-play. The value network was trained on game data to evaluate board positions and predict the winner.

Policy Network and Value Network:
The policy network takes a board position as input and outputs a probability distribution over possible moves. The value network takes a board position as input and outputs a single real number indicating the probability of white or black winning.

Monte Carlo Tree Search with Neural Networks:
Monte Carlo Tree Search is used to plan moves by expanding the search tree and evaluating positions. The policy network is used to narrow the search width by suggesting probable moves. The value network is used to evaluate the desirability of positions, reducing the search depth. AlphaGo combines value network evaluation with Monte Carlo rollouts for accurate position assessment.

Neural Networks Reduce Search Complexity:
The policy network reduces the width of the search tree by suggesting probable moves. The value network reduces the depth of the search tree by providing an evaluation of board positions. This combination of neural networks makes the search more efficient and accurate.

Performance against Commercial Go Programs:
AlphaGo was tested against CrazyStone and Zen, the two leading commercially available Go programs. AlphaGo achieved an impressive 494-1 win record against these programs. Even when given a four-stone handicap, AlphaGo maintained a 75% win rate. AlphaGo’s single-machine version outperformed the distributed versions of the competitor programs.

ELO Rating System:
A numerical ELO ranking system, similar to the one used in chess, was developed to assess the strength of Go programs. A gap of 200-250 ELO points corresponds to an 80% win rate.

AlphaGo’s Superiority:
AlphaGo surpassed the other best programs by more than 1,000 ELO points.

Challenge to Human Players:
After defeating the top computer programs, AlphaGo was ready to face human opponents. Fan Hui, a Chinese-born Go player based in France, was selected as the first human challenger.

AlphaGo’s 5-0 Victory Against a Professional Go Player:
AlphaGo, a computer program developed by Demis Hassabis and his team, achieved a groundbreaking victory against a professional Go player, becoming the first program to reach professional status in the game. The defeated player, a two-dan professional and three-time European champion, initially preferred a slow and strategic approach but switched to a more aggressive style in subsequent games, resulting in a 5-0 loss to AlphaGo.

Expert’s Perspective on AlphaGo’s Victory:
AI experts and top programmers had predicted that this milestone would take at least another decade to achieve, making AlphaGo’s victory a significant surprise. The Go world, including professional players, believed that it would take much longer for a computer program to reach professional levels in the game.

AlphaGo’s Impact on the Professional Go Player:
After the match, the defeated player joined AlphaGo’s team as a consultant, contributing to the program’s development. The player’s experience playing against AlphaGo led to a significant improvement in his own skills, moving him up in the world rankings. The player attributed his improvement to AlphaGo’s unique style, which freed his mind from traditional constraints and allowed him to think creatively about the game.

AlphaGo’s Match Against Lee Sedol:
AlphaGo’s next challenge was to face Lee Sedol, a legendary Go player known as the “Roger Federer of Go.” Lee Sedol had won 18 world titles and was renowned for his creative style and brilliance. The match between AlphaGo and Lee Sedol took place in early March in Korea, with a million-dollar first prize at stake.

Key Findings from AlphaGo’s 4-1 Match Win against Lee Sedol:
AlphaGo’s Compute Power: Despite using the same compute power as in the Fan Hui match, AlphaGo’s strength only showed a modest increase, suggesting diminishing returns and the limited effectiveness of parallelization. AlphaGo’s Drastic Improvement: AlphaGo’s new version was significantly stronger than the old version, winning 99.9% of matches. This vast improvement occurred within five months. AlphaGo’s Honorary 9 Dan Certificate: The Korean Go Association recognized AlphaGo’s creative play by awarding it an AlphaGo with an honorary 9 Dan certificate.

AlphaGo’s Creativity and Intuition:
AlphaGo’s Move 37 in Game Two: AlphaGo made an astounding move, defying conventional wisdom in Go, by playing on the fifth line instead of the traditional third or fourth lines. The Significance of Move 37: This move highlighted AlphaGo’s ability to think strategically and plan long-term, as its impact became evident 50 moves later, influencing a fight in the bottom left corner. Reactions to Move 37: Commentators and professionals were astounded by AlphaGo’s move, with some believing it was a misclick due to its unconventional nature. Statistics Behind Move 37: The policy network gave the prior probability of this move as less than 1 in 10,000, indicating that AlphaGo did not learn it from professional games.

AlphaGo’s Originality and Potential:
AlphaGo’s Original Move: Many professionals commented that Move 37 was not a human move, emphasizing AlphaGo’s ability to generate novel strategies. AlphaGo’s Potential: AlphaGo’s capacity for original thought and strategic planning suggests its potential to contribute to scientific and creative fields beyond Go.

AlphaGo’s Surprising Moves:
AlphaGo displayed originality by making moves that were unexpected even to its developers. It favors influence in the center of the board and values it so much that it considers fifth line influence to be a good trade. This move challenges conventional wisdom in the game of Go and may lead to a reevaluation of acceptable trades.

Lee Sedol’s Win in Game 4:
Lee Sedol won Game 4 by playing an incredible move with a probability of less than 1 in 10,000. AlphaGo had to start its pondering and search again, leading to a mis-evaluation in the value net. This move highlights Lee Sedol’s creativity and fighting spirit, making him one of the top game players Demis Hassabis has met.

Cultural Impact of the Match:
The match attracted 280 million viewers worldwide, surpassing the Super Bowl’s viewership. There was widespread media coverage, with 35,000 press articles daily and front-page features in Korean newspapers. The match popularized Go in the West, leading to a shortage of Go boards and increased interest in the game.

AlphaGo’s Progress:
AlphaGo’s progress has been remarkable, with a one-rank improvement per month over the last 18 months. The project uses techniques that create more data, train new versions, and deliver higher-quality data. The rate of progress has been astonishing, and it remains to be seen how far it can go before reaching optimal play.

AlphaGo’s Impact on the Game of Go:
AlphaGo has revitalized the game of Go and brought new ideas and creativity to the game, potentially improving the overall standard of Go. Professional Go players are excited about AlphaGo’s release to the public, believing it will further enhance their understanding and enjoyment of the game.

Comparison of AlphaGo with Deep Blue:
Unlike Deep Blue, which relied on handcrafted chess knowledge, AlphaGo learns solely from expert games and self-play, demonstrating its ability to acquire knowledge without explicit programming. AlphaGo’s search algorithm is highly selective, guided by two neural networks, allowing it to evaluate 100,000 positions per second compared to Deep Blue’s 200 million positions per second.

Defining Intuition and Creativity in the Context of AlphaGo:
Intuition in Go is considered implicit knowledge gained through experience, not consciously accessible or expressible but verifiable through behavioral output. Creativity in Go involves synthesizing accumulated knowledge to produce novel or original ideas. AlphaGo demonstrates both intuition and creativity within the constrained domain of Go.

DeepMind’s Goals Beyond AlphaGo:
DeepMind aims to apply the general-purpose technologies developed for AlphaGo to solve challenging real-world problems. Areas of interest include healthcare, robotics, and personal assistance.

Future Developments and Challenges:
Exploring the possibility of a group of top professional Go players competing against AlphaGo, leveraging their collective strengths in different aspects of the game. Developing new analysis, statistical, and visualization tools to better understand the inner workings of deep learning systems like AlphaGo. DeepMind’s Virtual Brain Analytics project seeks to create tools to analyze and visualize the representations formed by neural networks.

Openings for Collaboration:
DeepMind is actively hiring research scientists and software engineers to join their team and contribute to their ongoing work.

New Experiments in Reinforcement Learning:
Researchers at DeepMind plan to conduct an experiment to train an AI system from scratch using reinforcement learning, without the use of supervised learning or human expert play. This experiment aims to determine if it is possible for an AI system to achieve expert-level performance through reinforcement learning alone.

Challenges of Generalizing AI from Games:
There is a risk that AI systems trained on games may not generalize well to real-world applications. It is crucial to ensure that the AI system only has access to information that it would have access to in the real world. DeepMind’s evaluation team is separate from the algorithm development teams and responsible for creating the environments and APIs for the AI systems. This ensures that the AI systems cannot access information they are not supposed to.

Approaches to AI Development:
DeepMind is pursuing both self-training and research in new architectures and parameters to develop AI systems. The company is utilizing a combination of approaches to achieve the best results.