Origins of Interest in the Brain:
At high school, Hinton’s classmate introduced him to the concept of holograms and their potential role in memory storage in the brain, sparking his initial interest in understanding how memories are stored.

Exploring Different Fields:
Hinton began his academic journey studying physiology and physics at Cambridge University. Later, he switched to philosophy but found it lacking in methods for distinguishing false statements. Eventually, he settled on psychology but was dissatisfied with the simplistic theories used to explain brain functions.

Transition to AI and Neural Networks:
After a brief stint as a carpenter, Hinton decided to pursue AI. He joined the AI program at the University of Edinburgh under Christopher Longuet-Higgins, who had previously worked on neural networks. Despite Longuet-Higgins’ initial discouragement, Hinton persisted in his belief in neural networks, leading to disagreements and debates.

PhD and Moving to California:
Hinton completed his PhD in AI but faced difficulties finding a job in Britain due to the skepticism towards neural networks. He secured a Sloan Fellowship in California, which provided an opportunity to pursue his research interests in a more receptive environment.

California’s Openness to Neural Networks:
In contrast to Britain’s skepticism, California offered a welcoming atmosphere for exploring neural networks and their potential applications. Researchers like Don Norman and David Ronald Hart were open to Hinton’s ideas, fostering a supportive environment for his work.

Backpropagation and the Inflection of Community Acceptance:
Geoffrey Hinton, David Rommelhart, and Ron Williams developed the backpropagation algorithm in 1982. Backpropagation was not novel; others had invented it independently, but Hinton’s paper in Nature in 1986 was influential in its acceptance by the community. The paper’s success may be attributed to Hinton’s political work and his ability to impress the referee, Stuart Sutherland, with the algorithm’s ability to learn representations for words and extract meaningful features. This work unified two different views of knowledge: the psychological view of concepts as bundles of features and the AI view of concepts as structural relationships.

Early Word Embeddings and Learned Features:
The backpropagation example showed that a graph structure or family tree could be converted into feature vectors, which could then be used to derive new consistent information, demonstrating generalization. This to-and-fro between graphical and feature vector representations was significant.

Bengio’s Contribution to Word Embeddings:
In the early 1990s, Bengio applied the same techniques to real data, such as English text, to obtain embeddings for real words.

The Role of Computing Power in Deep Learning’s Advance:
Hinton highlights the significant increase in computing power from the 1980s to the 1990s, which facilitated the use of neural networks.

Hinton’s Favorite Inventions:
Hinton considers the work he did with Terry Synofsky on Baltimore machines to be his most beautiful invention. The learning algorithm they discovered was simple, applied to densely connected nets with hidden representations, and resembled the kind of learning that could occur in the brain. Restricted Boltzmann machines, derived from Baltimore machines, were effective in practice and contributed to the resurgence of neural networks and deep learning in the late 2000s.

Deep Restricted Boltzmann Machines:
Hinton’s work on deep restricted Boltzmann machines involved training a restricted Boltzmann machine with one layer of hidden features, treating those features as data, and repeating the process multiple times. This approach was practical and paved the way for treating the entire model as a single entity.

Early Research on Neural Networks:
Hinton’s work on sigmoid belief nets led to an efficient way of doing inference in densely connected neural nets, bridging the gap between neural nets and graphical models. The deep belief nets Hinton and his colleagues developed had a bound that improved with each added layer, guaranteeing better performance. Hinton’s collaboration with Bradford Neal on variational methods significantly improved the EM algorithm, making it more widely applicable. In 1993, Hinton and Van Kamp published a paper on variational Bayes, providing a tractable approach to Bayesian learning in neural networks.

The Development of ReLU Activations:
Hinton’s research with restricted Boltzmann machines showed that ReLU activations were almost equivalent to a stack of logistic units. This finding helped popularize ReLUs, as it demonstrated that the learning algorithm could work with ReLUs, and the math supported their use. Hinton later realized that initializing ReLU networks with the identity matrix allowed for efficient training of networks with many hidden layers.

Backpropagation and the Brain:
Hinton’s current thoughts on the relationship between backpropagation and the brain are not discussed in this segment.

Backpropagation Implementation in the Brain:
Evolution could have developed a biological implementation of backpropagation. The complexity of cells suggests that implementing backpropagation is plausible. The brain likely employs an algorithm similar to backpropagation.

Recirculation Algorithm:
Developed in 1987 with Jay McClelland. Trains an autoencoder without backpropagation. Learning rule adjusts weights based on presynaptic input and postsynaptic input change rate. Similar to spike-time-dependent plasticity.

Deep Boltzmann Machines and Reconstruction Error:
In a stack of restricted Boltzmann machines, trained weights allow for backpropagation implementation. Reconstruction error provides the derivative of discriminative performance. This idea was initially ignored but later explored by Yoshio Bengio and Geoffrey Hinton.

Fast Weights for Recursion:
Proposed in 1973 during Hinton’s graduate studies. Fast weights adapt rapidly but decay quickly, enabling short-term memory. True recursion with weight and neuron reuse. Memory storage in fast weights during recursive calls. Recent work with Jimmy Barr in 2015-2016 demonstrated this concept.

Capsules:
Hinton’s current focus, despite rejections. Hinge on the idea of equivariance, where transformations of input do not alter output. Aims to address the limitations of convolutional neural networks.

Distributed Representation of Features:
Represent multidimensional entities using a small vector of activities, assuming only one of that feature exists in a region. Use multiple neurons to represent different aspects of the feature, such as coordinates, orientation, speed, color, brightness, etc.

Capsule as a Feature Instance:
A capsule represents an instance of a feature with multiple properties. Unlike neurons in traditional neural networks, capsules have a vector of properties instead of a single scalar property.

Routing by Agreement:
Capsules vote for the parameters of a higher-level feature based on their own parameters. Agreement among capsules in a high-dimensional space indicates they likely represent the same entity. This mechanism is crucial for generalization, viewpoint changes, segmentation, and statistical efficiency.

Discriminative Training with Iteration:
Capsule networks can be trained discriminatively using supervised learning. The forward pass involves iterative refinement of feature groupings, such as determining whether a mouth and nose belong to the same face. Backpropagation can be applied to train the entire process.

Research and Development:
Geoffrey Hinton is currently leading a team in Toronto, part of the Google Brain team, to develop and refine capsule networks. Research is ongoing to bring this technology to fruition.

His Intellectual Journey in Deep Learning:
Early focus on backpropagation and discriminative learning in the 1980s. Shift towards unsupervised learning in the 1990s, influenced by Yann LeCun’s ideas. Recent emphasis on supervised learning due to its practical success, while acknowledging the importance of unsupervised learning in the long run.

Unsupervised Learning and Generative Models:
Excitement about the potential of unsupervised learning for future breakthroughs. Variational autoencoders and generative adversarial nets (GANs) seen as promising approaches. GANs considered a major breakthrough in deep learning.

Critique of Sparsity and Slow Features:
Less emphasis on sparsity compared to Yann LeCun’s approach. Slow features deemed a mistake; advocating for features that change in predictable ways.

Guiding Principle for Modeling:
Transform observables to a state vector where linear operations can effectively manipulate underlying variables. Example of changing viewpoints by converting pixels to coordinates, applying a matrix multiplication, and mapping back to pixels.

Advice for Aspiring Researchers:
Read the literature, but don’t overdo it; focus on developing intuitions. Embrace contrarian thinking and challenge established norms. Trust your intuitions and persevere despite skepticism. Replicate published papers to gain practical insights and identify crucial details. Continuously engage in programming to develop a deep understanding of implementation intricacies.

Finding Great Research Ideas:
Hinton encourages researchers to pursue ideas they believe in, even if others dismiss them as nonsense. He cites the example of variational methods, which were initially met with skepticism but later gained acceptance.

Choosing an Advisor:
Hinton advises new graduate students to find an advisor whose beliefs align with their own. This ensures that students receive valuable guidance and support for their research.

Research Topics for New Grad Students:
Hinton suggests that new grad students consider working on topics such as capsules and unsupervised learning.

Advice for Learners:
Hinton acknowledges the challenge of choosing between pursuing a PhD or joining a top company or research group. He believes that currently, there is a shortage of academics trained in deep learning, leading to a reliance on big companies for training.

Academia’s Slow Response to the Deep Learning Revolution:
Hinton criticizes computer science departments for being slow to recognize the significance of deep learning. He argues that they need to adapt to the changing landscape of computing, where showing computers is as important as programming them.

The Role of Big Companies in Training:
Hinton acknowledges the role of big companies like Google in training individuals in deep learning, given the slow response from academia. He expresses hope that universities will eventually catch up and provide adequate training in this field.

Deep Learning Specializations and MOOCs:
Andrew Ng highlights the popularity of deep learning specializations and MOOCs, including Hinton’s pioneering deep learning MOOC on Coursera in 2012. Hinton acknowledges Ng’s role in encouraging him to create the MOOC and expresses gratitude for its positive impact on learners.

Early AI Paradigms:
In the early days, pioneers like von Neumann and Turing favored brain-inspired AI over symbolic AI. Symbolic AI dominated early AI research, with a focus on logic-based representations for intelligence.

A New View of Thought:
Hinton challenges the symbolic AI view, proposing that thoughts are not symbolic expressions but rather “great big vectors of neural activity.” This perspective contrasts with the idea of thoughts as strings of words or symbolic representations.

Causal Powers of Thought Vectors:
Hinton emphasizes the causal powers of these thought vectors, asserting that they cause other thought vectors. This differs significantly from the standard AI view that thoughts are symbolic expressions.

Symbolic AI Persistence:
Despite the emergence of the new view, Hinton acknowledges that many in the AI community still adhere to the symbolic AI paradigm.

Gratitude for the Opportunity:
Andrew Ng expresses gratitude to Hinton for sharing his insights and driving the evolution of deep learning. Hinton appreciates the opportunity to share his perspectives.