Geoff Hinton’s Background and Contributions:
Geoff Hinton is a pioneering figure in the field of deep learning, often referred to as the godfather of the field. He earned a BA in experimental psychology from Cambridge and a PhD in artificial intelligence from Edinburgh. Hinton’s early work revolved around neural networks, both as models of the mind and as tools for artificial intelligence.

Hinton’s Career and Achievements:
Hinton obtained his first tenure-track position at Carnegie Mellon in the 1980s. He later joined the University of Toronto in the late 1980s due to ethical objections to DARPA funding. Throughout the 1990s, Hinton continued to champion neural networks and machine learning. He briefly moved to University College London in 1998 to establish the Gatsby Computational Neuroscience Unit. In 2001, Hinton returned to Toronto, becoming a university professor in 2006 and an emeritus professor in 2014.

Recognition and Impact:
Hinton’s contributions have earned him widespread recognition, including the prestigious Turing Award, the Order of Canada, and a distinguished fellowship from the Canadian Institute for Advanced Research. His work has garnered over 270,000 citations, surpassing those of Einstein, Ramon y Cajal, and Alan Turing combined. Hinton’s influence on other scientists is remarkable, with many of his graduate students and postdocs becoming prominent figures in artificial intelligence. Hinton’s insights have reshaped the way computer scientists and neuroscientists approach their respective fields.

Concluding Remarks:
Hinton’s groundbreaking work in deep learning has revolutionized artificial intelligence and has the potential to greatly impact neuroscience in the coming years. The presentation aims to offer attendees a glimpse into Hinton’s influential ideas and their potential implications for the understanding of the brain.

History of Deep Learning and AI:
Two paradigms for AI: reasoning and logic vs. adapting connections in the brain. People who focused on adapting connections between fake neurons eventually achieved better results than those doing symbolic AI.

Deep Learning and Example-Based Programming:
Deep learning allows computers to learn from examples without explicit programming. Neural networks can perform tasks like image captioning, which were challenging for conventional AI.

Simplified Model of a Neuron:
Neurons receive input, multiply it by weights, and sum it up (depolarization). Output is proportional to the input. Networks are created by connecting neurons in layers.

Training Neural Networks with Backpropagation:
Start with random weights. Show inputs and compare outputs with desired outputs. Change one weight slightly and show inputs again to check if performance improves. Keep weight changes that improve performance.

Advantages of Backpropagation:
Millions of times faster than a mutation-based algorithm. Effect of weight change on output is predictable within the network. Backpropagation calculates the gradient of the cost function with respect to the weights.

New Algorithm for Weight Adjustment:
A new algorithm allows for simultaneous weight adjustment based on output improvement. Weights are updated incrementally to optimize output quality.

Online Algorithm for Efficient Learning:
Instead of updating weights after processing all examples, the online algorithm updates weights after each case or small batch. This approach has been surprisingly effective, even for large datasets.

Neural Nets for Complex Tasks:
Contrary to initial predictions, neural nets with random weights can learn complex tasks such as language translation and image recognition. In 2009, researchers demonstrated improved speech recognition using neural nets.

Neural Net Success in Image Recognition:
In 2012, neural nets outperformed traditional computer vision techniques in image recognition tasks. Since then, neural nets have become the standard for object recognition.

Neural Net Breakthrough in Language Translation:
In 2014, researchers developed a neural net that could translate fragments of words between languages. This approach surpassed symbolic AI in translation quality and is now used by Google for machine translation.

Factors Contributing to Neural Net Advancements:
Increased computational speed. Larger datasets. Clever tricks and techniques.

Examples of Neural Net Capabilities:
A team at OpenAI trained a neural net with 1.5 billion learnable connections on billions of words of English text. The net can predict the next word in a sequence based on probabilities.

Revealing the Neural Net’s Beliefs:
By feeding predictions back into the net and prompting it to generate more text, researchers can uncover the net’s underlying knowledge and beliefs.

Further Innovations:
Transformers and improved overfitting prevention techniques are among the latest advancements in neural networks.

Neural Net Text Generation:
Geoffrey Hinton presents a neural net-generated story about unicorns speaking English in Argentina. The neural net creates a plausible narrative with elements like “Dr. Jorge Perez” and “the University of La Paz,” despite these details being fictional. The generated text demonstrates the neural net’s ability to remember and incorporate initial context, unlike recurrent neural nets.

Understanding Magic Realism:
The neural net’s story includes elements of magic realism, such as unicorns and their connection to a lost race of people. Hinton highlights the net’s understanding of magic realism by mentioning the “descendants of a lost race” and the commonplace nature of such incidents in South America.

Neural Net Capabilities and Limitations:
The neural net’s text generation shows its ability to produce coherent and contextually relevant content. However, the net still makes mistakes, such as confusing “dialect” and “dialectic” or attributing the unicorns to Argentina instead of Bolivia.

Increased Neural Net Capacity:
Hinton discusses a larger neural net with 50 billion connections, trained on Google’s latest cloud hardware. He predicts that this net’s output will be even more impressive and accurate than the one with 1.5 billion connections.

Neural Net’s Memory and Contextual Processing:
The neural net’s ability to remember and utilize initial context is attributed to its mechanism of comparing current hidden unit patterns with previous patterns. This allows the net to incorporate earlier information into its text generation, resulting in context-dependent content.

Storing Long-term Memory:
Hinton challenges the traditional view of storing long-term memory as copies of neural activity patterns. He proposes that instead, temporary changes in synaptic weights, or “fast weights,” are responsible for long-term memory storage. These fast weights are constantly modified through Hebbian learning, allowing the brain to store and access information over long periods of time.

Location of Short-term Memory:
Hinton suggests that short-term memory is stored in the temporary changes to synaptic weights, rather than in the activities of neurons themselves. This allows for a much larger capacity for memory storage compared to using individual neurons. The temporary changes in synaptic weights decay over time, providing a natural mechanism for forgetting.

Backpropagation in the Cortex:
Hinton acknowledges the skepticism among neuroscientists regarding the brain’s ability to perform backpropagation, a key algorithm in deep learning. He points out that backpropagation is not necessary for Hebbian learning, which can drive the temporary changes in synaptic weights underlying long-term memory.

Backpropagation and the Brain:
Backpropagation is a powerful algorithm for training neural networks that has been used to solve many practical problems. Neuroscientists have argued that backpropagation is not biologically plausible because it requires neurons to send signals backwards down their axons, which they do not do. Hinton argues that these objections are not valid and that backpropagation is the right way to compute the changes in the receptive fields of early feature detectors that are necessary to improve decision-making.

Supervision Signals:
Neuroscientists have argued that the brain does not have access to the supervision signals that are necessary for backpropagation. Hinton argues that supervision signals can be obtained by using part of the input or a small part of an image as the right answer.

Real-Valued Activities:
Neuroscientists have argued that neurons do not send real-valued activities, but only spikes. Hinton argues that backpropagation is tolerant of noise and that it is possible to treat neurons as if they could send real-valued activities.

Reciprocal Connections:
Neuroscientists have argued that neurons have reciprocal connections, which would make it difficult to implement backpropagation in the brain. Hinton argues that there are ways to overcome this problem.

Dropout:
Hinton introduces Dropout, a regularizer that can be used to prevent neural networks from overfitting to the training data.

Introduction:
Hinton explores the concept of removing half the neurons in a neural network with one hidden layer during training. This randomization creates exponentially many different models sharing parameters, allowing for better generalization.

Noise and Generalization:
Adding noise to a large model, like the brain, improves generalization compared to a small model with no noise. Poisson neurons with a firing rate and added noise lead to better generalization.

Dropout and Backpropagation:
Dropout models, where neurons are randomly dropped out during training, are a form of noise addition that improves generalization. These models are trained using backpropagation, making them compatible with the backpropagation algorithm.

Neurons and Feature Representation:
Neurons in a neural network represent the presence of features in the current input. The same neuron cannot represent both the feature value and the error derivative simultaneously.

Conclusion:
Hinton emphasizes the importance of noise in neural networks for better generalization. The use of dropout models with backpropagation is an effective approach to achieve this goal. Neurons have distinct roles in representing features and propagating error derivatives.

Hypothesis:
Neurons communicate through firing rates, which are stochastically communicated by spikes. The rate of change of the firing rate represents the error derivative. This allows for positive and negative derivatives without changing the signs of synapses.

Mathematical Representation:
yj (output of neuron j) is the output of neuron j. The rate of change of yj over a short time interval is the error derivative.

Learning Rule:
Incoming weights are changed by the activity of the presynaptic neuron multiplied by the difference between the green and red activations. This learning rule communicates an error derivative without explicit backpropagation.

Challenges:
The initial hypothesis had the wrong sign, which was later corrected. The theory has been inconclusive and does not perform as well as expected.

Introduction:
Geoffrey Hinton presents a method for deep learning using a stack of autoencoders and spiking neurons.

Learning with Autoencoders:
Each layer of the autoencoder learns to reconstruct the activity in the layer below. A stack of autoencoders is built, allowing each layer to reconstruct the activity in the layer below.

Top-Down Passes:
Two top-down passes are performed. The first pass reconstructs activities everywhere based on the predicted output. The second pass changes the output to be more like the desired output.

Backpropagation:
The difference between the two reconstructed activities is the signal needed for backpropagation. The learning rule involves changing the synapse by the presynaptic activity and the rate of change of the postsynaptic activity.

Spiking Neurons:
Spiking neurons represent underlying firing rates that change. The learning rule for spiking neurons involves measuring the rate of change of the postsynaptic firing rate.

Derivative Filter:
The learning rule for spiking neurons is a derivative filter that measures the rate of change of the postsynaptic firing rate.

Temporal Derivatives:
Using the rate of change of a neuron to represent an error derivative prevents the use of temporal derivatives to communicate the temporal derivatives of what the neuron represents.

Conclusion:
Hinton discusses the implications of using spiking neurons for deep learning and the limitations it poses on representing temporal derivatives.

Neurons and Representation of Derivatives:
Neurons cannot use the rate of change of a representation to represent changes in the world. Velocity cannot be represented by the rate of change of a position neuron. Acceleration cannot be represented by the rate of change of a velocity neuron.

Error Derivatives and Temporal Derivatives:
The arguments against backpropagation support the idea that temporal derivatives of neurons are used to represent error derivatives. Error derivatives can be represented as temporal derivatives. The same neuron can send temporal derivatives backwards and communicate activities forwards.

Spike-Dependent Plasticity:
The fact that spike-dependent plasticity is observed in the brain suggests that error derivatives are represented in the brain.

Conclusion:
The evidence suggests that neurons use temporal derivatives to represent error derivatives.

The Brain as a Computer:
Geoffrey Hinton acknowledges that the brain can be considered a computer. He explains that there are numerous ways to perform computation using physical materials. He emphasizes that the brain is a computer with unique strengths and weaknesses, such as its slower speed but immense parallelism. Hinton highlights a crucial distinction between computers and brains: every brain is unique due to different connections, making it impossible to directly transfer knowledge from one brain to another.

The Connectome and Hardwiring:
Hinton views the efforts to fully map the brain’s connectome as scientifically valuable, especially for understanding certain structures like the retina with extensive hardwired components. However, he believes that such comprehensive mapping is not necessary to grasp the fundamental principles of the brain.

Innate Behavior and Neural Networks:
Hinton challenges the earlier notion that language is primarily innate, emphasizing that language can be learned through data without the need for extensive innate structures. He argues that it is unnecessary to incorporate innate knowledge if it can be quickly learned, citing the example of motion perception.

Psychologists’ Beliefs about the Brain:
Hinton criticizes the previous conviction among psychologists, influenced by Chomsky, that the brain possesses a vast amount of innate knowledge and that learning is limited. He believes that psychologists were mistaken in their assumptions about the extent of innate knowledge and the brain’s capacity for learning representations from scratch.

Genetic Hardwiring and Learning:
Geoffrey Hinton challenges the traditional view that innate knowledge in the brain is a product of evolution. He proposes that this knowledge is instead discovered through learning and later becomes hardwired through genetic evolution.

The Mating Circuit Experiment:
Hinton illustrates this concept using a simplified example of a mating circuit. He shows how random mutations and sexual combination can lead to a slow and inefficient process of finding a successful circuit.

Introducing Learning:
Hinton introduces a third allele for each connection in the circuit, allowing it to be left to learning. With this modification, a small population of organisms can rapidly find a successful circuit through learning, reducing the number of trials required.

Baldwin Effect:
The process of learning uncovering solutions that evolution can then hardwire is known as the Baldwin effect. Hinton argues that much of the hardwired structure in the brain was first discovered through learning and gradually backed up into the genetic code.

The Role of Compute in Deep Learning’s Success:
Hinton suggests that the success of deep learning can be attributed to the availability of powerful compute resources. He acknowledges that clever network designs and learning rules also play a role, but emphasizes the importance of compute.

Yann LeCun’s Contributions:
Hinton highlights Yann LeCun’s invention of convolutional neural nets in the late 1980s. He notes that these ideas did not gain traction until the advent of powerful computer hardware.

The Future of Neuroscience and AI:
Hinton sees potential for a new cognitive science that studies general intelligence using brain-centric models. He also acknowledges the possibility of the two fields departing in the future.

Understanding the Brain through Computation:
To comprehend how the brain performs computations, we must bridge the gap between the intricate nature of brains and the simulations we can run on computers.

Bridging the Gap:
Computational neuroscientists approach the study of brain computation from the biological perspective, while Geoffrey Hinton attempts to build a bridge from the artificial neural networks’ perspective.

The Goal:
The ultimate aim is to develop artificial neural networks that perform computations in a manner similar to the human brain.

Conventional AI:
Conventional AI also aims to build a bridge between brains and computers but approaches it differently.

Questions from the Audience:
The audience is given the opportunity to ask questions, with a first-come, first-served system implemented.

Hebbian Synapses:
A question arises regarding the extent to which biological memory mechanisms, such as Hebbian synapses, are implemented in AI for deep learning.

Back Propagation vs. Hebbian Synapses:
Hinton clarifies that most deep learning systems employ back propagation, an error correction rule, rather than Hebbian synapses, which rely on strengthening connections through repeated use.

Reinforcement Learning and Dopamine:
Reinforcement learning utilizes dopamine as a reward prediction error signal. This signal represents the difference between expected and obtained rewards. While reinforcement learning can solve problems, it’s computationally expensive and not as successful as error correction learning in AI.

Hierarchy in Neural Networks:
Neural networks use multiple layers to represent different levels of complexity. In vision, hierarchical layers process information from large structures (galaxies) to smaller ones (atoms). The brain can only process a limited range of hierarchy at a time, similar to moving a small window of high-resolution focus. Hierarchical structures in the real world are mapped onto brain structures through attention and flexible mappings. Some hierarchies, like phonemes and words in language, have fixed mappings in the brain.

Over-parameterization and Regularization:
Contrary to statistical intuition, over-parameterized models can be effectively trained with regularization. Regularization prevents the model from fitting data too closely and reduces overfitting. Bayesian statisticians may prefer higher-order polynomials for fitting data, even if they don’t exactly match the data points.

Hebbian Synapses:
Hebbian synapses are simple synapses that can encode long-term memories. While successful in humans, AI is currently focused on error correction learning and unsupervised learning for better results. Temporary memories with fast-component synapses can improve the performance of neural networks.

AI Models and Data:
Geoffrey Hinton discusses the relationship between AI models and data. Large databases have been crucial for AI’s biggest successes. Regularization techniques can help, but a significant amount of data is still necessary.

Polynomial Fitting and Dropout:
Hinton explains how fitting polynomials to data points can provide valuable insights. A large number of polynomials are fitted, and their average provides a good estimate. The variance of the polynomials indicates the uncertainty in the estimate. Dropout works similarly, resulting in good mean answers and a sense of variance.

Challenges with Sparse Data:
Sparse data, with limited channels, signals, or voxels, presents a challenge for AI models. Hinton acknowledges the need for more data in such scenarios.

Personalizing Medicine:
Personalizing medicine requires training models on individual data, which is often sparse. While it may seem impossible to bridge the gap between model capabilities and individual needs, Hinton believes there is potential for progress.

Transfer Learning:
Transfer learning can be used to train models on large datasets and then fine-tune them on smaller, individual datasets. This approach can help overcome the challenges of sparse data in personalized medicine.

Dropout and Block Dropout:
Geoffrey Hinton discusses dropout, a technique used in deep learning to randomly leave out units or groups of units in a neural network during training. Block dropout is a variation of dropout where groups of units are randomly left out, allowing units within a group to collaborate and promoting independence between groups.

Randomness in Dropout:
Hinton acknowledges the randomness of dropout and the concern that the burden of prediction is placed on the random part of the computer.

Attempts to Improve Dropout:
Hinton mentions that researchers have explored methods to improve upon random dropout, such as block dropout, which has shown effectiveness in certain applications.

Finding Structure in Dropout:
A question is raised about whether there are similarities in the structure of the network that produce the best results when using dropout iteratively. Hinton acknowledges that people have investigated this idea, but he does not have a definitive answer on whether there is a more structured and sensible approach to dropout.

Conclusion:
The discussion on dropout and block dropout concludes, with Hinton expressing uncertainty about whether there is a better alternative to random dropout. The event host, Paul Franklin, thanks Hinton and the host, Blake, for their contributions.