Jeff’s Introduction of Geoffrey Hinton:
Jeff presented Geoffrey Hinton as the “father” or “godfather” of machine learning, particularly deep learning. Hinton has been working on deep learning for over 30 years and is responsible for some original algorithms and techniques that have enabled models to work on complex problems. Hinton has won numerous awards, including the Primal Heart Prize, the Clerk Maxwell Award, and the Hertzberg Gold Medal. Many of Hinton’s students and postdocs have become prominent figures in machine learning, both in academia and industry.

Geoffrey Hinton’s Personal Note:
Hinton acknowledges that he is a “machine learning fossil” and will discuss an idea he began working on in 1973.

Hinton’s Idea for Storing Temporary Memories:
Hinton proposes an alternative mechanism for storing temporary memories in the brain, based on temporary changes to synapse strengths, rather than the traditional view of neurons remaining active to store memories. This idea is relatively unexplored despite its simplicity.

Fast Weights for Rapid Adaptation:
Neural networks use slow weights for long-term knowledge and fast weights for temporary memory. Fast weights adapt rapidly and decay quickly, allowing recently occurred states to become attractors. Priming, where recognizing a word becomes easier after hearing it recently, is an example of fast weights in action.

Increased Capacity with Temporary Weight Changes:
Storing memories in temporary weight changes offers greater capacity compared to activity vectors. Fully connected networks have n squared capacity for temporary storage, significantly more than n for activity vectors. Efficient storage rules are being explored in neural networks, but simple one-shot learning rules suffice for storing temporary memories.

Hopfield Learning Rule for Temporary Memory:
The Hopfield learning rule is a simple and effective method for storing temporary memories. It involves increasing weights between active units and decreasing weights between active and inactive units. This rule is not suitable for long-term memory storage.

LSTM vs. Fast Weights for Temporary Knowledge:
LSTMs store sequence knowledge in hidden unit activities, which have limited capacity. Storing multiple independent vectors in LSTM hidden vectors leads to interference and reduced processing capacity. Fast weights store information about recent events without affecting hidden unit capacity or processing. Fast weights make previously experienced states stronger attractors, guiding the network towards those states.

Historical Context: Neural Networks in 1973:
Hinton’s early work on neural networks in 1973 was not well-received. Minsky and Papert’s 1969 paper influenced the prevailing belief that neural networks were useless and could only learn simple associations. The prevailing ideology in AI emphasized software and dismissed the relevance of hardware, including the brain. Additionally, neural networks were considered incapable of recursion, a key feature in Lisp programming.

Implementation of Recursion in Neural Networks:
The challenge in implementing recursion in neural networks lies in utilizing the same neurons and weights for recursive subcalls, preserving information from earlier processing steps. Traditional approaches involve using stacks to store activity vectors, but Hinton proposes a method based on fast weights.

Fast Weights for Recursive Processing:
Hinton’s approach involves storing the intermediate processing results using fast weights. During recursive subcalls, the network can temporarily switch to the stored weights, process the embedded sentence, and then return to the original processing context.

Historical Context:
Hinton’s initial program in 1973 failed to gain traction due to the lack of knowledge on learning hidden representations. Backpropagation, which was later discovered, enables the learning of hidden representations, making Hinton’s approach more feasible today.

Starting with a Non-Recursive Task:
Hinton begins by exploring a simple non-recursive task to demonstrate the principles of his approach before applying it to recursive problems.

Introduction:
Geoffrey Hinton presents his early work on neural networks for character drawing, emphasizing its recursive capabilities.

The Non-Recursive Task:
The task involves drawing capital letters based on their identity, pose, and position, using a sequence of stroke specifications. The neural network consists of groups of neurons with multiple states, and the current state determines the next state based on weights.

The Recursive Version:
The recursive version defines a capital letter I as a capital letter T with a stroke across the bottom. To draw an I, the network first draws a T and then adds the stroke. Fast associative memory is introduced to remember the intermediate state of the T while drawing the stroke.

The Learning Algorithm:
The network is trained using the perceptron convergence rule, which adjusts weights based on correct and incorrect state transitions. The training sequences are designed to encourage the desired behavior.

Challenges and Solutions:
The network initially struggled to learn certain sequences due to linear inseparability. Hinton introduced temporally intermediate states to make the sequences linearly separable, effectively acting as hidden units.

Conclusion:
Hinton’s early work demonstrated the feasibility of recursive neural networks, which paved the way for more advanced architectures and applications.

RECURRING NEURAL NETWORKS (RNNS):
RNNs are a type of neural network that can process sequential data. Traditional RNNs tend to “blow up” and not perform as well as Long Short-Term Memory (LSTM) networks. If initialized with the identity matrix, RNNs can hold information better, though not quite as well as LSTMs.

FAST RATES RNNs:
Fast Rates RNNs are a simpler variation of RNNs that use rectified linear units (ReLUs) and an outer product learning rule. Outer product learning rule: At each time step, the hidden state is used to update a fast weight matrix, which is then used to compute the next hidden state. Layer normalization is crucial for making Fast Rates RNNs work well. It helps to keep the hidden state vectors roughly the same length.

ASSOCIATIVE MEMORY TASK:
A toy problem is used to test the capabilities of Fast Rates RNNs. The task involves memorizing associations between letters and numbers. Fast Rates RNNs can solve this task with only 20 hidden units, compared to 50 hidden units for LSTMs.

CONCLUSION:
Fast Rates RNNs are a promising approach for storing information in neural networks. They are simpler than LSTMs and can achieve comparable performance on certain tasks.

The Theory of Vision:
Vision consists of a sequence of glimpses where a small part of the image is taken at very high resolution, a bit more at low resolution, and the rest at extremely low resolution. The eye can zoom in on things by translating some of the high-resolution fobias to the center.

The Problem of Putting It All Together:
The outer loop of vision involves looking somewhere and getting a retinal image with very high resolution for some parts and very low resolution for others. The challenge lies in integrating all this information efficiently.

Breaking the Problem Apart:
To simplify the problem, suppose someone else tells you where to look next (fixed sequence of looking places). Focus on solving the integration problem: how to integrate the information obtained from these fixed places.

Sequences of Glimpses and LSTMs:
Sequences of glimpses are not ideal for LSTMs because the actual order of the glimpses does not convey information. The information conveyed is independent of the order, so the serial order is not relevant.

Information Sources for High-Up Infratemporal Cortex:
Bottom-up processing of the current fixation. Information from the previous state (recurrent net) representing the immediate temporal context. Information from previous glimpses, especially of sub-parts of the object.

Fast Weights and Associative Memory:
Bottom-up information from an image, immediate temporal context, and stored information about previous glimpses are used in determining high-level temporal representation. Fast weights are introduced to integrate information from different glimpses.

Integrating Glimpses:
A fixed sequence of fixations is used to process different parts of an image. Each quadrant of the face is glimpsed twice, once at the beginning and again after processing the sub-quadrants. Low-level glimpses are accumulated in an IRNN, while fast weights are used to access information from previous glimpses at higher levels.

Visual Representation and Memory:
A convolutional neural network (ConvNet) is used to extract features from the image. These features are used as a key to retrieve information from a fast weight memory. The retrieved information is combined with the bottom-up input to create a state representation.

Experimental Results:
The architecture is tested on the MultiPy face recognition dataset. With 100 hidden units, an IRNN performs reasonably well, with an LSTM performing slightly better. Introducing fast weight memory significantly improves performance compared to both IRNN and LSTM.

Glimpses vs. Fixed Scans:
Glimpses allow for high-detail processing of specific image patches, but require intelligent fixation selection. Fixed scans provide parallel processing of the entire image, eliminating the need for intelligent fixations.

Fast Weights:
Fast weights adapt based on input, allowing for efficient integration of glimpses. Fast weights are not compatible with mini-batches, limiting their use in GPU-based training.

Vector Retrieval:
Fast weights can be approximated by using a retrieval key vector to access stored activity vectors. The retrieved vector is a weighted sum of previous activity vectors, with weights determined by similarity and decay factor. This approach allows for attention to the recent past without additional free parameters.

Fast Weight Memory:
Fast weight memory enables true recursion in recurrent neural networks. Real brains may possess fast weight memory, allowing for proper recursion.

Neural Nets with Extra Memory:
Neural nets with external memory, such as the Differentiable Neural Computer, can learn algorithms from examples. Exploring different forms of memory for neural nets beyond activity patterns and flow rates is a promising area of research.

Google’s Recruiting Pitch:
Google encourages attendees to consider moving to California for employment opportunities. The Brain Resident program offers a year-long employment opportunity at Google for researchers and postdocs.

Iterative Associative Retrieval:
Iterative associative retrieval allows for focused attention on specific vectors over time. This can be useful for processing subclauses and storing information about the rest of the sentence.

Parse Trees and Hierarchical Structures:
Many structures in the world can be represented as parse trees, with deep hierarchies. Neural networks can potentially learn to process these hierarchical structures effectively.

Philosophical View of Brain Function:
Hinton suggests that we have a limited window of attention on a hierarchy of information. We move this window around to process different aspects of the world.

Multiple Timescales in Synaptic Changes:
Synapses can change at various timescales, not just one. Terry Sanofsky’s work on frogs demonstrated fast-changing synapses.

Order of Glimpses and Memory Decay:
The order of glimpses in visual processing matters due to memory decay. However, the sequential structure created by the order is less significant.

Sequential Structure in Video:
Time is crucial in video processing because it’s not an arbitrary creation from a static scene.

Visual Attention Optimization:
Humans have learned to fixate on specific locations to gather necessary visual information efficiently. This is evident in activities like making tea and driving around curves.

Mail Slot Example:
Hinton recalls a large mail slot array at CMU, allowing individuals to quickly access their mail.

Fast Weights vs. ConvNets:
Fast weights were tested against ConvNets. Both achieved similar performance when operating in parallel. Fast weights’ effectiveness suggests proper temporal integration.

Frame Problem in AI:
Most knowledge is inferred from the absence of information. Monitoring systems detect changes and reinforce confidence in stability.

Motion Detection and Attraction:
Motion detectors aid in smooth tracking movements. Localized motion attracts attention, especially when other motions are absent. Outdoor environments with windy conditions pose challenges.

Upcoming Talk:
Rich Sutton, the “godfather of reinforcement learning,” will speak in two weeks. Talks are usually held on Thursdays at noon.