Setting the Stage:
WSDL 2023 is the second installment of a rigorous winter school focused on deep learning. The inaugural program in 2022 reached 190 candidates with 130+ hours of tutorial material, including 120 hours of theory and hands-on lectures, and 10 hours of doubt clearing. The 19 regular course instructors hailed from internationally acclaimed educational institutes worldwide.

Program Expansion:
WSDL 2023 aims to expand its reach and scope, targeting near about 400 participants. The program will maintain the benchmark of 130+ hours of course material, covering a wide range from machine learning fundamentals to cutting-edge research on diffusion models. Learners will benefit from real-time hands-on tutorials alongside precise theoretical concepts.

Distinguished Speakers:
The lineup of plenary speakers for WSDL 2023 includes leading thinkers and innovators in the field of deep learning. Notable speakers include Turing Award Fellow Geoffrey Evers-Tinton, inventor of variational autoencoders and Adam optimizers Diederik P. Sigma, and Michael Montefiore-Inria Prize recipient Gerald Biao. Mikhail Belkin, Lawrence Vandermatten, Lehman Afoglu, Phyllis Gabriel, and Thomas Peake are among the other renowned speakers.

Special Lecturers and Advanced Courses:
A diverse group of special lecturers will further enrich the program. Advanced courses based on ongoing research will be offered, covering topics like explainable artificial intelligence, diffusion-based generative models, and deep reinforcement learning. Practical applications such as medical image analysis, COVID detection, business analytics, graph data analysis, and federated learning will also be explored.

Technical Difficulties:
There were initial technical difficulties with sharing screens and audio during the presentation. The presenter, Shabbatam Das, addressed the issues and worked to resolve them.

Background of Professor Geoffrey Hinton:
Distinguished professor at the University of Toronto and researcher at Google. Co-founder and chief scientific advisor of the Vector Institute in Toronto. Turing Award recipient in 2018 for foundational research on deep learning. Numerous accolades and awards in the field of machine learning.

Professor Shabbatam Das’s Welcome Speech:
Acknowledgement of Professor Hinton’s contributions to machine learning. Appreciation for Professor Hinton’s virtual presence despite the unfortunate lack of physical attendance. Invitation to visit Calcutta and other branches of the Indian Statistical Institute.

Motivation for the Forward-Forward Algorithm:
Current applications of deep learning rely heavily on the backpropagation algorithm for gradient computation. Despite its effectiveness, backpropagation’s implementation in the brain remains unclear. Neuroscientists seek evidence of error derivative signals in the brain, but none has been conclusively found.

Introduction to the Forward-Forward Algorithm:
Professor Hinton’s upcoming talk will focus on the forward-forward algorithm. The algorithm is designed to address the limitations of backpropagation in the context of neuroscience. It aims to provide an alternative approach to learning in neural networks that aligns better with brain-like processing.

Backpropagation and the Brain:
Backpropagation through time, used in language models, is implausible for video processing and contradicts the brain’s anatomy. The brain’s visual system has backward connections, but they don’t go to the same neurons as the forward connections. Backpropagation requires a precise model of the forward computation, which is not available in the brain.

Big Language Models vs. Cortex:
Big language models have fewer connection strengths than the brain but know more facts. Cortex has more connections but less training data. Cortex needs an algorithm that works in this regime, different from backpropagation’s typical use.

A Different Kind of Computer:
Modern neural nets are general-purpose but learn from data rather than written programs. This challenges traditional notions of computation and the separation of software and hardware.

Immortal vs. Mortal Computation:
Digital computers separate software and hardware, allowing programs to run on different physical computers. “Mortal computation” abandons this separation, using the hardware’s quirks for computation specific to that hardware.

Learning in Hardware:
Learning algorithms are needed that run efficiently in hardware with unknown properties. Parameters learned in one hardware cannot be shared with another due to hardware differences. This opens up new possibilities but requires different learning algorithms.

Example: Vector-Matrix Multiplication in Hardware:
Standard method: Digital transistors, high power, bit operations proportional to the square of the vector length. Alternative method: Neural activities as voltages, weights as conductances, multiplication and addition by voltage-conductance multiplication and charge addition.

Introduction to the Forward-Forward Algorithm:
The forward-forward algorithm is a method for training multi-layer neural networks. The goal is to train each layer to have high goodness for positive data and low goodness for negative data. The algorithm is greedy, meaning each layer is trained independently.

Determining Goodness and Generating Negative Data:
Goodness is initially defined as the sum of the squared activities of the neurons in a layer. The algorithm considers real data as positive examples and negative data as negative examples. Various methods can be used to generate negative data, including using neural networks to generate visual images or videos.

Rectified Linear Units (ReLUs) and Logistic Function:
Hidden layers in the network consist of rectified linear units (ReLUs). The goodness of a layer is calculated by subtracting a threshold from the sum of the squared activities and passing the result through the logistic function. The logistic function outputs a probability indicating whether the input data is a positive or negative example.

Updating Weights and Normalization:
The algorithm adjusts the weights on the connections to change the goodness of a layer. It aims to increase confidence in positive examples and decrease confidence in negative examples. To prevent the next hidden layer from simply detecting the length of the vector in the first hidden layer, the length of the vector is normalized.

Principle Component Analysis Analogy:
The normalization process is analogous to finding the first principle component of data, removing it, and then finding the next principle component. This ensures that the second layer learns to turn the relative activities of units into new features.

Conclusion:
The forward-forward algorithm trains multi-layer neural networks by greedily training each layer to distinguish between positive and negative data. It utilizes ReLUs, the logistic function, and normalization techniques to achieve this goal. The algorithm enables the network to learn meaningful features from the input data.

Using Labels as Input:
To incorporate label information for classification, a unique approach is employed. Positive images are given the correct label, while negative images are given an incorrect label as part of the input. The first 10 pixels of the image are used for label representation in a 1-of-n encoding.

Testing the Model:
Two methods are used to test the model’s performance: Quick method: A neutral label is applied to the input image, and a linear classifier is trained on the hidden layer activities to produce a probability distribution across labels. Slow method: Each possible label is tried with the image, and the label that maximizes the sum of squared activities in all layers is selected.

Improving Training and Results:
Hard negative cases are generated by providing a false label that is similar to the correct label. Jittering input images by up to two pixels in each direction augments the training data and improves performance, achieving an error rate comparable to convolutional neural networks.

Visualizing Learned Features:
The weights connecting the first hidden layer units to the input pixels are visualized to reveal the receptive fields of hidden units. The features learned by the network are interpretable and make sense in the context of the MNIST dataset.

Limitations:
The algorithm suffers from a lack of interaction between layers during learning, which may limit its applicability to modeling perceptual systems in psychology.

Contextual Interpretation of Features:
The interpretation of features depends on the context in which they appear. For example, the middle letter “H” in “HAT” is perceived differently than the middle letter “A” in “CAT,” despite being identical. Similarly, a shape may be interpreted as a nose due to its context in a face, even though it may not have the exact shape of a nose.

Context as a Signal for Learning Feature Extraction:
The strong influence of context on feature interpretation suggests that context provides a valuable signal for learning how to extract individual features. This is because the context helps to determine the meaning and significance of individual features.

Example of Context-Dependent Meaning:
The made-up word “scrummed” can have multiple meanings depending on the context in which it is used. For example, “she scrummed him with the frying pan” could mean she attacked him, impressed him with her cooking, or was cross with him and hit him with the frying pan. The context provides clues that help determine the most likely meaning in a given instance.

Background:
The perceptual system is designed to process video, which is a sequence of images. Static images can be treated as videos with every frame being the same.

Glom Architecture:
Introduced a biologically realistic architecture called Glom for representing part-whole hierarchies in the cortex. Glom architecture consists of multiple levels of representation, where each level receives input from the level below, the level above, and the same level at the previous time step.

Using Glom for Handwritten Digit Recognition:
The input to the bottom layer is the pixel values of the handwritten digit without the label. The output at the top layer is the label of the digit. The middle layer receives input from the bottom layer (bottom-up) and the top layer (top-down), as well as input from the same level at the previous time step.

Predictive Nature of Top-Down Input:
Top-down input is delayed relative to bottom-up input. For static images, top-down input predicts the current frame. For moving images, top-down input predicts the future frame.

Learning Algorithm:
The learning algorithm aims to make the activity of the middle layer high for good examples (correct labels) and low for bad examples (incorrect labels). Initially, the squared activities were used as the goodness function. Later, the negative of the squared activities was found to work better. Most recently, minimizing the distance between top-down and bottom-up inputs was found to be the best goodness function.

Conclusion:
The forward-forward algorithm is a biologically realistic learning algorithm for the Glom architecture. It can be used for recognizing handwritten digits by extracting features and predicting future frames. The algorithm’s effectiveness is improved by using a goodness function that minimizes the distance between top-down and bottom-up inputs.

Negative Data and Training Efficiency:
The network strives to make positive data match and negative data not match, preventing the network from collapsing. Matching positive data and avoiding matches with negative data allows for more interesting and specific identification of correct labels.

Comparison with Backpropagation:
Backpropagation uses forward and backward passes for derivative propagation, offering exact derivatives and efficient weight adjustment. This recurrent network employs “back relaxation,” where intermediate layers settle on a compromise between top-down and bottom-up information. Top-down information is diluted as it descends through layers, affecting later layers more. Back relaxation is less efficient than backpropagation, resembling diffusion rather than propagation. It’s biologically realistic and sufficient for learning in networks with a limited number of layers.

Local Information and Parallel Learning:
The algorithm can be made more efficient by using local goodness functions for groups of units, promoting parallel learning at lower levels. This approach reduces interference and accelerates learning.

Separate Phases for Positive and Negative Data:
Positive learning: During wakefulness, the network focuses on positive data, matching top-down and bottom-up information to improve goodness. Negative learning: During sleep, the network generates negative data and tries to minimize goodness in all layers.

Inspiration from Crick and Mitchison’s Theory of Sleep:
Francis Crick and Graham Mitchison proposed a theory of sleep in 1980, suggesting that sleep serves as a negative training phase. This theory aligns with the idea of using generated negative data during sleep to improve network performance.

Introducing the Forward-Forward Algorithm:
The forward-forward algorithm is a novel approach to learning, inspired by the theory that sleep serves as a mechanism for forgetting irrelevant information.

Key Insights:
During sleep, the algorithm undergoes a “negative phase” where it learns to distinguish real data from generated data, akin to unlearning erroneous beliefs. In contrast, during waking hours, the algorithm engages in a “positive phase” focused on learning new information.

Why Sleep is Essential:
The algorithm demonstrates the crucial role of sleep in maintaining neural networks’ stability and preventing catastrophic forgetting. Depriving the algorithm of sleep leads to a collapse of knowledge and an inability to distinguish between different types of data.

The Function of Dreams:
Dreams are theorized to be the manifestation of the brain’s unlearning process, which occurs during sleep. This theory provides a compelling explanation for why dreams are often bizarre and disconnected from reality.

Comparison with Boltzmann Machines:
Boltzmann machines, another type of contrastive learning algorithm, require a complex energy function and Markov chain Monte Carlo methods for training. The forward-forward algorithm employs a simpler energy function and straightforward derivatives, making it more computationally efficient.

Preventing Modeling of Own Wiring:
The forward-forward algorithm incorporates positive and negative examples to prevent neurons from modeling their own wiring, avoiding the trap of self-referential learning.

Correlations in Hidden Layer:
When a network has purely random inputs and weights, correlations can arise in hidden layer activities due to the weights’ influence. These correlations reflect the network’s wiring, not anything meaningful about the world.

Learning Based on Correlations:
A learning algorithm that solely relies on finding correlations in data without contrasting positive and negative examples may mistake network wiring correlations for real-world patterns. This approach leads to learning about the network’s wiring, which is not beneficial for survival or understanding the world.

Stacking Boltzmann Machines and Modeling Wiring:
Stacking Boltzmann machines can partially overcome the problem of modeling network wiring by initializing them with a model of the wiring in the layer below. However, this approach limits the machine’s capacity to learn meaningful patterns beyond the wiring structure.

Conclusion:
Positive and negative examples are crucial in learning to prevent modeling the network’s own wiring. Relying solely on correlations without contrasting positive and negative data can lead to misleading conclusions and hinder the learning of meaningful patterns.

SimClear:
SimClear is a successful contrastive learning method that uses two random crops of the same image to extract similar representations. It requires backpropagation and has several drawbacks, including the need for random crop locations and limited constraint per patch pair.

Generative Adversarial Nets (GANs):
GANs use a discriminative model to distinguish between real and generated images, and a generative model to generate images that fool the discriminator. GANs require backpropagation and adversarial learning, which can be challenging to optimize.

Forward-Forward Algorithm:
The forward-forward algorithm uses hidden layers of a discriminative model as the hidden representations of a generative model. It eliminates the need for backpropagation and adversarial learning, making optimization easier. It also prevents mode collapse, a problem in GANs where the generative model ignores large sets of real data.

Early Contrastive Learning Experiments:
The speaker describes an early contrastive learning experiment using stereo images and non-overlapping crops. The goal was to extract depth information from random dot stereograms, which have a fourth-order statistical regularity. The experiment attempted to maximize mutual information between representations of similar images while minimizing mutual information between representations of different images.

Challenges and Insights:
The speaker reflects on the challenges and insights gained from this early experiment. It took 20 years to understand why the experiment did not work as expected. The speaker emphasizes the importance of linear transformations in preserving the ratio of entropy to variance.

Key Concepts:
Linear transformations preserve the ratio between entropy and variance, while nonlinear transformations can change this ratio. Manipulating entropy by manipulating variance is only reliable for linear transformations. In forward-forward training, bad labels are chosen based on the network’s predictions, leading to faster learning. Forward-forward training does not require the entire forward path to be differentiable, unlike backpropagation.

Nonlinear Transformations and Entropy:
Linear transformations maintain the ratio between entropy and variance, but nonlinear transformations can alter this ratio. Optimizing a nonlinear transformation for entropy using variance as a proxy is unreliable.

Entropy Manipulation:
Manipulating variance to control entropy is only effective for linear transformations. For nonlinear transformations, actual negative examples are necessary to manipulate entropy effectively.

Bad Label Selection in Forward-Forward Training:
Choosing bad labels randomly is acceptable, but selecting labels that the network is most likely to confuse leads to faster learning. This approach involves selecting the incorrect label from the network’s predicted distribution, excluding the correct label.

Forward-Forward Training and Differentiability:
Forward-forward training does not require the entire forward path to be differentiable, unlike backpropagation. This advantage allows for the inclusion of non-differentiable components within the network, such as black boxes.

Conclusion:
The forward-forward algorithm offers a biologically plausible alternative to traditional backpropagation for neural network training. It addresses the limitations of manipulating entropy through variance in nonlinear transformations and provides a more efficient approach to selecting bad labels. Furthermore, its ability to accommodate non-differentiable components makes it a versatile tool for exploring various neural network architectures.

Strengths of the Forward-Forward Algorithm:
Robustness: The forward-forward algorithm is robust to stochastic, nonlinear, and stationary transformations in the black box, making it suitable for scenarios where the exact behavior of the black box is unknown or unpredictable. Insensitivity to Unknown Transformations: Unlike backpropagation, the forward-forward algorithm does not require knowledge of the transformation performed by the black box, allowing it to handle unknown transformations without any issues. Resilience to Changes in Neuronal Behavior: The forward-forward algorithm is resilient to changes in neuronal behavior, including different behaviors on different occasions and correlations between neurons. It can still learn effectively by averaging the learning over different occasions or conditions.

Limitations of the Forward-Forward Algorithm:
Inefficiency on Digital Computers: The forward-forward algorithm is less efficient when implemented on digital computers due to the inability to share weights between different copies of the hardware. Unsuitability for Large Models: The lack of weight sharing limits the size and complexity of models that can be trained using the forward-forward algorithm, making it impractical for applications that require large models, such as deep learning tasks like natural language processing. Power Requirements: The forward-forward algorithm is more suitable for low-power and cheap hardware, making it a potential match for biological systems. However, current deep learning technology is more effective than what biology can achieve, requiring high power and digital hardware.

Key Points:
The goodness function in a layer can be any function that is easy to compute, manipulate, and differentiate. Five different goodness functions have been tried, including the sum of squared activities, the sum of activities, the difference between top-down and bottom-up inputs, and the sum of squared activities with negative goodness. The difference between the top-down and bottom-up inputs worked best for allowing sleep to be separated from wake and for allowing many positive updates followed by negative updates. Any objective or goodness function can be used with the forward-forward algorithm. The forward-forward algorithm can be implemented on power-efficient analog circuits, enabling the running of large language models in a time-efficient manner.

Additional Points:
The remaining questions in the chat will be answered during the course lecture. The other lectures will be through the Zoom link that was sent to participants’ emails. If participants have not received the Zoom link, they should email the organizers to receive it. The next session will be in Classroom.