Backpropagation History:
Backpropagation was invented multiple times in the 70s and 80s, but it didn’t live up to its promise until the late 1990s due to slow computers, small datasets, and poor initialization. The popular explanation for backpropagation’s failure was that it couldn’t handle multiple layers of nonlinear features, but the real issue was practical limitations, not theoretical ones.

Support Vector Machines (SVMs):
SVMs are a reincarnation of perceptrons with a clever trick called the kernel trick, which allows them to avoid overfitting by finding a maximum margin hyperplane in a high dimensional space. SVMs can be viewed as using non-adaptive features and one layer of adaptive weights, which limits their ability to learn multiple layers of representation.

Generative Models:
Generative models aim to model the input data rather than predicting a label, which can be done with gradient descent learning. Graphical models combine discrete graph structures with probability distributions to represent how variables depend on one another, providing a powerful framework for generative modeling.

Historical Bet:
In 1995, Larry Jackal and Vladimir Vapnik made a bet on whether big neural nets trained with backpropagation would be understood theoretically by 2000. Vapnik bet that they wouldn’t, but he also made a side bet that if he was the one to figure it out, he would still win. It turned out that both were wrong, as the limitation was practical rather than theoretical.

Key Points:

Early Graphical Models:
Boltzmann machines were early examples of graphical models, utilizing real-valued computations to infer variable probabilities. Radford Neal introduced sigmoid belief nets in 1992, extending Boltzmann machines to directed graphical models.

Challenges in Deep Learning:
Learning time for deep networks with multiple hidden layers was slow due to poor weight initialization. Backpropagation can get stuck in poor local optima, despite its usefulness. Overcoming these limits requires moving beyond backpropagation.

Unsupervised Learning for Generative Models:
Unsupervised learning aims to maximize the probability of a generative model generating sensory input. This approach focuses on modeling input structure rather than input-output relationships. Generative models can be energy-based (like Boltzmann machines) or causal models made of idealized neurons.

Historical Antipathy towards Probability in AI:
In the 1970s and early 1980s, probability was widely rejected in AI. Discrete symbol processing was favored, with the introduction of probabilities seen as contaminating. Patrick Henry Winston’s AI textbook exemplified this anti-probability sentiment.

Historical Perspective on AI and Probabilities:
John von Neumann predicted that theories of computation would align with thermodynamics, particularly Boltzmann’s work. Probabilities entered AI via graphical models, combining graph theory and probability theory.

Graphical Models:
Graphical models represent variable dependencies using discrete graphs. Belief nets are directed acyclic graphs with sparsely connected nodes, allowing efficient inference algorithms. Belief nets enable inference (determining unobserved variables) and learning (adjusting interactions to better generate training data).

Relationship Between Graphical Models and Neural Networks:
Early graphical models relied on experts for graph structure and conditional probabilities. Neural networks focused on learning, not interpretability or sparse connectivity. Neural network versions of belief nets, such as Boltzmann machines and sigmoid belief nets, exist.

Boltzmann Machines:
Boltzmann machines are energy-based generative models with binary stochastic neurons and symmetric connections. Learning in Boltzmann machines is challenging, especially with multiple hidden layers.

Sigmoid Belief Nets:
Sigmoid belief nets are causal models with binary stochastic neurons and directed acyclic graphs. Generating samples in sigmoid belief nets is straightforward compared to Boltzmann machines. Learning in sigmoid belief nets is difficult, and future videos will discuss methods for addressing this challenge.

Learning Sigmoid Belief Nets:
Sigmoid belief nets are locally normalized models, so learning is easier compared to Boltzmann machines. Unbiased distribution over hidden units is hard to achieve due to the “explaining away” phenomenon.

Explaining Away:
Judea Pearl’s “explaining away” phenomenon explains why inferring posterior distribution over hidden causes is difficult. The number of possible patterns of hidden causes is exponential, making sampling from the posterior challenging.

Maximum Likelihood Learning Rule:
Learning in sigmoid belief nets involves maximizing the log probability of the inferred binary state of a unit given the inferred binary states of its parents. The learning rule is local and simple, involving adjusting weights based on the state of the parent units and the difference between observed and inferred binary values.

Sampling from the Posterior:
Sampling from the posterior over hidden nodes is difficult due to “explaining away.” Explaining away causes anti-correlation between hidden causes when an effect is observed.

Deep Sigmoid Belief Nets:
Learning deep sigmoid belief nets with millions of parameters is challenging. The network has multiple layers of hidden units, making sampling even more difficult.

Challenges in Learning Deep Belief Nets:
The posterior distribution over the first layer of hidden variables is not factorial, leading to correlations between them even in the prior due to higher layers. To learn the weights between the first hidden layer and the data, one needs to know the posterior in the first hidden layer or an approximation to it. Computing the prior term involves integrating out all the hidden variables in higher layers, making it computationally complex.

Monte Carlo Method by Radford Neal:
This method considers patterns of activity over all hidden variables and runs a Markov chain to obtain samples from the posterior. However, it is computationally slow, especially for large deep belief nets.

Variational Methods:
Variational methods approximate the posterior distribution with a simpler distribution. While they do not provide unbiased samples from the posterior, they can still be used for maximum likelihood learning. Variational methods guarantee an improvement in a lower bound on the log probability that the model generates the data.

Wake-Sleep Algorithm:
The wake-sleep algorithm is a variational method specifically designed for directed graphical models like sigmoid belief nets. It involves two phases: Wake phase: The model is trained using the data to update the weights. Sleep phase: The model is run as a generative model to generate samples from the approximate posterior distribution. The wake-sleep algorithm led to the development of variational learning, a widely used approach for learning complicated graphical models in machine learning.

Background:
Sigmoid belief nets are complicated models that are difficult to learn due to the challenge in obtaining samples from the true posterior distribution over hidden configurations.

Proposed Solution:
Variational learning involves using an approximation to the posterior distribution, rather than the true distribution, to drive the learning process.

Key Points:
During learning, two terms drive the weight adjustments: One term aims to improve the model’s fit to the data. The other term pushes the weights towards values that make the approximate posterior a good fit for the real posterior. This process is effective in learning sigmoid belief nets, as demonstrated by experiments conducted in the mid-90s. The concept of variational learning has been adopted by Carl Friston, who believes it plays a significant role in real neural learning.

Challenges:
The difficulty in learning sigmoid belief nets lies in the complexity of the model and the associated challenges in obtaining unbiased samples from the true posterior distribution.

Alternative Approach:
Variational learning addresses this challenge by employing an approximation to the posterior distribution, which assumes that the posterior over hidden configurations factorizes into a product of distributions for each separate hidden unit. This assumption simplifies the learning process, though it is incorrect in the context of sigmoid belief nets.

Next Steps:
The presentation will delve into the details of how variational learning is applied to sigmoid belief nets, focusing on the factorization of the posterior distribution and its implications.

Wake-Sleep Algorithm:
A generative model with two sets of weights: recognition weights for approximate posterior distribution and generative weights for defining the model’s probability distribution.

Wake Phase:
Data is input at the visible layer. Forward pass using recognition weights generates stochastic binary states for hidden units. Maximum likelihood learning is performed for the generative weights.

Sleep Phase:
Random vector starts at the top hidden layer. Binary states are generated from the prior and propagated down the system using generative weights. Recognition weights are trained to recover hidden states from the generated data.

Flaws of the Algorithm:
Recognition weights learn to invert the generative model in data-sparse regions. Recognition weights don’t follow the correct gradient, leading to incorrect mode averaging. Independence approximation for the posterior distribution may be inaccurate.

Mode Averaging:
Due to the independence approximation, the recognition distribution assigns equal probability to multiple modes of the posterior distribution. This can result in poor recognition performance, as the model fails to select the most likely mode.

Criticisms:
Carl Friston proposed that the brain uses this algorithm, but Hinton believes it has too many problems. Hinton suggests that better algorithms exist for learning generative models.

Bimodal True Posterior:
The posterior distribution of the hidden units is bimodal, meaning it has two peaks. The peaks are located at (0, 0) and (1, 1), indicating that the hidden units are most likely to be either all 0s or all 1s.

Approximation:
The approximation learned using the sleep phase of the wake-sleep curve gives equal probability to all four states of the hidden units. This approximation is not as accurate as the true posterior, but it is simpler to work with.

Best Solution:
The best solution would be to pick one of the four states and give it all the probability mass. This would be the most accurate approximation of the true posterior, but it is not always feasible to find this solution.