Introduction:
Geoffrey Hinton, a pioneer in deep learning, shared the 2018 Turing Award for his work on Capsule Networks. Hinton has been largely silent since then and is interviewed to discuss his recent work on Capsule Networks, unsupervised learning, and NGRADS.

Capsule Networks:
Hinton’s Capsule Networks are an alternative to Convolutional Neural Networks that take into account the pose of objects in a 3D world. Capsule Networks solve the problem in computer vision in which elements of an object change their position when viewed from different angles.

Stopped Capsule Autoencoders:
Hinton’s recent work on stopped capsule autoencoders was presented at NeurIPS in December 2021. Stopped capsule autoencoders are a type of unsupervised learning algorithm that can learn to generate new data that is similar to the data it was trained on.

Capsule Networks for Unsupervised Learning:
Hinton believes that Capsule Networks are key to unsupervised learning, which is a type of learning that does not require labeled data. Unsupervised learning is a challenging problem in machine learning, and Hinton’s work on Capsule Networks is a promising step towards solving it.

Neural Gradient Representation by Activity Differences (NGRADS):
Hinton’s recent work on NGRADS was presented at the AAAI conference in February 2022. NGRADS is a new learning function that is inspired by the way the brain learns. NGRADS is a more efficient and effective learning function than traditional backpropagation.

Conclusion:
Hinton’s work on Capsule Networks, unsupervised learning, and NGRADS is at the forefront of machine learning research. Hinton is a brilliant and influential researcher who is making significant contributions to the field of artificial intelligence.

Unsupervised Learning:
Switched from supervised to unsupervised learning for capsule networks. Unsupervised learning is preferred since labels are not required.

Set Transformers:
Set transformers are used instead of capsules to allow parts to interact and become more confident about their type. This disambiguates the parts, making them more specific and confident in their votes for forming objects. Results in fewer votes to deal with and easier clustering.

Stack Capsule Autoencoders:
Stack Capsule Autoencoders are used to connect parts to wholes. The objective function is to find holes (combinations of parts) that are good at reconstructing the parts. This is done without labels, making the learning unsupervised.

Hole Learning:
Holes are combinations of parts that are good at reconstructing the parts. Once holes are learned, they can be associated with names through supervised learning. This is similar to how a child learns to recognize objects and their names.

Relationships and Laws of Physics:
Capsule networks can recognize objects by seeing parts in the correct relationships. In scenes, objects can be recognized in the correct relationships to form a particular kind of scene. The ability to infer relationships between objects and learn laws of physics is a long-term goal.

Sinclair:
Sinclair is a different learning algorithm that is not focused on viewpoint equivariance like capsule networks. Sinclair aims to learn to represent patches of images in a way that allows for efficient processing.

Unsupervised Learning with Contrastive Learning:
Contrastive learning involves creating neural representations of image crops such that similar representations are obtained for crops from the same image and different representations for crops from different images. Ting Chen’s work on contrastive learning with a ResNet architecture significantly improved its performance on ImageNet, achieving comparable results to supervised methods. Data augmentation techniques, such as modifying color balance, are crucial to prevent the model from relying on simple features like color distribution for classification.

Capsule Networks and SimClear:
Capsule networks and SimClear are different approaches to unsupervised learning that aim to capture relationships between image parts. Combining these methods could potentially enhance their performance, although it is not currently being explored.

NGRADS and Backpropagation in the Brain:
NGRADS (Neural Gradient Representation by Activity Differences) proposes a mechanism for error representation and learning in the brain using activity differences. Activity differences, or temporal differences in activity, may serve as error derivatives for spike-time dependent plasticity, a learning rule observed in the brain. Hinton suggests that the brain’s learning problem differs from that of neural networks due to the large number of parameters and limited training data in the brain.

Back Relaxation:
Back relaxation is a proposed learning algorithm for the brain that involves comparing top-down predictions with bottom-up extractions of parts in a hierarchical representation. This method is simpler to implement in the brain compared to backpropagation and may be better suited for the brain’s learning scenario with many parameters and limited training data.

Back Relaxation vs. Greedy Bottom-Up Learning:
Geoff Hinton initially showed interest in back relaxation as a potential explanation for how the brain learns multilayer nets. However, he later discovered that greedy bottom-up learning performs comparably to back relaxation, leading to disappointment. Hinton expressed a desire to revisit back relaxation to see if it can outperform greedy bottom-up learning.

Top-Down Prediction and Bottom-Up Extraction:
Hinton emphasizes the importance of top-down prediction and bottom-up extraction working together for efficient learning. Training a stack of autoencoders one layer at a time, followed by fine-tuning, has been shown to work well.

Deep Learning and Unsupervised Pre-training:
Deep learning gained momentum in 2006 with the discovery that training stacks of autoencoders or restricted Boltzmann machines one layer at a time and then fine-tuning yielded good results. Supervised learning became prevalent, but as datasets and networks grew larger, researchers returned to unsupervised pre-training. Unsupervised pre-training is now widely used in models like BERT and GPT-3.

Current Unsupervised Learning Algorithms:
Craig Smith mentions SimClear and Stack Capsule Autoencoders as sophisticated unsupervised learning algorithms, particularly relevant in computer vision.

Temporal Difference Learning and Cortex Learning:
Temporal difference learning, studied by Rich Sutton, is considered to describe learning in lower brain functions. Geoff Hinton highlights the success of computational neuroscience in relating temporal differences to experimental studies on the brain and dopamine. The work of Peter Dayan is particularly notable in establishing this connection.

Reinforcement Learning as Icing on the Cake:
Most learning in AI should be unsupervised, focusing on understanding how the world works without relying on reinforcement signals.

Convergence of Computer Vision and NLP:
Capsule networks aim to represent objects more like humans do, allowing for multiple interpretations of the same image.

Frames of Reference:
Humans impose frames of reference to understand objects, which is a crucial aspect of perception that neural networks need to incorporate.

Google’s Patent Filing for Capsule Networks:
Google’s patent filing is primarily protective, aiming to prevent others from suing them for using their own research in products.

Transformers for Image Recognition:
Transformers, known for their effectiveness in NLP, can also be applied to image recognition by enabling interactions between parts of an image.

Massive Parameters in Transformers:
Transformers require a large number of parameters, potentially leading to a search-like approach when ingesting and matching representations.

Capsule Networks and Stack Capsule Autoencoders:
Stack capsule autoencoders combine unsupervised learning with transformer-like interactions, refining representations and jumping to high-level concepts.

Similarities and Differences:
Similarities exist between capsule networks and the use of transformers in NLP, but they differ in training methods and specific applications.

AI Constructs Representations:
AI fills in missing data with existing data. It produces similar results to observed data without directly matching specific instances. Capsule networks create new representations and can handle new views of the same object.

Geoff Hinton’s Research Interests:
Unsupervised learning: Hinton believes most human learning is unsupervised. Developing capsules further: He wants to improve capsule networks. SimClear: A self-supervised learning method for contrastive representation learning. Distillation: Training a smaller model using a larger model’s knowledge.

Data Mining Analogy:
Hinton compares data mining to mining gold. Big models extract structure from data, like extracting gold from pay dirt. Smaller models are more agile and easier to use, like using gold to make jewelry.

Capsule Networks and Yann LeCun’s Research:
Hinton and LeCun share similar intuitions and goals. Both are exploring unsupervised learning and contrastive representation learning. They are extending these methods to video using attention mechanisms.

Machine Learning and Human Learning:
Hinton believes unsupervised learning is analogous to human learning. Capsule networks learn representations that can be connected to language. Robots using deep learning can understand and respond to language commands.

Capsule Networks and Supervised Learning:
Initial versions of capsule networks used supervised learning for simplicity. Unsupervised capsule networks now perform better and align with Hinton’s beliefs. Language is used to demonstrate the learned representations’ meaningfulness.

NLP and Understanding:
NLP’s ability to comprehend tasks, such as opening a drawer and retrieving a block, makes it challenging to argue that it lacks understanding.

Physics Learning without Language:
Physics learning, like understanding the impact of throwing an object, does not require linguistic input.

Skill Acquisition through Trial and Error:
Skills, such as throwing a basketball through a hoop, are learned through trial and error, not explicit language-based instruction.

Perception in Robots:
Robots’ need to decide where to focus attention shifts the emphasis from static image processing to attention mechanisms.

Convergence of Disciplines:
Unifying computer vision, natural language processing, unsupervised learning, supervised learning, and reinforcement learning is beyond the immediate focus of basic research.

Supervised vs. Unsupervised Learning:
The distinction between supervised and unsupervised learning is often misunderstood.

Correlation-Based Learning:
Geoff Hinton proposes that learning, whether supervised or unsupervised, involves identifying correlations between sensory inputs. In supervised learning, a label (e.g., “cow”) is provided, creating a correlation between visual and auditory inputs. In unsupervised learning, correlations are identified without explicit labels.

The Role of Correlations in Learning:
Hinton emphasizes the importance of correlations in learning, regardless of whether they are supervised or unsupervised. He believes that most learning is unsupervised, as correlations with payoffs (reinforcement learning) often lack sufficient structure for efficient learning.

Payoff-Based Learning:
Hinton acknowledges the role of reinforcement learning, where correlations are associated with payoffs or rewards. However, he suggests that payoff-based learning alone may not be sufficient for comprehensive learning.

Conclusion:
Hinton’s perspective highlights the significance of correlations in sensory inputs as the fundamental mechanism underlying both supervised and unsupervised learning. While reinforcement learning plays a role, Hinton emphasizes the importance of unsupervised learning in acquiring knowledge.