Neural Network Limitations:
Current neural networks lack sufficient structure and levels of organization. They do not explicitly represent entities and their properties.

Capsule Networks:
Capsules are proposed as a new type of neural network architecture that addresses these limitations. Each capsule represents an entity and its properties, such as orientation, size, velocity, and color. Capsules communicate with each other hierarchically, passing on information about the entities they represent.

Capsule Computation:
Capsules receive predictions from lower-level capsules about the generalized pose of an entity. They identify and agree upon consistent predictions, ignoring outliers. This process is similar to computer vision techniques like RANSAC and Hough transforms.

Potential Benefits:
Capsule networks may be more robust to noise and occlusions than traditional neural networks. They may also be better at modeling complex relationships between entities.

Future Research Directions:
Hinton discusses potential future research directions for capsule networks, including: Investigating different ways to represent entities and their properties. Developing more efficient algorithms for training capsule networks. Applying capsule networks to a wider range of tasks, such as natural language processing and reinforcement learning.

Capsule Pose Space:
Capsules have high-dimensional pose spaces, which are represented as 20-50 dimensional spaces. Capsule predictions are vectors that represent the pose of the capsule. The goal is to find a cluster of predictions that indicate the presence of a capsule.

High-Dimensional Coincidences:
High-dimensional coincidences are unlikely to happen by chance. The more dimensions in which two things agree, the less likely it is that their agreement is a coincidence. This principle can be used to detect meaningful patterns in data, such as intelligence information.

Marian Perspective on Computing in the Brain:
To understand the brain, we need to understand the computations it performs. One computation that the brain needs to perform is detecting high-dimensional coincidences. This computation may be performed by columns in the brain.

ConvNets for Object Recognition:
ConvNets are neural networks that use multiple layers of learned feature detectors for object recognition. Feature detectors are local and replicated across space, and their spatial domains get bigger as they go up the network. Pooling layers reduce the dimensionality of the feature maps and provide translational invariance.

Psychological Misfit:
Hinton argues that CNNs, specifically the pooling layers, are a poor fit for the psychology of shape perception. He emphasizes the importance of rectangular coordinate frames in human shape recognition and how a slight shift in the coordinate frame can make the same object unrecognizable. CNNs lack the notion of imposing coordinate frames, making it difficult to explain how the same pixels can be processed differently based on the coordinate frame.

Tetrahedron Puzzle:
Hinton presents a simple puzzle involving two pieces of a tetrahedron that most people find challenging to reassemble. He demonstrates how people’s natural coordinate frame for the tetrahedron differs from the one imposed on the individual pieces, leading to the puzzle’s difficulty. This puzzle highlights the importance of coordinate frames in shape perception and the limitations of CNNs in capturing this aspect.

Map Perception:
Hinton uses an example of a map to illustrate how the perception of familiar objects can be influenced by the imposed coordinate frame. People’s inability to recognize a map of Africa when the orientation is altered demonstrates the crucial role of coordinate frames in object recognition.

Pooling as a Routing Problem:
Hinton criticizes the pooling operation in CNNs, viewing it as a primitive approach to routing information. He emphasizes the need for a more sophisticated routing mechanism to handle viewpoint changes and dimension hopping, where information moves from one set of pixels to another. He draws a parallel to the coding of patient records in hospitals, where mixing different coding schemes can hinder machine learning efforts.

Underlying Linear Manifold:
Hinton points out that CNNs fail to utilize the underlying linear manifold, which is a powerful tool in computer graphics and is believed to be involved in human shape perception. He argues that CNNs’ lack of attention to this linear manifold limits their ability to deal with viewpoint effects effectively.

Conclusion:
Hinton concludes his critique by asserting that CNNs, specifically the pooling aspect, are not psychologically sound and do not align well with the way humans perceive shapes. He highlights the importance of coordinate frames and routing mechanisms in shape perception and argues for a better understanding of these aspects in neural network architectures.

Argument for Coordinate Frames:
Hinton argues that the human visual system uses rectangular coordinate frames embedded in objects and parts of objects to represent their shape and position. Evidence for this comes from studies showing that people can accurately judge the orientation of objects within 1 degree, even when they are rotated or tilted. Hinton proposes that these coordinate frames are represented by a bunch of separate neural activities, rather than a single neuron, which allows for more precise representation.

Equivariance:
Hinton discusses the concept of equivariance, which refers to the property of a neural representation that changes in a predictable way when the input changes. He distinguishes between place equivariance, where the active neurons change as the input moves across space, and rate-coded equivariance, where the activity of the same neurons changes as the input moves. Hinton suggests that the visual system uses both types of equivariance at different levels of processing, with place equivariance at lower levels and rate-coded equivariance at higher levels.

Linear Manifolds:
Hinton proposes that the visual system uses linear manifolds to represent objects and their properties. This means that the relationship between the input pixels and the internal representation of an object is linear, allowing for efficient extrapolation and generalization across different viewpoints. Hinton argues that current neural networks lack this built-in bias towards linear generalization, which limits their ability to generalize across viewpoints without extensive training.

Inverse Graphics:
Hinton suggests that vision can be thought of as inverse graphics, where the goal is to reconstruct the 3D structure of a scene from 2D images. He proposes that the visual system performs this reconstruction by using a process that is similar to computer graphics in reverse, where it starts with the input pixels and gradually builds up a representation of the objects in the scene.

Conclusion:
Hinton’s arguments provide strong evidence for the use of coordinate frames, equivariance, and linear manifolds in the human visual system. These concepts can help to explain how the visual system can achieve high accuracy and generalization across different viewpoints, even with limited training data. Hinton’s proposal to use inverse graphics as a model for vision offers a promising direction for developing more powerful and efficient neural networks for computer vision tasks.

Inverse Graphics for Viewpoint Invariance:
Hinton advocates for using inverse graphics in computer vision, which involves representing the relationship between a whole and its parts as a matrix of weights. This matrix of weights remains constant regardless of viewpoint changes, providing complete viewpoint invariance.

Invariance in Weights vs. Neural Activities:
Hinton emphasizes the importance of invariance in the weights rather than in neural activities for robust shape recognition. Perfect invariance in the weights is achievable through inverse graphics, leading to viewpoint-invariant representations.

Limitations of Capsules and Crowding:
Hinton acknowledges the limitations of capsules in handling multiple instances of the same entity due to their simultaneity-based binding mechanism. Crowding occurs when objects are placed too close together, impairing perception.

Shape Recognition in the 1980s:
Hinton presents a method for shape recognition from the 1980s involving the identification of familiar parts, such as noses and mouths. Logistic units estimate the probability of a part’s presence, and matrices operate on pose parameters to predict the pose of larger parts from smaller parts.

Coincidence Filtering and Hough Transforms:
Predictions from different parts are compared using coincidence filtering to determine the presence of a larger part, such as a face. This approach is analogous to Hough transforms, but with high-dimensional features extracted through machine learning for reliable point predictions.

Modern Half-Transforms:
Hinton distinguishes his approach from traditional Hough transforms by utilizing high-dimensional features obtained through machine learning. This enables point predictions without the need for bins and multiple votes, resulting in more robust shape recognition.

Routing Visual Information Based on Viewpoint:
Viewpoint changes affect the location of objects in an image. Traditional ConvNets use pooling units to route information based on activation strength.

Geoffrey Hinton’s Proposed Routing Principle:
Route information to capsules that can interpret it. Assume the world is opaque and can be modeled by a parse tree. Each part has one parent (or none) according to the single parent constraint.

Sending Weak Bets to Multiple High-Level Capsules:
When a low-level capsule discovers a part (e.g., a circle), it sends its pose to multiple high-level capsules. Each high-level capsule receives multiple weak bets from different low-level capsules.

Clustering and Agreement in High-Level Capsules:
High-level capsules look for clusters of weak bets that agree. Initially, the capsule relies on prior probabilities to determine which high-level capsule to send the information to. Clusters of weak bets represent parts belonging to the same parent object.

Top-Down Feedback and Lateral Interactions:
Top-down feedback and lateral interactions refine the routing process. Capsules that can interpret the information request more output from the low-level capsule. Capsules that cannot interpret the information request less output from the low-level capsule.

Establishing Parse Trees:
Through iterations of top-down feedback and lateral interactions, the routing process establishes a parse tree. The parse tree defines the hierarchical relationships between parts and objects in the image.

Introduction of Capsule Networks:
Hinton proposes a novel approach to neural networks called capsule networks, which aim to overcome the limitations of traditional convolutional neural networks.

Bilateral Interaction and Routing by Agreement:
Hinton suggests two mechanisms for routing information between capsules: bilateral interaction and top-down routing by agreement. Bilateral interaction involves adjusting the weights between capsules based on their agreement, allowing for more precise routing. Top-down routing by agreement utilizes consistency to direct information to appropriate capsules.

De-rendering and Parts-Based Hierarchy:
Hinton introduces the concept of de-rendering, which involves transforming pixel intensities into poses, enabling the network to understand the geometric relationships between objects. He envisions a deep system that can learn a parts-based hierarchy, where larger entities are assembled from smaller ones, without the need for manual engineering.

Proof of Concept with MNIST Digits:
Hinton demonstrates the basic principles of capsule networks using MNIST digits as an example. The network consists of two levels: primary capsules that extract pose parameters from patches of the image and secondary capsules that predict the pose and presence of digits based on the primary capsules’ output.

Picasso Weights and Geometric Constraints:
Picasso weights are used to predict the presence of an object based solely on its type, disregarding geometric constraints. The network combines the predictions from different primary capsules to make a final prediction, taking into account both geometric and type-based information.

Conclusion:
Hinton concludes the presentation by emphasizing the potential of capsule networks for building deep systems that can learn a parts-based hierarchy and understand the geometric relationships between objects.

Coordinate Transformations for Primary Capsules:
Each type B capsule predicts the pose of a digit based on the black patch. Weight matrix relating pose of type A feature to prediction is the same for all patches. To account for patch offset, first two coordinates of pose prediction are adjusted by patch location.

Backpropagation for Class Agreement:
Goal: Maximize agreement for correct class and minimize agreement for incorrect classes. Measure of agreement needed for backpropagation. Learn weights to convert pixel intensities to primary capsule poses, coordinate transforms, and biases.

Coordinate Transformations for High-Level Capsules:
Linear transformations are used to obtain pose of larger thing from pose of feature.

Prediction and Agreement in High-Level Capsules:
Each high-level capsule receives predictions from multiple primary capsules. Predictions are weighted and summed, with weights treated as fractions of an observation. High-level capsule seeks agreement among predictions.

Gaussian Mixture Model for Agreement:
Mixture of Gaussian and uniform distribution fitted to predictions. Model fit is evaluated by comparing log probabilities. Mean, variance, and mixing proportion of Gaussian are dynamic parameters. Coordinate transforms and initial bet weights are fixed.

Cluster Score and Routing:
Cluster score quantifies how good a cluster of predictions is.

Background:
Hinton introduces a novel approach to digit classification using capsule networks. Capsule networks utilize a mixture of Gaussian and uniform distributions to model the data. The goal is to find clusters of data points and assign them to different classes.

Key Points:

1. Calculating the Score:
The score is computed as the difference in log probabilities of the data under a mixture of Gaussian and uniform distributions. The score helps identify clusters of data points and determine the class of each data point.

2. Softmax and Decision Making:
The score is used as the logit in a Softmax function to make decisions about the class of a data point. Backpropagation is employed to adjust the network’s weights based on the decision made.

3. Inner Loop for EM and Cluster Finding:
An inner loop involving Expectation-Maximization (EM) is used to find clusters of data points. The EM algorithm iteratively refines the cluster assignments and the parameters of the Gaussian distribution.

4. Interpreting the Results:
The votes for each class are visualized as clusters, with the size of each cluster representing the posterior probability of the data point belonging to that class. The variance of the Gaussian distribution indicates the sharpness of the cluster.

5. Comparison with Convolutional Neural Networks:
Capsule networks achieve similar performance to convolutional neural networks on the MNIST dataset. However, capsule networks are computationally more expensive due to the inner loop for EM.

6. Limitations and Future Improvements:
The current approach focuses on single-digit classification and lacks deeper hierarchies and real image processing capabilities. Future work aims to incorporate unsupervised learning to obtain primary capsules and explore more efficient algorithms for cluster finding.

Conclusion:
Hinton presents a novel capsule network architecture for digit classification that utilizes a mixture of Gaussian and uniform distributions to model the data. The network identifies clusters of data points and assigns them to different classes based on the calculated score. While the approach is computationally expensive, it demonstrates the potential of capsule networks for various computer vision tasks.

Applying Graphics to Capsule Networks:
A graphics system is incorporated into the capsule network to reconstruct images from extracted capsules. The graphics system learns templates for each entity and can translate, scale, and add them to the image.

Unsupervised Learning of Templates and Pose Parameters:
The network learns to extract capsules representing entities and their poses unsupervised. It discovers templates that can be combined to form digits.

Supervised Learning with Mixture of Factor Analyzers:
After unsupervised learning, supervised learning is applied to labeled data. A mixture of factor analyzers is fitted to the concatenated pose parameters of the capsules. This captures the relationship between the whole and its parts.

Results:
With 25 factor analyzers, the network achieves 1.75% error on MNIST with only 25 questions (class labels). This is comparable to human performance and significantly better than previous unsupervised methods.

Comparison with Other Methods:
Unsupervised pre-training with autoencoders followed by supervised learning yields similar results to previous methods. Mixture factor analyzers applied directly to pixels perform worse than the proposed method.

Conclusion:
The unsupervised learning of templates and pose parameters, followed by supervised learning with a mixture of factor analyzers, enables digit classification with very few labeled examples.

Hinton’s Theory of Object Perception:
Very young children track proto-objects that aren’t fully fleshed out, possibly due to their low-resolution perception helping them learn more easily. The relational aspect of object perception, such as the linear transformation of objects, is hardwired into the system.

Mental Rotation and Recognition:
Humans are good at recognizing objects and finding orientations quickly, within about 250 milliseconds. Mental rotation of objects, especially in 3D, is a much slower process, taking over 100 milliseconds.

Computational Time for Learning:
Hinton’s algorithm for learning object representations takes two days, compared to existing algorithms that run in 10 minutes. Heuristic approximations and computational optimizations can be used to significantly reduce the learning time.

Core of Hinton’s Algorithm:
Unlike traditional neural networks, Hinton’s algorithm focuses on finding agreement between activity vectors rather than between weight vectors and activity vectors.

Efficiency in Capsule Networks:
Capsule networks aim to find agreement between activity vectors, essentially chasing the covariance structure. The computation involved in finding these agreements efficiently among high-dimensional random data presents a challenge. Hinton suggests he has ideas for improving this efficiency but will discuss them only if they prove successful.

Capsule Networks for Speech Recognition:
Hinton’s graduate student attempted to apply capsule networks to speech recognition. The task proved challenging, but the student managed to achieve performance comparable to standard neural networks using capsule-based ideas. Capsule networks appear to be more naturally suited for vision tasks.

Panel Discussion:
Hinton reminds the audience of a panel discussion on “The Path to Intelligence” scheduled for 6 pm in the same room. Attendees are invited to provide feedback and engage in discussion during the panel.