Three Theories of Knowledge Storage in the Brain:
Separate Appearance Models for All Viewpoints: This theory is unrealistic as it requires learning an enormous number of models. Shared Weights for Translations: This theory incorporates translation invariance but still requires learning how to handle rotations and scalings. Viewpoint-Independent Knowledge in Weights: This theory proposes storing knowledge about the relationships between wholes and parts in the weights, making neural activities viewpoint-equivariant.

Accessing Knowledge Stored in Weights:
Mental Image Formation: Viewpoint-independent knowledge is converted into viewpoint-dependent neural activities by forming a mental image. Example: When comparing the distance between a German shepherd’s ears and eyes, one forms a mental image of the dog’s head to read the comparative differences. Mental Image Characteristics: Mental images are not necessarily visual pixels but rather a representation of information about wholes and parts from a specific viewpoint. Viewpoint Selection: Forming a mental image involves choosing a viewpoint that is neither extreme nor rotated.

Defining Spatial Relationships:
Mathematically, a spatial relationship is a coordinate transformation mapping points in one rectangular coordinate frame to another. People impose rectangular coordinate frames on objects to represent their shapes.

Psychological Evidence:
Irving Rock’s Experiment: Participants perceived a tilted country as Africa only after being informed of its 45-degree tilt. The imposed frame of reference determines the perceived shape. Diamond or Square: The perception of a shape as a tilted square or upright diamond highlights different features. Alignment with a rectangular coordinate frame, not right angles, is crucial.

MIT Professors’ Puzzle Challenge:
Two identical shapes viewed from different perspectives can be assembled into a tetrahedron. Despite its simplicity, MIT professors took over a minute to solve the puzzle. The longer a professor had been at MIT, the more time they took to solve it.

Coordinate System Misalignment:
A natural coordinate system based on the tabletop and gravity is imposed on the pieces. This natural coordinate system differs from the rectangular coordinate system used for perceiving a tetrahedron. The misalignment of coordinate systems makes it difficult to see how the pieces fit together.

Alternative Solutions:
Solving the puzzle using other strategies, such as assembling faces with triangular remainders, simplifies the task.

Theory 3: Identity-Specific Capsules:
Proposes that knowledge about a specific shape is localized into a specific group of neurons called a capsule. Each capsule knows how all the parts are related to the whole through its weights. Shapes are recognized when multiple different parts predict the same pose for the whole. The pose of the whole refers to the coordinate transform between the retina and the intrinsic frame of the whole or the part.

Universal Capsules:
Introduced a newer variation of Theory 3 with universal capsules that are not specific to particular shapes. Universal capsules are ubiquitous and dedicated to representing whatever occupies a location at a certain level of representation. The goal is to have a hierarchy of embeddings at different levels, where each location has a universal capsule that represents the object, part, or subpart at that location.

Islands of Agreement:
The embeddings at different levels form islands of agreement, which represent the paths of the image and how it’s segmented into holes and parts at multiple levels. These islands of agreement can be thought of as a multilevel real-valued Ising model with coordinate transforms between levels.

Glom Architecture:
Represents the architecture for a single location or column in the hierarchy of embeddings. It consists of neurons set aside for multiple levels of representation, forming a recurrent net that uses the same weights at each location.

Overview:
Hinton presents a novel approach to image segmentation using capsules and distillation. The method involves bottom-up and top-down neural networks interacting with each other and with neighboring locations. Attention-weighted averaging is used to form cohesive groups of embeddings, creating “echo chambers” that aid in segmentation. The bottom-up neural network allows ambiguous parts to disambiguate each other through coordinate transforms. A Hough transform is used to predict the identity of the whole object occupying a location, rather than direct interactions between parts.

Bottom-Up and Top-Down Neural Networks:
Each location has bottom-up and top-down neural networks that predict the presence of specific features. Bottom-up networks take input from lower levels and predict features at the current level. Top-down networks take input from higher levels and predict features at the current level. These networks interact to refine predictions and disambiguate ambiguous parts.

Attention-Weighted Averaging:
Attention-weighted averaging is used to combine predictions from different locations. It helps form cohesive groups of embeddings, creating “echo chambers” that reinforce each other’s predictions. This process aids in image segmentation by grouping similar features together.

Coordinate Transforms for Disambiguation:
The bottom-up neural network allows ambiguous parts to disambiguate each other through coordinate transforms. A part seeking confirmation from another part transforms its coordinates to match the expected pose of the other part. This allows parts to confirm or deny each other’s presence based on their relative positions.

Hough Transform for Whole Object Prediction:
Instead of direct interactions between parts, a Hough transform is used to predict the identity of the whole object occupying a location. Each part makes an ambiguous multimodal prediction for the identity and pose of the whole object. Attention-weighted averaging is used to find the common mode among these predictions, identifying the most likely whole object.

Weight Sharing and Distillation:
The approach requires the bottom-up and top-down neural networks to be the same at every location, which may seem wasteful in the brain. Hinton proposes using distillation to extract knowledge from a large model with many parameters and transfer it to a smaller model with fewer parameters. This allows weight sharing and reduces the computational cost of learning multiple networks at each location.

Distillation as Metamorphosis:
Hinton draws an analogy between distillation and metamorphosis in insects. The large model is like a caterpillar, optimized for extracting structure from data but cumbersome to use. Distillation is like the transformation from caterpillar to butterfly, creating a smaller, more efficient model with the same knowledge.

Transferring Knowledge from Big Models to Small Models with Knowledge Distillation:
In knowledge distillation, the function learned by a large model is transferred to a smaller model with a different architecture.

Information Revealed by Softened Outputs:
Hard targets (e.g., one-hot encoding) only impose a small amount of constraint on the function learned by a large model, hiding a lot of information about the function. Softening the outputs of the large model by using a high temperature in the softmax reveals more information about the function, including “dark knowledge” that would otherwise be hidden.

Benefits of Softened Outputs for Training Small Models:
Training small models on softened outputs from a large model provides more informative data for learning. Softened outputs reveal class hierarchies, allowing small models to learn the relative plausibility of different classes, including wrong answers.

Examples from MNIST:
Examples from MNIST demonstrate how softened outputs provide richer information for learning. A “2” that looks like a “0” provides more information than just a “2.”

Distillation Experiment on MNIST:
A small model trained with softened outputs from a large model achieves better performance compared to a small model trained with hard targets. This demonstrates the effectiveness of knowledge distillation in transferring knowledge from big models to small models.

Transfer Knowledge from a Big Neural Network to Smaller Networks:
Train a large neural network and then transfer its knowledge to a smaller network through distillation. The smaller network benefits from the knowledge and generalization abilities of the big network. The distilled network achieves a lower error rate compared to training directly on data.

Training the Small Network with Soft Targets:
Instead of training the small network with hard targets (e.g., one-hot encoding), use soft targets. Soft targets provide the network with information about the second-best answer, enabling generalization. This approach trains the small network to generalize similarly to the big network.

Distillation for a Specific Class:
Distillation can be applied to teach a small network about a specific class, even if it has never seen examples of that class during training. By adjusting the bias of the output category, the network can learn to recognize the new class.

Co-Distillation for Multiple Networks:
Instead of training a single big network and then distilling knowledge to smaller networks, use co-distillation. Train multiple small networks simultaneously and encourage them to agree with each other’s softened answers. This peer group pressure approach improves the performance of the individual networks.

Co-Distillation:
Ensembles of neural networks can improve individual network performance by sharing knowledge during training. Co-distillation is a technique where networks learn to agree with each other’s predictions, even if they see different subsets of data. This allows knowledge transfer between networks, including knowledge about data that a network has never seen.

Consensus Embedding:
The consensus embedding is an average of predictions from multiple sources, including bottom-up, top-down, and nearby columns. The consensus embedding provides a better training signal for bottom-up and top-down neural networks. Knowledge transfer occurs when nearby columns provide opinions about the same object or part of an object.

Co-Distillation Advantages:
Co-distillation can transfer knowledge even if columns pre-process their input differently, unlike weight sharing. Co-distillation can be applied to non-uniform retinas, where the size of receptor fields and the spacing of photoreceptors vary.

Conclusion:
The brain’s inability to do weight sharing does not prevent it from gaining the benefits of convolutional neural networks. Co-distillation allows local neural networks to share knowledge and achieve better performance. This mechanism may underlie the knowledge transfer and efficient processing observed in the brain’s retinotopic maps.