Different Models Notice Different Things:
Training multiple diverse models extracts more knowledge from large datasets. Each model focuses on different aspects of the data. The more diverse the models, the better.

Ensemble Learning:
At test time, averaging predictions from multiple models improves performance. This technique is commonly used to win machine learning competitions.

Challenge for Practical Applications:
Running numerous models for real-time tasks (e.g., speech recognition) is computationally expensive. Ensembles are redundant, with limited knowledge per parameter. They lack the efficiency and speed required for practical applications.

Insects as an Analogy:
Insects need energy from food and agility for flight and mating. These requirements are incompatible. Solution: Two distinct forms – larva for feeding and adult for reproduction. Larva is optimized for eating, while the adult is designed for mobility and reproduction.

Knowledge Extraction from Ensembles:
Geoffrey Hinton compares knowledge extraction from large data sets to mining for gold. The aim is to extract knowledge in a “clumsy” form, such as a large ensemble of models, and then refine it into a smaller, more efficient model. This two-stage process involves obtaining a function that maps inputs to outputs and then fitting that function into a smaller model.

Multi-way Classification:
Hinton focuses on multi-way classification tasks with large neural networks using n-way softmax outputs. During training, the targets are typically 1s and 0s, resulting in limited constraint on the model’s parameters.

Soft Outputs and High-Temperature Softmax:
Hinton introduces the concept of using a high-temperature softmax to generate soft outputs. This approach reveals more information about the function learned by the ensemble model. Hinton compares it to a teacher providing more detailed feedback, such as identifying a painting as a Rubens with characteristics similar to Titian.

Averaging Model Predictions:
Two methods for averaging model predictions are discussed: Mixture way: Averaging class probabilities directly. Geometric mean: Averaging logits before applying the exponential function.

Dark Knowledge:
Hinton introduces the concept of “dark knowledge,” which refers to knowledge that is not captured by the cross-entropy cost function due to very small probabilities assigned to target values. This knowledge is represented as black regions in a graphical display of the output probabilities.

Example of Extracting Dark Knowledge:
Hinton provides an example of extracting dark knowledge from an ensemble model for object recognition. The ensemble model assigns high probabilities to the correct class (dog) and low probabilities to incorrect classes (cat, cow, car). The dark knowledge lies in the very small probabilities assigned to the incorrect classes, which are ignored by the cross-entropy cost function.

Extracting Dark Knowledge:
Geoffrey Hinton emphasizes that much of the knowledge learned by a classifier lies in the relative probabilities of things that are almost certainly wrong. This “dark knowledge” is often overlooked.

Raising the Temperature of Softmax:
To extract this dark knowledge, Hinton employs a technique called distillation. The first step involves raising the temperature of the softmax function. By increasing the temperature, the probabilities of less likely outcomes become more prominent, allowing for richer information in the target values.

Using Soft Targets for Knowledge Transfer:
Training models with soft targets, obtained from a larger ensemble, enables efficient knowledge transfer to smaller models. The soft targets provide more information compared to hard targets, leading to better transfer results.

Combining Soft and Hard Targets:
Hinton suggests combining soft and hard targets during training for optimal results. Soft targets are weighted less than hard targets due to their smaller gradients.

Distillation Process:
The distillation process involves training a smaller model to match both soft and hard targets. The model uses a high temperature softmax to match soft targets and a temperature of 1 to match hard targets. Using two different temperatures in a softmax for the same model yields the best results.

Efficient Training of Big Ensembles:
Training big ensembles of models can be parallelized, making it an ideal problem for distributed computing. Using Dropout, an efficient technique introduced by Hinton, allows for the training of a large ensemble while sharing weights among the models. Dropout involves randomly removing a subset of hidden units during each training iteration, leading to a diverse set of architectures. Weight sharing regularizes the models, enabling learning even with limited training examples. At test time, averaging the predictions from the individual models approximates the geometric mean of all possible ensemble configurations.

Benefits and Limitations of Dropout:
Dropout facilitates the training of large ensembles without the need for explicit coordination or communication between the models. The strong weight sharing in Dropout limits the expressive capacity of the ensemble.

Applicability to Multiple Hidden Layers:
The Dropout technique can be applied to neural networks with multiple hidden layers, although the mathematical justification becomes less precise. Empirically, Dropout still provides a good approximation of the results obtained by averaging over all possible ensemble configurations.

Experimental Results on MNIST:
Hinton presents experimental results on the MNIST dataset, demonstrating the effectiveness of Dropout in training a vanilla backpropagation network.

How to Improve a Neural Network’s Performance:
Use more layers: Instead of two layers of 800 pin units, the network could have more layers to improve its performance. Use dropout: Dropout is a technique that helps reduce overfitting by randomly dropping out some units during training. This can improve the generalization performance of the network. Constrain the weights: Limiting the L2 norm of the weight vector for each unit helps prevent some weights from becoming too large and overpowering the others. This forces the network to distribute the weights more evenly and can improve performance. Jitter the input: Applying random transformations to the input data that do not change the label (such as translations) can help the network learn more robust features. This is especially beneficial for non-convolutional networks.

Using Soft Targets for Transfer Learning:
Soft targets: Soft targets are predicted probabilities for each class, rather than hard targets, which are one-hot vectors indicating the correct class. Transfer learning from a larger network: A smaller network can be trained using soft targets from a larger, more powerful network. This can help the smaller network achieve better performance than it would if trained with hard targets.

Results:
Using soft targets from a larger network trained with dropout, the smaller network achieved 74 errors, which is a significant improvement over the 146 errors it got when trained with hard targets.

Conclusion:
Soft targets can be a valuable tool for improving the performance of neural networks, especially when used in conjunction with other techniques such as dropout, weight constraints, and input jittering.

Background Information:
Geoffrey Hinton introduced the concept of transfer learning with soft targets, a technique to improve the generalization of a neural network model by transferring knowledge from a large, pretrained model (the Big Net) to a smaller model (the Little Net).

Benefits of Transfer Learning with Soft Targets:
The Big Net learns a similarity metric, determining what concepts are similar to each other. Transferring this knowledge to the Little Net helps it generalize better, even though the Little Net is trained on the same data without transformations. The Little Net can perform well with a much smaller size and fewer resources compared to the Big Net.

Understanding Soft Targets:
Soft targets are probability distributions over the output classes for each training example. They provide more information than hard targets (one-hot labels) by indicating the similarity of the input to different classes. The Big Net learns these soft targets during training.

Generalization and Similarity Metrics:
The Big Net learns to generalize by understanding the similarities between different concepts. This knowledge is encoded in the soft targets and transferred to the Little Net. The Little Net benefits from this knowledge and generalizes better, even when it has never seen certain classes during training.

Example with MNIST Digit Recognition:
Hinton showed an experiment with MNIST digit recognition, where the Little Net was trained to classify handwritten digits. The soft targets provided information about the similarity of different digits. For example, some 2s were more similar to 3s and 8s, while others were more similar to 7s.

Transfer Learning without Access to Certain Classes:
Hinton conducted a surprising experiment where the Little Net was trained without ever seeing a specific class (3) during transfer learning. Despite not seeing any examples of 3, the Little Net was still able to correctly classify 3s, demonstrating the effectiveness of transfer learning with soft targets.

Conclusion:
Transfer learning with soft targets is a powerful technique that enables smaller models to achieve better generalization performance by leveraging knowledge learned from larger models. This approach provides a significant advantage in resource-constrained environments or when dealing with limited data.

A Unique Approach to Training:
By training a large model on the entire dataset with jitter and dropout, then transferring knowledge to a smaller model, surprising results can be achieved.

Unseen Digit Recognition:
Training a model to recognize digits by only showing it specific digits (e.g., 7s and 8s) leads to impressive results. After training, the model can recognize other unseen digits with reasonable accuracy, demonstrating the effectiveness of soft targets.

Advantages of Soft Targets:
Soft targets provide more informative data for training, allowing models to generalize better. Transforming targets is a cheaper and more efficient alternative to transforming inputs, as it requires fewer training examples.

Soft Targets in Practice:
Researchers have explored using hierarchical trees to generate soft targets for training classifiers, providing more information per training phase. However, hierarchical trees have limitations in capturing visual similarity, which is crucial for image recognition tasks.

Using Soft Targets as a Regularizer:
Instead of providing hard targets (0 or 1) for classification, provide soft targets. Soft targets are the outputs of a large, already-trained model at a higher temperature. Soft targets allow the new model to learn from the mistakes of the big model. This regularization technique prevents overfitting and improves performance.

Results with Soft Targets:
On an acoustic model task, training a new model with soft targets from an ensemble of 10 models achieved 60.8% accuracy. This is 6/7ths of the improvement achieved by running the ensemble. Using a single model with soft targets is more efficient and cost-effective than running an ensemble of models.

Prior Work by Rich Caruana:
Rich Caruana had a similar idea earlier, using the average logits from an ensemble for regression. Caruana’s method uses half the squared distance difference between the logits.

Relationship between Softmax and Logits:
As the temperature in the softmax function increases, the distribution gets softer and cross-entropy gradients become smaller. Scaling up the temperature and loading rate can match the logits.

Derivation of the Gradient for Logits:
The gradient of the cross-entropy cost function with soft targets is derived. The simplified form of the gradient is presented.

Mean-Subtracted Logits:
Geoffrey Hinton introduces the concept of subtracting the mean from logits in knowledge distillation. This results in zero-mean logits where the sum of all logits is zero.

Derivation of the Distillation Loss:
Hinton mathematically derives the distillation loss function, which is similar to a regression loss. He shows that in the limit as the temperature approaches infinity, the distillation loss becomes equivalent to the method proposed by Rich Caruana.

Optimal Temperature and Model Size:
Hinton discusses the optimal temperature to use in knowledge distillation. A higher temperature is better if the distilled model is large enough to capture all the knowledge from the large model. However, if the distilled model is small, a lower temperature is preferred to focus on capturing the most important knowledge.

Microsoft and Google’s Approaches:
Microsoft uses a temperature of 1 (cross-entropy loss) in knowledge distillation. Google explores a range of temperatures and has found that high temperatures generally work well.

Ensemble Distillation:
Hinton proposes an interesting experiment involving an ensemble of networks that learn from each other during training. Each network stores the logits produced by the other networks on the same mini-batch of data. This allows the networks to collectively learn and potentially achieve better performance.

Ensembling Models for Regularization:
Ensembling models during training can help regularize the learning process and improve model performance. Each model in the ensemble provides soft targets for the other models, in addition to the hard targets from the real data. This approach allows models to learn from each other’s predictions, leading to better generalization. In experiments on MNIST, ensembling models during training reduced the test error from 158 to 126 for individual models and from 143 to 120 for the ensemble.

Mining Data Efficiently with Specialist Models:
To address the computational cost of training large ensembles, a divide-and-conquer approach can be used. A generalist model is first trained on the entire dataset. Soft targets are generated from the generalist model and used for k-means clustering. The dataset is divided into subsets based on the clustering results, and specialist models are trained on each subset. By focusing on smaller subsets, specialist models can learn more efficiently and avoid overfitting.

Benefits of Specialist Models:
Specialist models can focus on specific subsets of the data, allowing them to learn more efficiently and accurately. By enriching the data with examples relevant to a specific specialist model, the training process can be accelerated. Overfitting can be mitigated by combining the predictions of multiple specialist models, resulting in a more robust and generalizable model.

Mixtures of Experts:
Mixtures of experts is a theoretical approach where multiple specialized models (experts) collaborate to make predictions. However, mixtures of experts tend to overfit, especially on large datasets.

Preventing Overfitting in Mixtures of Experts:
One method to prevent overfitting is to train a generalist model and then initialize specialist models based on the generalist model. The specialist models are then fine-tuned on specific tasks or classes. Early stopping is used to prevent overfitting during the training of specialist models.

Correcting Overweighting of Classes:
After training, specialist models may overemphasize the classes they were trained on due to the enriched training data. To correct this, the logit of the dustbin class (non-mushrooms in the example) is adjusted to compensate for the overfitting.

Experiments on the GFP Dataset:
Jeff Dean and Oriol applied this method to the GFP dataset, which contains 15,000 classes and 100 million images. They trained a large convolutional neural network for six months, achieving 25% correct top-1 accuracy.

Fine-tuning the Generalist Model:
Instead of training a new model from scratch, the goal is to improve the performance of the existing generalist model. A copy of the generalist model is created and fine-tuned on specific classes or tasks, resulting in better performance.

Using Multiple Specialist Networks:
Jeff, Dean, and Aurel trained 60 specialist networks using k-means clustering to create overlapping clusters. These specialist networks were used to refine predictions within specific classes.

Triage with the Generalist Model:
The generalist model is used to determine which specialist networks to apply to a test case. This approach reduces the computational cost by only running specialist networks when necessary.

Handling Errors:
The generalist model may sometimes make incorrect class predictions. In such cases, the specialist networks may not be able to provide accurate refinements.

Specialist Application Patterns:
Analysis of test cases revealed that in many cases, none or only a few specialist networks were applied. This indicates that the generalist model was effective in making correct class predictions.

Model Combination and Specialist Models:
Combining a generalist model with specialist models can significantly improve accuracy. Initializing copies of a generalist model and training them briefly on highly enriched data can quickly improve performance. Combining models with dustbin classes is challenging due to constraints on probability sums.

Combining Specialist Opinions:
Iteratively combining specialist opinions can be done by minimizing cross-entropy. This process resembles a version of the EM algorithm.

Training Specialists with Hard and Soft Targets:
Training specialists with hard targets on their specialist class and soft targets on other classes can prevent overfitting. This approach helps specialists focus on fine-tuning within their expertise while aligning with the generalist model’s knowledge.

Knowledge Transfer and Model Compression:
Once knowledge is extracted from data into a large model, it can be transferred to smaller models using soft targets. This approach allows for efficient training of small models on large datasets.

Future Directions:
Training a generalist model and many specialist models, regularized by fitting soft targets to the generalist model, may enable efficient learning on even larger datasets.

Conclusion:
Geoffrey Hinton emphasizes the importance of first extracting knowledge from data into a large model and then using soft targets to train smaller models. This approach can lead to significant improvements in accuracy and efficiency.