Introduction of Turing Award Winners:
The Turing Award winners’ story exemplifies remarkable grit and perseverance. Neural networks were unpopular during their key contributions, but they are now fundamental to modern computer vision, natural language processing, and speech recognition. Their contributions inspire scientific exploration beyond popular trends.

Event Format:
Each winner will give a 30-minute lecture followed by a panel discussion. Online questions can be submitted according to the instructions on the screen.

Geoff Hinton’s Introduction:
Geoff Hinton’s fundamental contributions to AI include the discovery of backpropagation, AlexNet, Boltzmann machine, Capsule Network, and many other models and algorithms. A humorous anecdote about Geoff Hinton’s daughter’s reaction to his claims of understanding the brain.

Geoff Hinton’s Talk:
Hinton focuses on recent joint work with Adam Kosoriak and Sarah Saboy on UIT. Two approaches to object recognition: parts-based (hand-engineered, limited hierarchy) and convolutional neural nets (end-to-end learning, generalization across position). Convolutional neural nets are different from human perception and have limitations. The first part of Hinton’s talk addresses these problems and criticizes CNNs.

CNNs and Viewpoint Changes:
CNNs excel at handling translations, but struggle with rotations and scaling. Training CNNs on multiple viewpoints is inefficient. Ideal neural nets should effortlessly generalize to new viewpoints. Equivariance vs. Invariance: Equivariant representations change with viewpoint, while invariant representations remain constant. Hinton believes perceptual systems have equivariant representations of percepts, but invariant representations of labels.

CNNs and Image Parsing:
CNNs don’t explicitly parse images, but learn rich descriptions of pixel locations. CNNs recognize objects differently from humans, evidenced by their sensitivity to noise.

Activation by Scalar Product vs. Coincidence:
Scalar product activation is commonly used in CNNs, which is less effective than coincidence-based activation. Coincidence activation is significant in high dimensions, especially for filtering information. Transformers, unlike CNNs, use coincidence activation, making them better filters.

CNNs and Coordinate Frames:
CNNs don’t use coordinate frames, leading to different percepts based on imposed coordinate frames. Human perception heavily relies on coordinate frames, which CNNs fail to explain. Adversarial examples and CNNs’ distinct perception from humans may be linked to the lack of coordinate frames.

Inverse Computer Graphics Approach:
Proposes an inverse computer graphics approach to computer vision. Graphics programs use hierarchical models with matrices relating coordinate frames of wholes and parts.

Linear Structure of 3D Objects:
Geoff Hinton emphasizes the significance of leveraging the linear structure inherent in 3D objects. He highlights that 3D computer graphics successfully utilizes this structure to manipulate objects from various angles. Generalization and extrapolation are more effective for linear models compared to higher-order models.

Stacked Capsule Autoencoders:
Stacked Capsule Autoencoders are introduced as a new approach to 3D object recognition. The previous versions of capsules, including those involving discriminative learning and part-whole relationships, are considered incorrect. The current version focuses on unsupervised learning and whole-part relationships.

Capsules as Building Blocks:
Capsules are designed to capture more structure in neural networks. Each capsule represents an entity, its existence, a vector of properties, and its pose relative to the camera. They aim to capture intrinsic geometry effectively.

Relationships Between Capsules:
The pose of an object can predict the poses of its parts, regardless of viewpoint changes. This knowledge is essential for viewpoint-invariant recognition.

Autoencoder Architecture:
A greedy autoencoder approach is used to derive higher-level capsules from lower-level capsules. The decoder is explained, demonstrating how high-level capsules predict the poses of lower-level capsules.

Generative Model:
The generative model predicts low-level data based on high-level capsules. A mixture model is employed to find the best explanation for the observed poses of lower-level capsules. Backpropagation is used to train the model.

Key Features of the Generative Model:
It assumes that each lower-level capsule is explained by exactly one higher-level capsule (parse tree structure). The pose of a lower-level capsule is derived from the pose of its explaining higher-level capsule. The model inherently incorporates viewpoint invariance and parse tree derivation.

Encoder and Perception:
The encoder, representing perception, is not discussed in detail due to time constraints.

Transformers for Difficult Inference Problems:
Traditional capsule networks faced challenges in engineering the encoder to make votes for the poses of high-level capsules. The introduction of transformers, initially used for language, provided a solution to this problem.

Multilayer Transformer as a Complicated Encoding Model:
A multilayer transformer is employed as the encoding model to handle the complex inference problem. The transformer takes all of the extracted capsules and revises their vector descriptions as it moves up through its layers. The final layer of the transformer converts these revised descriptions into predictions for the entire object.

Training the Set Transformer:
The set transformer is trained using derivatives obtained from a generative model. The objective of training aligns with that of the generative model, maximizing the log probability of observed poses given the existence and poses of high-level capsules. A sparseness prior is introduced to encourage the activation of only a few high-level capsules.

Understanding Transformers:
Transformers process sentences by applying a convolutional neural network to a sequence of word vectors. This enables the reconstruction of word vectors if some are left out during unsupervised training. Transformers introduce queries, keys, and values to refine the representations of word vectors. Queries compare with keys to identify similar neighboring word vectors and combine their values to generate a new representation.

Application to MNIST Digits:
The set transformer and generative model are applied to MNIST digits, a simple dataset from the 1980s.

Introduction to Capsule Networks for MNIST Digit Recognition:
Capsule networks aim to model MNIST digits using a layer of parts (small stroke templates) and a layer of holes (high-level capsules) that learn to extract meaningful features from the input images.

Learning Parts and Holes:
Parts are 11×11 templates learned to capture specific features within the digits. Holes are learned to explain combinations of parts and model high-level concepts, such as whole digits.

Reconstruction and Visualization:
The network reconstructs the original image from the extracted parts and holes, allowing for visualization of the learned representations. The red color represents the reconstruction from parts, and the green color represents the reconstruction from high-level capsules. Yellow indicates perfect reconstruction, while red and green fringes show slight differences.

Activation of High-Level Capsules:
The network learns 24 high-level capsules that activate based on the extracted parts. These capsules represent complex features, not necessarily corresponding exactly to digits.

Part Transformations and Compositionality:
Parts can be affinely transformed to instantiate themselves differently within the image. This allows the same part to be used in various ways to compose different digits.

Unsupervised Learning and Natural Class Separation:
The network learns without any labels, demonstrating unsupervised learning capabilities. When embedding the high-level capsule activations using t-SNE, the network naturally separates the 10 MNIST classes without any prior knowledge.

Classification Accuracy:
With minimal or no labels, the network achieves 98.7% classification accuracy on the MNIST dataset.

Challenges and Future Directions:
Despite its success, the approach has limitations, including scalability to larger and more complex datasets. Future research directions involve addressing these challenges and exploring applications in other domains.

Vision as Sampling Process:
Vision involves a tiny fovea that decides where to focus, resulting in a sampling process where not everything is seen at high resolution.

Figure and Ground:
During each fixation, vision perceives one figure with ground. Illusions like the vase-faces example demonstrate the psychological distinction between figure and ground. Capsule models focus on modeling the perception of the figure, not the ground.

Variational Autoencoders for Ground:
To model the ground, a texture modeling approach is more suitable than a parts-based model. Variational autoencoders excel at modeling the textured background.

Combining Capsule and Variational Autoencoders:
Sarah trained a combined stack capsule autoencoder and variational autoencoder to explain MNIST digits with textured backgrounds. This approach outperforms models that do not consider the textured background.

Background Clutter:
To effectively deal with background clutter, it should be treated as such, without detailed modeling using a high-level parts-based model.

Extending to 3D Images:
Sarah previously applied a version of Capsule’s work to the NORB images, designed to test 3D shape recognition without color histograms.

Introduction to Inverse Graphics:
To make capsule networks effective, it is crucial for primary capsules to represent meaningful object parts. Vision can be viewed as inverse graphics, where shapes are broken down into smaller parts until they resemble basic elements like triangles. Inverse graphics involves inverting the rendering process to extract sensible parts from an image.

Levels of Inverse Graphics:
The bottom level of inverse graphics deals with light properties like albedo. Higher levels are concerned with geometry and the spatial arrangement of objects.

Approaches to Extracting Sensible Parts:
Surface meshes can be used to represent object parts. Known parts, also known as geons, can be extracted using intersections of half spaces. Various other approaches exist for extracting sensible parts.

Conclusion:
Prior knowledge about coordinate transforms and parse trees can be easily incorporated into a generative model.

Complexity of Generative Models:
The complexity of a generative model is more significant than the complexity of the recognition model in terms of model selection criteria like minimum description length and Bayesian inference.

Simplifying the Generative Model:
It is advantageous to create a simple generative model with extensive wired instructions. The challenging task of inverting the generative model can be delegated to a large transformer network.

Success with Large Transformer Networks:
With sufficient layers, size, and training data, large transformer networks can achieve success in various tasks.

Introduction to Ian Le Guin:
Ian Le Guin is a renowned professor at NYU, specializing in various fields including computer science, data science, neural science, and electrical and computer engineering. He has made significant contributions to machine learning, computer vision, mobile robotics, and computational neuroscience. His collaborative efforts with others led to the establishment of the Partnership on AI, aiming to advance AI for beneficial purposes.

Ian Le Guin’s Personal Traits:
Known for his positive attitude and passion for research. Enjoys his work and communicates it with enthusiasm and a smile. Has a love for life, sailing, good food, especially French cuisine.

Ian Le Guin’s Status in the AI Community:
Considered one of the “godfathers of AI” along with Jeff and Yoshua. Recognized for his significant contributions to the field.

Introduction to Self-Supervision:
Ian Le Guin shifts the focus to self-supervision, presenting a higher-level and inspirational perspective rather than a technical one. He defines deep learning as building a system by assembling parameterized modules.

Definition of Deep Learning:
Deep learning is a branch of machine learning that involves optimizing computation graphs through gradient-based learning. It allows engineers to create architectures that incorporate prior knowledge and inductive bias. Unlike traditional neural networks, deep learning systems can involve complex computations, such as minimizing energy functions, for inference. It is applicable to various learning paradigms, including supervised, reinforcement, self-supervised, and unsupervised learning.

Supervised Learning with Deep Learning:
Supervised learning with deep learning has been highly successful, especially with large datasets. Applications include speech recognition, image recognition, natural language processing, translation, and computer vision. The availability of open-source code has facilitated the adoption of deep learning for various computer vision tasks.

Unexpected Applications of Deep Learning:
Recent research has demonstrated the ability of deep learning to perform symbolic manipulation, such as solving integrals and differential equations. This is achieved through supervised learning with generated data and large neural networks, without explicit symbol manipulation or logical rules.

Pervasiveness of Deep Learning:
Deep learning has widespread applications in various industries, including automotive, medical imaging, and social media. Automatic emergency braking systems, powered by deep learning, have significantly reduced collisions and saved lives. Deep learning is also used for hate speech filtering, preventing violence, stopping weapon sales, and combating terrorist propaganda on social networks.

Deep Learning’s Applications:
Deep learning has become integral to major tech companies like Facebook, Instagram, Google, and YouTube. It powers various applications, including image recognition, natural language processing, and speech recognition.

Limitations of Reinforcement Learning:
Reinforcement learning excels in games and simulations but is slow and requires extensive training. Training a Go player at Facebook took two weeks on 2,000 GPUs and 20 million games, surpassing a human lifetime’s worth of gameplay. DeepMind’s AlphaStar in StarCraft required 200 years of real-time play to achieve slightly above human performance on a single map. OpenAI’s Rubik’s Cube manipulation system took 10,000 years of simulated training.

Challenges of Deep Learning:
Learning with fewer labels, samples, or trials. Learning to reason and making reasoning compatible with gradient-based learning. Learning to plan complex action sequences and decompose tasks into subtasks.

How Humans Learn:
Humans learn quickly and efficiently through unsupervised or self-supervised learning. Babies learn about gravity and object permanence around eight to nine months. They learn about animate and inanimate objects, categories, and physical concepts like inertia and conservation of momentum. Animals like orangutans also exhibit intelligence without language or linguistic communication.

The Essence of Self-Supervised Learning:
Self-supervised learning involves training a system to fill in missing parts of an input, such as predicting the future in a video or missing words in text.

Transformers and BERT’s Success in Text Prediction:
Transformers and BERT-like systems excel at predicting missing words in text. The uncertainty of missing words can be represented as a vector of probabilities over the dictionary, making training straightforward.

Challenge of Uncertainty Representation in Images and Videos:
Unlike text, representing uncertainty in images and videos is more complex due to the continuous nature of these modalities. Predicting a single future frame results in blurry predictions as the system averages all possible futures.

Energy-Based Models for Resolving Blurry Predictions:
Latent variable energy-based models are proposed to address the blurriness issue. An energy function measures the compatibility between an observation (e.g., initial video segment) and a predicted future. By varying a latent variable, a manifold of plausible predictions is generated.

Ongoing Research and Progress:
There is ongoing research on energy-based models for prediction. Various training methods, including adversarial methods, show promise in reducing blurriness.

Key Argument:
Jeff Dean argues that self-supervised learning provides more information for machines to learn compared to supervised or reinforcement learning.

Self-Supervised Learning vs. Supervised and Reinforcement Learning:
In reinforcement learning, machines receive limited feedback in the form of a single scalar. In supervised learning, machines receive information such as categories or labels for each sample. Self-supervised learning involves training machines to predict entire video frames or videos, providing more information.

Challenges of Self-Supervised Learning:
Self-supervised learning data is unreliable due to uncertainty and multiple possible futures.

Analogy:
Self-supervised learning is the bulk of the cake (genoise), supervised learning is the icing, and reinforcement learning is the cherry. All components are essential for a complete cake, just like all three learning types are important for AI.

The Next Revolution in AI:
The next revolution in AI will involve a combination of learning methods, not just supervised or reinforcement learning alone.

Energy-Based Models:
Energy-based models are a class of machine learning models used for self-supervised learning. There are two main categories of methods for training energy-based models: contrastive and architectural. Contrastive methods train models to give low energy to observed samples and high energy to unobserved samples.

Introduction of Contrastive Learning:
Contrastive learning methods are used in pre-training models to learn representations that are useful for downstream tasks. One example is denoising autoencoders, where a corrupted sample is given to a network, and the network is trained to recover the uncorrupted version. This can be seen as a form of energy-based training, where the energy of the corrupted point is forced to be higher than the energy of the uncorrupted point.

Contrastive Embedding for Image Recognition:
Contrastive embedding is another successful contrastive learning method for images. It involves using Siamese networks to compare two images, one corrupted and one uncorrupted, and training the networks to output similar representations for semantically identical images. Recent papers have shown record-breaking performance on computer vision tasks using this method as a pre-training phase.

Energy-Based Models with Latent Variables:
In energy-based models with latent variables, inference involves finding the value of the latent variable that minimizes the overall energy output by the system. This can be seen as a form of optimization-based inference, which is used in many probabilistic models and structural prediction learning.

Learning a World Model for Prediction in Stochastic Environments:
A simple example of how to do prediction in a stochastic environment is to train a forward model that predicts the behavior of other agents in the environment. This forward model can be used to plan ahead and take actions that avoid collisions or other undesirable outcomes. The forward model can take the current state of the world, the agent’s action, and a random latent variable as input and output the state of the world at some future time.

Self-Supervised Learning for AI:
Self-supervised learning allows AI systems to learn background knowledge about the world through observation, potentially leading to the emergence of common sense. Massive networks can be trained with self-supervised learning due to the availability of abundant unlabeled data. Forward models of the world can be learned for control using self-supervised techniques.

Challenges in Self-Supervised Learning:
Handling uncertainty in self-supervised learning is a significant challenge. Reasoning through vector representation and energy minimization requires further research and development. Replacing symbols by vectors and logic by continuous functions to make them compatible with gradient-based learning is a complex task. Learning hierarchical representations of action plans remains a significant challenge.

Yoshua Bengio’s Contributions and Vision:
Yoshua Bengio has made significant contributions to AI, including higher-dimensional word embeddings, attention mechanisms, and generative adversarial networks. He continues to tackle grand challenges such as the nature of consciousness and environmental sustainability. Bengio is committed to reducing the carbon footprint of conferences through remote attendance.

Inductive Biases in Machine Learning:
The no-freedom theorem implies that there’s no completely general machine learning or AI. Priors are necessary for machine learning, but the amount of prior knowledge to incorporate is a critical question. Evolution has provided animals with specific priors, and some of these priors in human brains are very general, allowing for a wide range of tasks. General priors require fewer bits to specify, making them easier for evolution to discover.

Current Deep Learning and System 1 Processing:
Current deep learning models, such as ConvNets, incorporate priors that exploit invariance to translations and deformations in images. Deep learning is effective for intuitive or unconscious tasks, habitual behaviors that require practice, and System 1 processing in general.

System 2 Processing and Consciousness:
System 2 processing involves consciousness, takes more time to compute, and is sequential, with limited elements examined at each step. This type of processing is associated with reasoning, planning, and dealing with novel situations. Humans can recombine known elements in novel ways, allowing them to adapt to unfamiliar circumstances.

Motivation for Combining System 1 and System 2 in Deep Learning:
Out-of-distribution generalization: Deep learning models can handle novel situations beyond their training experience. High-level representations: The goal is to learn high-level representations that capture concepts and causality, enabling recombination on the fly. Generalization across changing environments: Agents need to understand how the world works to adapt to changes caused by other agents or unforeseen events.

Priors and System 2 Processing:
The consciousness prior suggests that high-level semantic variables manipulated consciously have a sparse graphical model distribution with dependencies between variables. These variables often refer to causality, agents, intentions, and consequences. The sparse factor graph represents rules that can be instantiated on various variable tuples, enabling efficient manipulation of objects.

Out-of-Distribution Generalization:
Evolution may have equipped humans with special processing systems to handle changes in distribution in complex worlds. Agents must deal with non-stationarities due to other agents’ actions or their perception of only part of the world. Systematic generalization, or dynamic recombination of concepts, is a crucial strategy for addressing these changes.

Systematicity and Recombination of Concepts:
Systematicity allows agents to recombine known concepts to explain new situations in the world. This dynamic recombination helps deal with changes in distribution, enabling understanding of novel scenarios like science fiction stories. Current machine learning models often struggle with changes in distribution, unlike humans.

Combining System 1 and System 2 Approaches:
The ideal AI system would combine the strengths of both System 1 and System 2 processing. Avoiding pitfalls of classical AI, such as rule-based symbol manipulation, while incorporating learning and efficiency. Grounding high-level concepts in the real world and using System 1 for efficient search and planning. Retaining distributed representations, learned attribute vectors, and uncertainty handling from deep learning. Integrating advantages of rules, facts, and reasoning for systematic generalization.

High-Level Insights from the Transcript:
Deep learning models can be improved by incorporating attention mechanisms, which allow for dynamic connections and the exchange of information between modules. Attention mechanisms are crucial for conscious processing and enable focusing on a few elements while considering the broader context. Transformers utilize soft attention mechanisms to match keys and queries, facilitating communication between modules and changing neural nets from vector-based to set-based operations. Attention mechanisms introduce the need for representing names of things indirectly, allowing dynamic connections and the exchange of information between modules. The consciousness prior assumes that knowledge is represented in a sparse graph, where dependencies involve a limited number of entities, and changes in this graph occur in a sparse manner. Sparse changes in the knowledge graph are often caused by physical interactions, and having the right abstractions can help models adapt quickly to these changes. The right causal model can adapt faster to distribution changes caused by interventions, requiring fewer examples compared to incorrect models. Recurrent independent mechanisms (RIMS) architecture features a modular structure with a bottleneck for communication between modules, enabling dynamic recombination of pieces and facilitating coherent interpretations.

Additional Details and Points of Interest:
Attention mechanisms allow for focusing on a few elements while considering the broader context, similar to conscious processing. Transformers use soft attention mechanisms to match keys and queries, sending value vectors from the bottom module to the upper module when there’s a good match. Attention mechanisms change neural nets from operating on vectors to operating on sets of vectors, providing more flexibility and generalization. The consciousness prior suggests that knowledge is represented in a sparse graph, where dependencies involve a limited number of entities, and changes in this graph occur in a sparse manner. Sparse changes in the knowledge graph are often caused by physical interactions, and having the right abstractions can help models adapt quickly to these changes. RIMS architecture has a modular structure with a bottleneck for communication between modules, allowing for dynamic recombination of pieces and facilitating coherent interpretations.

Module-to-Module Communication:
Modules in this approach communicate through typed arguments, similar to functions in programming. Each module specifies the type of object it expects as its first argument using a query vector. Modules with matching key vectors and query vectors are connected, allowing the source module to send values to the destination module.

Soft Attention:
The matching process is not discrete but rather done softly using soft attention. This allows for a more nuanced and flexible matching process.

Out-of-Distribution Performance:
The approach performs well out-of-distribution, meaning it can handle changes in data distribution, such as longer sequences. This is a significant advantage over other LSTM-based approaches.

Ongoing Work:
The author and their collaborators are exploring various extensions to the approach, but there is not enough time to discuss them in the presentation.

Conclusion:
The author has presented several hypotheses about incorporating system 2 abilities into machine learning systems. These hypotheses can lead to the development of more powerful and flexible machine learning models.

Connection Between Neural Networks and Natural Computation:
Geoff Hinton: Neural computation inspires engineering models, suggesting ways to train systems with millions of parameters from scratch. Yann LeCun: Convolutional nets are inspired by neuroscience. Yoshua Bengio: Simple principles in machine learning can help explain brain functions.

Representation and Reasoning:
Panellists mentioned aspects of representation and reasoning, such as compositionality and latent representations. Some disparaged symbolic AI, but Hinton suggested forgetting the past and exploring gradient descent in large systems.

Replacing Symbols and Logic with Vectors and Continuous Functions:
Panellists discussed the need to make reasoning compatible with learning, which requires gradient-based learning and differentiable functions. Symbolic AI and linguistic knowledge are less useful in modern approaches like transformers.

Alternatives to Gradient-Based Learning:
Panellists questioned whether there are viable alternatives to gradient-based learning, considering that most successful learning methods involve optimization. The question of whether the brain minimizes an objective function remains unclear.

The Role of Universities in AI Research:
Universities remain crucial for original ideas and groundbreaking research in AI, despite the resources available to large companies. The unique environment of universities fosters creative thinking and long-term research projects.

Importance of Toy Problems:
Focusing on extremely complex benchmarks with limited resources may hinder progress. Studying problems on a smaller scale, using toy problems, can provide valuable insights and allow for more comprehensive experimentation. A conference dedicated to deep learning on toy problems is proposed to encourage this approach.

Diversity in Reading and Perspectives:
It is essential to avoid monoculture in AI research and encourage diverse perspectives. Students should explore a variety of sources, including classic works and emerging ideas, to foster originality and creativity. Reading literature can be valuable after forming one’s own ideas and hypotheses.

Structural Biases and Mechanisms in AI Architectures:
The development of various structural biases and mechanisms in AI architectures raises questions about their limitations and sufficiency. The ideal number of such mechanisms for achieving human-level AI is uncertain, ranging from a small set to a larger variety.

Environmental and Ethical Concerns in AI Research:
The carbon footprint of large data centers and AI research is a growing concern. Ethical considerations regarding the use of AI in industries like fossil fuels and military applications are raised. Companies with AI expertise should balance profit-driven research with addressing societal issues and ethical concerns.

Intuition and Problem Identification in Research:
Researchers often rely on intuition and system one thinking when generating new ideas. Identifying crucial and important problems drives the development of innovative solutions. Ideas often become evident after solving complex problems, though they may take time to gain recognition.

Long-Term Impact and Practical Applications:
Researchers emphasize the importance of pursuing long-term impactful problems rather than solely focusing on improving practical systems. Self-supervised learning and dealing with uncertainty in prediction are highlighted as key areas for future research.

Persistence in Unpopular Ideas:
Despite the unpopularity of neural networks in the past, some researchers continued working on them, highlighting the value of persistence. The challenge lies in distinguishing between genuinely good ideas and those that are unpopular due to their lack of merit.

Combining Intuition and Evidence:
While intuition is important, researchers should also consider evidence to assess the validity of their ideas. A fine balance is needed between considering evidence and relying solely on faith in an idea.

Long-Term Commitment to Ideas:
Hinton emphasizes the importance of never giving up on an idea that one truly believes in. He continues to pursue Boltzmann machines despite challenges, driven by a logical belief in the idea.

Short-Term Focus in Research:
The current publication cycle is perceived as fast and short-sighted, encouraging researchers to focus on short-term gains rather than long-term exploration. This trend is seen as detrimental to the field, stifling innovation and risk-taking.

Structural Changes to Encourage Long-Term Research:
The need for structural changes to encourage researchers to take more risks and work on longer-term horizons is recognized. Conferences and venues should provide space for methods-oriented research that emphasizes complex questions rather than record-breaking results.

Publication Pressure and PhD Research:
Publication pressure has intensified compared to earlier times, leading to a high rate of paper production by PhD students. The concern is whether the quality of research has been compromised in the pursuit of quantity.

Geoff Hinton’s Model of AI Research:
Geoff Hinton has a model of the AI research process, where researchers work on an idea for a short time, make some progress, and publish a paper. This can be seen as someone filling in a few of the easy Sudoku puzzles in a book of hard Sudoku puzzles, which messes it up for others.

AI as Science:
AI can be considered science, as it involves both engineering and understanding. There is a creative aspect of conceiving an AI artifact and a scientific aspect of analyzing how it works and why it doesn’t. The creation of AI artifacts often precedes the theory that explains them, similar to the invention of the steam engine before the development of thermodynamics.

Measuring General Intelligence:
A paper by Cholette from Google discusses ways to measure general intelligence and distinguish it from what it is not.

Computational Complexity of Deep Learning:
Deep learning models are incredibly powerful, but they require a lot of computational resources (megawatts or kilowatts). It is unclear if neural architectures can achieve general intelligence without this computational complexity.

Priors, Self-Supervised Learning, and Unsupervised Learning:
The panelists agree on the importance of priors, self-supervised learning, and unsupervised learning for AI research.

Disagreements Among the Panelists:
Geoff Hinton and Yoshua Bengio disagree on whether Bengio’s email address should have a country code after “Quebec.”