Definition of Experience in AI:
Sutton defines experience as the ordinary data flowing in and out of an intelligent agent.

This consists of two time-series data: sensations received from the world and actions generated by the agent.

Experience is not a special training period as seen in supervised learning, but is rather the ordinary interaction with the environment.

In reinforcement learning, experience is the agent’s only access to the world, and its primary method of learning.

Significance of Experience:
Experience, as data flowing in, initially has no meaning or significance; its value is determined by its relationship to other experiences.

One specific element of experience, the reward, carries inherent meaning. The agent seeks to maximize this reward.

The reward is considered as a component of sensation and is the primary motivator for the agent’s actions.

Internal vs External Understanding of the Mind:
Sutton poses a question: Will we ultimately understand the mind in terms of experience (internal, agent-centric elements like sensations, actions, rewards) or in terms of objective terms (external world aspects like objects, places, people, distances)?

The answer to this question could significantly influence the direction of AI research and development.

Increasing Role of Experience in AI:
Sutton outlines four stages in the increasing role of experience in AI over the past 70 years.

These stages include: 1) recognizing the agent’s experience, 2) defining goals in terms of experience (reward), 3) representing the state of the agent in terms of experience, and 4) acquiring knowledge in terms of experience.

At each stage, AI has gradually shifted from an external perspective to incorporating an experiential viewpoint.

This shift towards an experiential point of view is undertaken to make AI more grounded, learnable, and scalable.

Early AI and the Absence of Experience:
In the early days of AI (roughly the first 30 years), most AI systems were problem solvers or question answerers, without the concept of sensations and actions – hence, they lacked experience.

Problems were framed with a start state, goal state, and operators defined as state transitions, not actions. Solutions were sequences of operators, but were not actually executed, as they were assumed to be deterministic.

This lack of execution and confirmation of the plan meant that there was no real ‘experience’ in the sense Sutton previously defined.

Doing AI Without Experience:
Sutton highlights how early AI systems managed to operate without experience, by focusing on cognitive tasks that did not involve interaction with the environment.

As an example, he cites an AI textbook from 1980 by Nils Nilsson, which listed the achievements of AI systems at the time, including diagnosing diseases, planning the synthesis of complex organic compounds, and solving differential equations, among others. None of these tasks required the system to have sensations and actions.

Shift to Agenthood and Experience:
Over the last 30 years, AI research has increasingly focused on building agents that interact with their environment, marking a shift towards incorporating experience.

Sutton cites the 1995 edition of the standard AI textbook by Stuart Russell and Peter Norvig, which unifies AI under the concept of an intelligent agent, receiving perceptions from the environment and performing actions.

The shift to an agent-based perspective was controversial at the time, but it has now become the standard approach in modern AI, making experience a foundational part of the field.

Importance of Reward in AI:
Sutton highlights the increasing recognition of reward as a crucial element in AI systems, citing strong statements from his own reinforcement learning textbook and from a paper by David Silver and others.

These statements argue that all goals and purposes can be understood as the maximization of a cumulative scalar signal called reward. This idea is known as the “reward hypothesis.”

Initial Resistance to Reward-Based Goal Formulation:
Despite the growing acceptance of the reward hypothesis, Sutton acknowledges that many people still find it insufficient or even demeaning.

People tend to believe that human goals are more complex and cannot be reduced to the maximization of a scalar reward. They argue that human motivations encompass grander objectives such as contributing to knowledge or making the world a better place.

Historical Shift to Reward-Based Systems:
In the early days of AI, goals were formulated as world states to reach, not in terms of experience or reward. This perspective is still present in some contemporary AI textbooks.

However, with the rise of machine learning and reinforcement learning, the reward formulation has become more standard, even in traditionally symbolic AI systems like the SOAR cognitive architecture.

Reluctant Acceptance of Reward Formulation:
The AI community has gradually, and somewhat reluctantly, accepted the reward formulation of goals as an experiential way of understanding intelligent behavior.

Sutton emphasizes that while the first two steps in the evolution of AI (agenthood and reward) have largely already occurred, the next steps are still in progress.

Illustrating Experience:
Sutton presents an illustrative, made-up example to concretely explain the concept of ‘experience’ in AI.

He describes how, at each moment in time, an AI system receives sensory signals (represented as colored pixels) and a reward value, then takes an action.

Patterns in Experience:
The task of intelligence is to find patterns in the sensory data in order to predict and control future data.

For example, Sutton demonstrates how certain sequences of colors (sensory signals) appear to predict other colors in future signals, revealing patterns that an AI system might learn to exploit.

Complexity of Predictive Relationships:
Sutton emphasizes that while some signals are easy to predict from others, the notion of prediction is far more complex and subtle.

One familiar form of prediction in AI is the value function, which seeks to predict the sum of future rewards. The value function can be thought of as an ideal prediction that the agent approximates.

General Value Functions (GVFs):
The concept of value functions has been expanded into General Value Functions (GVFs), which can predict any signal and weight it with any temporal envelope.

However, not all predictions can be made efficiently (with low memory, computation, and communication), which makes computational complexity an important area of research.

The Role of Intelligence:
Sutton concludes that the main role of intelligence is to understand, predict, and control experience, with the ability to control rewards serving as the ultimate measure of intelligence.

External State in Classical AI:
Traditionally, AI systems are designed to think of state in terms of the external world.

Classic AI characterizes state objectively, using facts about the world, not about experience. The state is often defined in terms of external world state variables, such as whether it is raining or the status of objects in the Blocks World.

The traditional ontology in AI includes physical objects, abstract objects, and other external entities.

Experiential State as an Alternative:
The alternative to objective or external state is experiential state, which defines the state of the world entirely in terms of past experience. The state becomes a summary of past experience that is useful for predicting and controlling future experience.

In this model, there is no mention of the external world or entities; the focus is on what the agent can perceive, predict, and possibly influence.

Modern AI Approaches Towards Experiential State:
Many modern AI approaches, such as DQN (Deep Q-Network), compression approaches, predictive state representations, observation operator models, LSTMs (Long Short Term Memory), and generative testing for feature discovery, have embraced experiential state.

These approaches learn or discover their experiential state as a summary of past experience, which is not given to them a priori.

The Role of Experiential State:
Sutton presents a diagram where experiential state lies between the sensation and the action, summarizing everything about how the agent arrived at its current situation.

This state is then used to predict future events and to control future actions.

To maintain this state effectively, it should be constructed recursively from the most recent action, sensation, and previous state.

Perception as State Generation:
The process of generating experiential state can be considered as “perception”, as it integrates all the agent has observed and done into a sense of where it is.

This perception-based approach to state generation is scalable, grounded, and learnable, making it a promising direction for AI research.

Common Sense and Experiential Knowledge:
Knowledge is traditionally thought of as common sense information, such as who the president of the US is or that most birds have wings. However, there’s also knowledge that’s more experiential, like understanding that it’s a long walk to the city center or that it’s cold outside today.

The Gap between Common Sense and Experiential Knowledge:
There’s a significant gap between common sense knowledge and experiential knowledge. While experiential knowledge has clear semantics and can be learned and verified, it’s much different than knowing static facts about the world. Bridging this gap is a major challenge in AI.

Historical Perspective on Knowledge in AI:
Early AI systems lacked experiential knowledge since they didn’t have the ability to predict or learn from experiences. This shifted over time, with modern AI increasingly viewing knowledge as predictive models of the world. This shift offers clearer semantics for knowledge statements about the world.

Different Types of Knowledge in AI:
Sutton distinguishes between two primary types of empirical knowledge in AI: knowledge about the state (which was discussed earlier) and knowledge about dynamics (how the world changes). Dynamics knowledge can be further divided into state-to-state knowledge and state-to-experience knowledge, and each can be either objective or experiential.

Abstracting Knowledge:
State-to-state predictive models in AI can be abstracted to a higher level, such as with options in reinforcement learning, which are ways of behaving that can be learned and predicted. This abstract approach to knowledge helps bridge the gap between experience and common sense knowledge.

Experiential Focus:
The focus on experience is fundamental to AI knowledge. This is because agents can only gain information about the world through their sensors and can only affect the world through their actions. This experiential perspective enables AI to be grounded, learnable, and scalable, and allows the agent to improve as more computation becomes available.

The Standard Model of the Experiential Agent:
Sutton presents a model of the experiential agent which combines all four steps discussed. This includes perception (to generate state), policies (to generate action), value functions (for primary predictions), and transition models (for planning processes). This model places the experiential state at the center of the agent’s decision-making process.

The Challenge of Experience:
Experience is often avoided in AI development because it is unfamiliar, unintuitive, and complex. It is also low level and private, making it difficult to discuss and verify. Traditional human communication and knowledge sharing are based on external, public terms rather than experiential ones.

Experience in AI Development:
Despite the challenges, experience is vital for AI development because it comes freely from the normal operation of AI. It provides grounding for the notion of intelligence, being about finding patterns and understanding data. It also grounds knowledge, as it forms the basis for prediction. Any fact about the world is, in essence, a fact about experience, and thus can be learned or verified from experience.

Experience as the Ultimate Data:
The slogan “Data drives AI, and experience is the ultimate data” encapsulates the centrality of experience in AI development. Experience, seen as the ultimate data, provides the ground truth for AI learning and understanding, making it an indispensable part of the AI learning process.

Addressing Potential Questions:
Not everything is learned from experience; some knowledge may be built in. However, even built-in knowledge should be understood in terms of experience. The focus is not on what is learned from experience, but what is about experience.

The historical approach of building in abstractions first and adding links to experience later has not been successful. Sutton suggests that it might be more effective to build in experiential abstractions first.

While the abstraction gap between experience and knowledge is large, our computational power and human ingenuity are also considerable. Sutton encourages embracing this challenge and working towards crossing the gap.

Conscious and Unconscious Actions:
Sutton agrees that there are internal processes occurring within the mind, which can be considered conscious or unconscious cognitive actions. He emphasizes that the importance of experience does not dismiss the relevance of these internal processes. Instead, these internal actions can be thought of as data, and can be predicted and controlled, much like external experiences.

The Role of Sensory Input:
Sutton affirms that having more sensors on a robotic agent can enhance the model. More sensors lead to a broader and more comprehensive collection of experiential data, which can improve learning and decision-making processes in AI.

State, Action, and Sensation in AI Models:
Sutton differentiates between external state and internal state. The external state of an AI refers to the conditions in the external world, while the internal state refers to the agent’s perception of those conditions. This internal state, analogous to a belief state, is the primary focus in his discussion.

Experience and Latent Features:
Sutton states that the concept of latent features is an external, objective point of view. He suggests that latent features are hidden aspects of the external world, which are entangled in the data. This contrasts with his argument for focusing on experiential data, which he considers primary.

Experience and Counterfactual Thinking:
In response to a question about the role of causality and counterfactual thinking, Sutton acknowledges that these concepts are more oriented towards the external world. He reiterates the importance of experiential data and encourages listeners to consider both perspectives and the historical trends in AI development.

Objective Perspective on Trends:
Sutton’s goal is to provide a descriptive account of what is happening in the field of AI, highlighting the trends and counter-trends. He stresses that understanding the evolution of these trends is more important than advocating for a specific perspective. He acknowledges that people may have differing views on whether knowledge is experiential or external and encourages listeners to make their own assessments based on historical observations and trends.