Animal Life in the Primordial Soup:
540 million years ago, animals lived in a simple world with few species.

The Cambrian Explosion:
A sudden increase in animal species occurred over a short geological period known as the Cambrian explosion. This event resulted in a diverse range of animals with different shapes and behaviors.

Theories of the Cambrian Explosion:
Various theories attempt to explain the Cambrian explosion, including climate change and chemical changes in the water.

Andrew Parker’s Theory:
One prominent theory suggests that the sudden evolution of vision triggered the Cambrian explosion.

Evolutionary Arms Race:
The ability to see and sense the world led to an evolutionary arms race, where animals either evolved or faced extinction.

Vision as a Primary Sensing Apparatus:
Most animals today rely on vision as one of their primary sensing mechanisms.

Visual Intelligence in Humans:
Humans use vision for survival, navigation, manipulation, interaction, and understanding the world. A significant portion of the human neocortex is dedicated to visual processing.

Understanding Visual Intelligence:
Studying visual intelligence is crucial for comprehending intelligence in general.

Powerful Human Vision Intelligence:
Humans have incredible visual intelligence, enabling them to understand and interact with the world.

Rapid Scene Understanding:
Humans can rapidly recognize objects in a scene, even in scrambled or unfamiliar images. Humans have the ability to see a holistic scene before focusing on specific objects.

Rapid Detection of Novel Objects:
Humans can detect previously unseen objects quickly and accurately. Demonstrated in studies using rapidly presented images.

Fast Visual Processing:
Studies using brain EEG show that humans can categorize complex objects, like animals, in 150 milliseconds.

Recognition in Brief Glimpses:
Humans can recognize objects and understand scenes even with brief exposure. Recognition possible with presentation times as short as 67 or 100 milliseconds.

Seeing is Understanding:
Humans comprehend the world by seeing more than just grayscale values or colors. Making sense of visual content is crucial for understanding the world.

Object Recognition: A Fundamental Understanding Task

Recognizing Objects in Images:
Object recognition is a fundamental ability for humans and computers to understand visual scenes. It involves labeling images with their main objects.

Challenges of Object Recognition:
Objects come in various shapes, forms, lighting, articulation, occlusion, and clutter. Mathematically, there are infinite ways of rendering a single object.

Early Models:
Early computer scientists designed hand-crafted models for object recognition. These models consisted of geometric shapes like geons and generalized cylinders. They had limited success due to conceptual limitations.

Hand-Designed Features and Statistical Machine Learning:
Around the turn of the century, the focus shifted to hand-designed features. Computer scientists used statistical machine learning to learn models from these features. Examples include textured patches of images. Datasets like Pascal VOC and Caltech 101 were introduced to test recognition capabilities.

Ecological Approach to Perception:
Fei-Fei Li was inspired by psychologist J.J. Gibson’s ecological approach to perception. Gibson emphasized the importance of considering the environment and purpose when studying representation and intelligence. This perspective led to a focus on recognizing and understanding the full scale of the world, not just a few dozen objects.

ImageNet Project and the Convolutional Neural Network Breakthrough:
Fei-Fei Li and her team created ImageNet, a dataset with 15 million images across 22,000 categories. The ImageNet Challenge pushed the field towards large-scale object recognition. In 2012, Geoffrey Hinton and his students achieved a breakthrough using a convolutional neural network on ImageNet. This led to a significant reduction in object recognition errors.

Recent Advances:
Since 2012, the field of object recognition has seen rapid progress. Winning algorithms in image data challenges consistently produce remarkable results. Improvements have been made in algorithm efficiency and the size of datasets used.

Scene Graphs:
Images contain more than just objects; relationships between objects are crucial for understanding the story of a scene. Scene graphs represent objects, their attributes, and relationships, providing a deeper understanding of the visual scene. The Visual Genome dataset contains 100,000 images with millions of objects, relationships, attributes, and descriptions.

Visual Relationship Estimation:
Algorithms can recognize relationships between objects, even novel compositions with zero training examples. Decomposable representations allow algorithms to learn new relationships by combining known ones.

Action Recognition:
Scene graphs can be extended to represent spatial-temporal relationships in videos. The Action Genome dataset contains 10,000 videos and 157 activities, enabling state-of-the-art action recognition.

Image Captioning:
Algorithms can generate sentences and paragraphs to describe images, capturing different parts of the scene.

Perception and Action:
Vision is not just about understanding the world; it’s also about participating in it. The relationship between perception and behavior is active and dynamic in the animal world.

Plato’s Allegory of the Cave:
Plato’s Allegory of the Cave illustrates the limitations of our visual ability. We are prisoners forced to interpret the world through a 2D projection on our retina.

The Other Minds:
The nervous system’s fundamental function is to link perception with action.

Evolution of Perception and Action:
Fei-Fei Li highlights the connection between the evolution of vision and the onset of active animal life. The development of vision allowed animals to perceive their surroundings and interact with them. This led to a long trajectory of more and more powerful brains.

The Kitten Experiment:
An experiment conducted in the 1960s involved two newborn kittens yoked together in an identical visual world. The active kitten was allowed to move its body and drive the yoke, while the passive kitten was only driven by the active kitten and had limited movements. After a few weeks, the active kitten developed a normal perception system, while the passive kitten’s perception was damaged. However, if the passive kitten later experienced the real world normally, it could regain its perceptual ability.

Mirror Neurons:
Neurons called mirror neurons fire in the mammalian brain not only when performing actions but also when observing others performing those actions. This link between perception and action demonstrates the brain’s capacity to understand and respond to activities in the environment.

The Importance of Perception for Action:
Perception is not a passive process; its purpose is not only to understand and label things but also to enable action in the world. Seeing is for doing in the real world.

Embodied Intelligence Systems:
Fei-Fei Li expresses excitement about exploring computer learning systems or embodied intelligence systems that can perform actions and interact with the world through sophisticated perception and understanding.

Curiosity-Driven Learning:
Curiosity-driven learning, also known as learning to play or explorative learning, involves training a dynamic world model in a self-supervised manner. The goal is to enable a learning system to form a world model and a self-model to learn to perceive the world. Infants explore without a task in mind, and this approach is inspired by human behavior. Different flavors of curiosity-driven learning exist, including novelty-based, goal-based, and world model-based approaches.

World Model and Self-Model:
The world model network predicts the consequences of actions, while the self-model network predicts the error of the world model. Action choices are made by the self-model, which is adversarial to the world model. This curious, intrinsic motivation drives action choices and leads to the emergence of structured behavior.

Exploration and Learning:
In the early stages of learning, the algorithm learns its own ego motion and then starts to pay attention to objects and their interactions. In a recognition task, the curiosity-driven algorithm outperforms random policies.

Task-Driven Learning:
Most robotic learning today is based on skill-level tasks and short-horizon tasks. The goal is to push for longer-horizon tasks, such as organizing a desktop by putting together forks and bowls.

Neural Task Programming:
Neural Task Programming proposes a divide-and-conquer algorithm to modularize learning and compose short-term skills for long-horizon tasks. Superior performance compared to state-of-the-art in handling complex tasks like object sorting and adversarial attacks.

Curriculum Learning with Task Generation:
Adaptive Procedural Task Generation (APGEN) generates tasks from simple to complex for curriculum learning. It utilizes a task generator and discriminator network to guide the generation process. Tested in manipulation tasks, APGEN shows advantages over other state-of-the-art methods.

Limitations of Current Robotic Research:
Most robotic research focuses on skill-level tasks and short-horizon goals in artificially simple environments. Tasks are often picked by experimenters without profound reasons, lacking standard metrics and generalizability. Real-world systems face challenges due to cluttered environments and the complexity of household tasks.

Ecological Approach to Robotic Learning:
Inspired by J.J. Gibson’s quote, Fei-Fei Li suggests an ecological approach to robotic learning. Creating robotic learning environments that mimic the real world, particularly household environments, is crucial. Household environments offer ecological complexity, dynamics, uncertainties, large viabilities, interactivity, social aspects, and inherent multi-tasking.

Large-Scale and Diverse Benchmark:
Behavior benchmark consists of 1,000 complex tasks or activities, scanned from 50 large-scale real-world scenes. It includes 1,000+ object categories, 3,000+ object models, and a large number of objects per activity. The simulation offers photorealistic graphics, kinematic and dynamic extended states, flexible materials, realistic fluids, and thermal effects.

Task Selection:
The benchmark’s activities were curated based on human needs and preferences, considering tasks like cleaning, taking care of the elderly, and daily household chores. A user study revealed that cleaning is a highly desired task for robotic assistance, while tasks like playing sports or opening gifts are less preferred. The top 1,000 activities were chosen based on human preferences for robotic help.

BDDL Language:
BDDL (Behavior Description Language) is a logical form used to describe the initial state and goal state of each task. It combines human annotation and GPT-3’s assistance to create a database for each task.

Dataset Features:
The benchmark emphasizes the diversity of objects, including thousands of object models. It provides annotations for object poses, affordances, relationships, and attributes. The environment includes various lighting conditions, textures, and dynamic effects.

Simulation:
Simulation environments are crucial for training embodied AI agents safely and efficiently. These environments can help overcome real-world biases and enable reproducible benchmarks.

Properties of Objects and Environments:
Objects in the simulation have various properties such as cookable, sliceable, freezable, burnable, deformable, and more. Environments range from houses to offices, providing diverse scenarios for tasks.

OmniGibson:
OmniGibson is a realistic simulation environment developed by combining efforts with NVIDIA’s Omniverse. It simulates thermal effects, transition machines, lights and reflections, fluids, deformability, and transparency. User studies indicate that OmniGibson achieves high realism in rendering compared to other simulation environments.

Evaluation of Embodied AI Algorithms:
The Behavior1K dataset is used to evaluate embodied AI algorithms on a diverse set of tasks. Current state-of-the-art algorithms, such as Soft Actor Critic and PPO, are evaluated on this benchmark.

Algorithm Performance and Limitations:
State-of-the-art algorithms perform well on short-scale tasks with simplified magic actions but struggle without them. Increased object pose and appearance diversity further decreases algorithm performance.

Long Horizon Tasks:
RL algorithms face difficulties in performing long horizon tasks like decorating a store or cleaning a table. Assumptions about action primitives and object states are necessary for reasonable algorithm performance.

SIM-to-Real World Transfer:
A digital twin of a real-world apartment was created for sim-to-real algorithms experimentation. This environment allows researchers to challenge robots in real-world settings.

Behavior Benchmark:
Behavior.stanford.edu provides access to the behavior benchmark environment and more information about the research.

Vision and Embodied Active Intelligence:
Vision is a crucial aspect of intelligence, especially in embodied active intelligence. The lab’s work focuses on a real-world ecological approach to perception and robotic learning.