Welcome:
Diane Beck, interim head of the Department of Psychology and a member of the Beckman Institute Executive Committee, welcomed attendees to the annual Beckman Brown Lecture on Interdisciplinary Science.

Lecture Series Background:
The lecture honors Arnold O. Beckman, the founder of the Institute, and Dr. Ted Brown, the founding director. The Arnold and Mabel Beckman Foundation endowed the lecture series and continues to support it.

Ted Brown’s Remarks:
Ted Brown expressed his gratitude for the Beckman Foundation’s support and reminisced about Arnold and Mabel Beckman’s friendship and contributions. He mentioned the endowment gift from the Beckman Foundation for the annual lectureship and postdoctoral fellowship in interdisciplinary science. He acknowledged the newly appointed postdoctoral fellow, Aman Ahmed, who will work with Martha Gillette and Brad Sutton.

Cognitive Sciences at Beckman Institute:
When the Beckman Institute opened in 1989, it had a group called Cognitive Sciences, formed from various disciplines. The recognition of cognitive sciences as a significant discipline was still emerging.

The Embodied Mind:
In 1991, two years after the Institute’s opening, the book “The Embodied Mind” by Varela, Thompson, and Roche exemplified the concept of embodiment and the mind’s interaction with the world. Since then, there has been a proliferation of research in this area.

Today’s Lecture:
Today’s lecture by Dr. Fei-Fei Li is an example of the reach of the concept of embodiment and our connection with the world through artificial intelligence.

Conclusion:
Diane Beck concluded the introduction and invited Dr. Li to deliver the lecture.

Introduction:
Fei-Fei Li, a renowned computer scientist and human neuroscientist, is a Sequoia Professor at Stanford University and co-director of the Stanford Institute of Human-Centered AI. Her research spans cognitively inspired AI, machine learning, deep learning, computer vision, and AI for healthcare.

Groundbreaking Work and Achievements:
Fei-Fei Li’s contributions include boundary-crossing work in computer science and human neuroscience. She served as director of Stanford’s AI lab and vice president at Google, leading AI and ML initiatives.

Prestigious Positions and Honors:
Fei-Fei Li is an elected member of the National Academy of Engineering, the National Academy of Medicine, and the American Academy of Arts and Sciences. She serves on the National AI Research Task Force commissioned by Congress and the White House.

AI for All Initiative:
Fei-Fei Li co-founded and chairs AI for All, a national nonprofit dedicated to training diverse K-12 students from underprivileged communities to become future AI leaders.

Historical Connection to Beckman Institute:
Fei-Fei Li considers Beckman Institute the starting point of her career, where she began as an assistant professor and collaborated with Diane Beck.

Upcoming Presentation:
Fei-Fei Li’s presentation is titled “From Seeing to Doing, Understanding and Interacting with the Real World.”

Origins of Vision:
540 million years ago, life was limited to the ocean. The Cambrian explosion, a rapid diversification of animal species, occurred within 10 million years. Andrew Parker’s theory suggests that the evolution of vision triggered this explosion, leading to an evolutionary arms race.

Vision as a Core Sensory System:
Vision is essential for animal survival and navigation. Most animals rely primarily on vision, with few exceptions. Fei-Fei Li’s research focuses on studying vision and applying insights to machine intelligence.

Understanding the World through Vision:
Human visual system excels at object recognition. Experiments show humans can quickly detect incongruent objects in complex scenes. RSVP studies demonstrate rapid object detection in fast-changing videos. EEG ERP studies reveal that humans can categorize animal pictures within 150 milliseconds.

Cognitive Science Study on Object Recognition:
Fei-Fei Li’s study explored the role of context in object recognition. Results showed that context can significantly influence the speed and accuracy of object recognition.

Conclusion:
Vision is a fundamental aspect of intelligence, enabling understanding and action in the world. Research in vision is crucial for developing intelligent machines and advancing AI.

History of Object Recognition:
Computer scientists initially used hand-designed models to recognize objects, but these models were limited. Machine learning brought about statistical models that learned parameters from hand-designed features, leading to more powerful models. The availability of data through the internet facilitated the creation of large datasets for training and benchmarking.

The Limitations of Traditional Datasets:
Traditional datasets for object recognition were limited in size and scope, often focusing on a few dozen object classes. This approach neglected the vast diversity and scale of the real world.

The Creation of ImageNet:
Fei-Fei Li and her students created ImageNet, a dataset of 15 million images across 22,000 object classes. ImageNet aimed to respect the scale of the real world and provide a more comprehensive dataset for object recognition.

The Dawn of Deep Learning:
The ImageNet Object Recognition Challenge in 2012 was a turning point in object recognition. Geoffrey Hinton and his students’ convolutional neural network algorithm won the challenge, significantly improving performance. This event marked the beginning of the deep learning revolution.

The Continued Progress of Deep Learning:
Since 2012, there has been tremendous progress in deep learning and neural network-based algorithms for object recognition. Deeper and bigger networks with more parameters have led to better and better performance.

What’s Beyond Object Recognition:
Fei-Fei Li emphasizes the need to explore beyond object recognition to understand visual scenes more comprehensively. The relationship between objects matters for visual understanding. Jeremy Wolf’s paper inspires the idea of encoding relationships between objects in visual scenes.

Scene Graph Representation:
Scene graph representation is introduced, where objects are depicted as nodes, and relationships between objects are represented as edges. Visual Genome dataset is created to promote algorithms that understand visual relationships and language descriptions of visual scenes.

Visual Relationship Prediction and Zero-Shot Recognition:
Visual relationship prediction involves identifying relationships between objects in a scene. Scene graph representation enables zero-shot recognition of unusual relationships, like “horse-wearing hat” or “person sitting on fire hydrant.”

SYNGRAPH Representation:
SYNGRAPH representation has gained popularity among researchers in different areas. Spatial temporal scene graph representation is used for action and activity recognition in videos.

Story Generation from Visual Scenes:
Computers can generate descriptions of a scene in sentences, providing a complete understanding of the story depicted in the image.

Key Ingredients for Understanding:
Representation, learning algorithms, data, and benchmarks are essential ingredients for understanding. Fei-Fei Li’s lab has made significant contributions in these areas.

Is Understanding All:
Fei-Fei Li reflects on the limitations of understanding visual scenes, comparing it to Plato’s Allegory of the Cave. Understanding the world based solely on 2D projections on the retina can be frustrating and incomplete.

Vision and the Nervous System:
Vision is an active process, not passive. The nervous system links perception with action. Seeing and doing are greatly connected.

The Importance of Interplay:
Kitten study demonstrates the importance of movement and interaction for visual development. Active kittens develop normal perceptual capabilities, while passive kittens have damaged capabilities. Mirror neurons in primates further illustrate the link between doing and seeing.

Perception Drives Interaction:
Perception was driving interaction in evolution. The interplay between perception and interaction drove the evolution of intelligence. Seeing is for doing in the world.

Robotic Learning:
Robotics is a classic field, but challenges remain in unstructured environments. Robotic learning focuses on curiosity-driven and task-driven learning. Visual understanding and representation are crucial for robotic learning.

Curiosity-Based Learning:
Learning to play: building AI systems intrinsically motivated to interrogate and learn the world. World model network predicts consequences of actions. Self-model network predicts errors of the world model. The learning process empowers the self-model to produce adversarial actions and learn through interactions. Results show developmental trajectory of learning: ego motion, object attention, one object learning, multi-object learning.

Traditional Robotic Learning:
Short-term, short-horizon punctual activities (e.g., grasping, rotating, opening doors). Long-horizon task: cleaning up tabletop, stacking bowls and utensils. Modularized short-term skills into a representation called neurotask programming. Robots can sort multiple objects and deal with human destructive behavior.

Perceptually Driven Robotic Learning:
Research has focused on algorithms, especially deep RL-based algorithms. Challenges include small-scale experiments, lack of principled task selection, and absence of standard metrics. Real-world applications often fail due to lack of generalization.

Ecological Approach to Robotic Learning:
Inspiration from computer vision and NLP: large training environments and benchmarks drive research. Focus on real-world household environments: ecologically complex, dynamic, uncertain, variable, interactive, social, inherently multitask.

Behavior Benchmark for Everyday Household Activities:
Rolling out a benchmark called Behavior, Benchmark for Everyday Household Activities in Virtual Interactive Ecological Environments of a Thousand Human Tasks. Simulation of behavioral world with 1,000 tasks. Large scale and diverse, ecological in general, complex and standardized evaluation. Realistic and photorealistic, allows for kinematic and dynamic states of objects, flexible and deformable materials, realistic fluids, thermal effects, and interactivity.

Task Selection:
Importance of selecting relevant and impactful tasks. Consideration of societal impact, aging population, declining ratio of caregivers to seniors, and lack of good quality in aging. Asking humans what tasks they would prefer robots to do in a household setting. User study to assess the potential benefits of robots performing various tasks.

Understanding the Need for Behavior Learning:
Fei-Fei Li emphasizes the importance of understanding tasks that humans want robots to help with, such as cleaning, preparing food, and playing games. Researchers created Behavior1K, a dataset of 1,000 activities ranked by their importance to humans.

Creating a Logical Form Language for Robotics:
To enable robots to understand and execute tasks, a logical form language called BDDL (Behavior Domain Description Language) was developed. BDDL allows researchers to define the initial and goal states for robots, enabling planning and control.

The Importance of Simulation Environments:
Fei-Fei Li highlights the crucial role of simulation environments for robotics research. Simulation provides fast training, transferability, safety, and allows for fair and reproducible benchmarks. Realistic simulation environments like OmniGibson, developed in collaboration with NVIDIA, capture important factors like thermal effects and deformability.

Challenges in Behavior Learning:
Current state-of-the-art reinforcement learning algorithms face difficulties in behavior learning. Experiments show that providing privileged information to algorithms improves performance, indicating the need for further research.

Closing the Gap between Simulation and Reality:
Fei-Fei Li presents an apartment built at Stanford and a real robot used to bridge the gap between simulation and real-world applications. While there have been successes in navigation and grasping tasks, there are still challenges and failures to overcome.

Vision as a Cornerstone of Intelligence:
Fei-Fei Li reiterates her belief that vision is fundamental to intelligence. Research in vision and visual robotic learning focuses on representation, learning algorithms, planning and control, data, and benchmarks.

Psychology and Human Neuroscience in Robotics Learning:
In response to a question, Fei-Fei Li discusses the influence of psychology and human neuroscience on her work. While humans have certain advantages, such as recognizing more car types than computers, AI algorithms excel in other areas like categorizing objects. She emphasizes the need to explore foundation models, neuro-symbolic approaches, and effective computing to improve the robustness, nimbleness, and flexibility of AI algorithms.

Effective Computing of Emotions in Learning:
Addressing another question, Fei-Fei Li acknowledges the importance of effective computing of emotions in learning but admits that research in this area is still in its early stages. She mentions theory of mind research and sentiment understanding through words as potential starting points.

End-to-End vs. Modular Approach in Robotics Learning:
In response to a question about the end-to-end versus modular approach in robotics learning, Fei-Fei Li expresses her preference for a modular approach. She believes that decomposing tasks into perception, planning, and control modules allows for better optimization and understanding of each component.

Robotic Learning:
End-to-end and modular approaches are debated in robotic learning. Fei-Fei Li believes some level of modularization and neuro-symbolic representation may be necessary for generalization and reasoning abilities. Industry’s focus on brute force compute can help explore this question further.

Data Representation:
Fei-Fei Li acknowledges the importance of data representation but expresses unfamiliarity with specific theories mentioned by a questioner. She suggests exploring theories that consider the interplay between objects, actions, and intelligence.

Artificial General Intelligence (AGI):
Fei-Fei Li questions the meaning of AGI as a marketing term and emphasizes the long-standing dream of AI researchers to create machines with human-level intelligence. She sees AGI as encompassing general cognitive, perceptual, and behavior abilities.

Social Learning in Robots:
Fei-Fei Li recognizes the importance of other agents, including humans, in robotic learning. Imitation learning from humans is a significant area of research. Multi-agent behavior environments can facilitate learning from other robots and humans.

Computational Cost in Computer Vision:
Fei-Fei Li acknowledges the increasing computational cost of computer vision tasks. She suggests that the limited availability of resources at research institutions could impact progress in this area.

Background:
Digital divide refers to the gap in access to digital resources, including AI technologies and data. The digital divide affects academia, the public sector, and underserved scientific and educational communities.

Stanford HAI’s Advocacy:
Stanford HAI, led by Fei-Fei Li, recognized the digital divide and its impact. Lobbied for a bill in 2021 advocating for a national AI research cloud.

National AI Research Cloud:
The Trump administration passed a bill supporting the establishment of a national AI research cloud. Biden administration and Congress formed a 12-person task force to develop a proposal for a national AI research resource. Fei-Fei Li is part of the task force working with the White House and NSF to establish the national cloud and data commons.

Addressing the Complexity of Language and Thought Processes in AI:
Ted Brown, with a background in linguistics, raised a question about how AI can handle the complexity of language and thought processes. Fei-Fei Li acknowledged the challenge of AI understanding the nuances of language and context.

Potential Solutions:
Fei-Fei Li mentioned research on how AI can interpret contextual information and use knowledge graphs to understand the relationships between concepts. She emphasized the importance of collaboration between researchers in different fields, including linguistics, to advance AI’s ability to process language and thought effectively.

Fei-Fei Li’s Response to GPT-3:
Fei-Fei Li is impressed by the advancements in AI language models, particularly GPT-3. She mentions that many students, especially those at Stanford, use GPT-3. Fei-Fei Li conducted an experiment where she asked GPT-3 to write an essay using the same prompt given to her fifth-grade students. GPT-3 generated a 4,000-character essay on the pros and cons of electric cars that was indistinguishable from human writing, although it was not a prize-winning essay.

Ted Brown’s Response to GPT-3:
Ted Brown is unfamiliar with GPT-3. He expresses interest in receiving papers on the subject from Fei-Fei Li.

Conclusion:
The conversation highlights the rapid progress in AI language models, particularly GPT-3, and its potential impact on education and various fields.