Introduction:
Alice Kaplan, Director of the Whitney Humanities Center, welcomes attendees to the Tanner Lecture on Human Values. The lecture is hosted by Yale University and presented by Fei-Fei Li, Sequoia Professor of Computer Science at Stanford University.

Fei-Fei Li’s Contributions:
Li is a leader in establishing a human-centered framework for artificial intelligence (AI) in research, education, practice, and policy.

Significance of the Lecture:
The lecture is particularly meaningful for the Whitney as it marks the first Tanner Lecture since the pandemic and the first held in the Humanities Quadrangle. It allows the Whitney to engage with scholars working in AI and related fields across Yale.

The Tanner Lectures on Human Values:
Established by Obert Clark Tanner and his wife, Grace Tanner. Tanner was a New Testament scholar and industrialist who founded a successful jewelry company. Tanner Lectureships recognize uncommon achievement and outstanding ability in the field of human values. Lecturers come from diverse fields and backgrounds, transcending ethnic, national, religious, and ideological distinctions.

Yale’s Selection Process:
Tanner lecturers at Yale are selected by the president in consultation with the director and the executive committee of the Whitney Humanities Center.

Introduction of Professor Marvin Chun:
Professor Marvin Chun is introduced as a leader in the study of brain imaging and artificial intelligence. Kaplan expresses her pleasure in welcoming Chun after working with him as Dean of the College.

Introduction of Dr. Fei-Fei Li:
Dr. Fei-Fei Li is a renowned researcher in artificial intelligence (AI) and has made significant contributions to the field. She is recognized for her pioneering work in scene perception and categorization, as well as her development of ImageNet, a labeled database of images that has revolutionized computer vision.

Dr. Li’s Achievements and Impact:
Dr. Li has published over 200 scientific articles and has received over 181,000 citations, demonstrating her measurable impact on the field. She has been recognized with numerous honors and awards, including election to prestigious academies and receipt of national awards. Dr. Li is a strong advocate for diversity and inclusion in STEM and AI, serving as a role model and leader in these areas.

Dr. Li’s Perspective on AI with a Human Perspective:
Dr. Li believes that AI should be developed with a human perspective, considering human values and understanding human needs. She emphasizes the importance of visual intelligence, which has been a driving force in the evolution of animal species and is crucial for human survival and thriving. Dr. Li draws parallels between the evolution of vision and the development of AI, highlighting the significance of vision in shaping intelligence.

The Cambrian Explosion and the Role of Vision:
Dr. Li discusses the Cambrian Explosion, a period of rapid animal speciation, and proposes that the evolution of vision played a key role in this event. Vision became a major sensory system for animals, enabling them to navigate, communicate, and interact with their surroundings. The evolution of vision led to the development of the nervous system and eventually to highly intelligent animals, including humans.

Conclusion:
Dr. Fei-Fei Li is a visionary leader in the field of AI, whose work has had a profound impact on computer vision and machine learning. Her emphasis on AI with a human perspective underscores the importance of considering human values and understanding human needs in the development of AI systems.

Machine Learning and AI Concepts:
AI: Coined by John McCarthy, it refers to creating intelligent machines. Machine Learning: A discipline combining statistics, applied math, neuroscience, and psychology to create statistical models for AI tasks. Neural Networks: A family of algorithms inspired by the human brain’s structure and function; also known as deep learning. Generalization: Developing an algorithm that performs well on unseen data (testing error) rather than just on the training data (training error).

Human Visual System:
Humans have remarkable visual abilities, including rapid object recognition and categorization. Studies have shown that the human brain can recognize objects within 150 milliseconds of exposure to visual stimuli. Specific areas of the brain are dedicated to object recognition, such as faces, places, and body parts.

Object Recognition:
A fundamental building block of visual intelligence. The task is to show an agent an image and have it output the category of the object in the image. Mathematically challenging due to infinite possibilities of lighting, appearance, background clutter, and viewing angles.

Early Days of Computer Vision:
Pioneering computer scientists used hand-designed features and models for object recognition. Limited by compute, data, and mathematical models.

Machine Learning Revolution:
Machine learning emerged as a powerful tool for computer vision. Researchers began using hand-designed features with machine learning models to learn parameters.

Data Availability and the Pascal VOC Dataset:
The advent of the internet made large amounts of data available for training AI and computer vision algorithms. A group of European scientists created the Pascal VOC dataset, consisting of a few thousand images with 20 object classes, to facilitate research in object recognition.

Cognitive Psychologists’ Findings:
Cognitive psychologists have discovered that humans can recognize over 30,000 object categories by the age of six. This contrasts with the limited number of object classes (20) used in the Pascal VOC dataset.

Realization of Data as a Limiting Factor:
Fei-Fei Li and her students examined previous work in object recognition and realized that the limited data available for training models might be a more significant problem than the hand-designed models themselves.

Collecting and Curating a Massive Image Dataset:
In a three-year effort, Fei-Fei Li and her collaborators downloaded over a billion images from the internet between 2006 and 2007. They used a crowd engineering method to manually curate this vast dataset.

Data, Compute, and Neural Networks: The Foundation of the Deep Learning Revolution:
Fei-Fei Li highlights the significance of ImageNet, a dataset comprising 15 million meticulously labeled images across 22,000 object categories, which fueled the deep learning revolution in artificial intelligence (AI). Neural network models, particularly convolutional neural networks, have been instrumental in computer vision’s success due to their ability to handle large amounts of data and extract meaningful features. The ImageNet dataset and neural network algorithms have enabled computers to achieve remarkable progress in object recognition tasks, such as categorizing images and finding objects in natural photos.

The ImageNet Challenge and the Dawn of Deep Learning:
The ImageNet Challenge, an international competition launched in 2010, invited researchers to participate in an object recognition task using the ImageNet dataset. In 2012, Professor Geoffrey Hinton’s team introduced AlexNet, a deep convolutional neural network model that significantly reduced error rates in object recognition tasks. AlexNet’s success is often regarded as the beginning of the deep learning revolution, marking a transformative era in computer vision and AI.

Progress and Impact of Computer Vision:
Computer vision has witnessed tremendous progress since the advent of deep learning, with annual winners of the ImageNet Challenge pushing the boundaries of the field. The deep learning revolution has had a profound impact on industries and academia, driving innovation and transforming various sectors. Computer vision has become a multi-billion-dollar market, with numerous startups emerging to harness its potential in various applications.

Beyond Object Recognition: Scene Graph Representation and Storytelling:
Fei-Fei Li emphasizes that object recognition is just one aspect of visual intelligence, and understanding relationships between objects is crucial for scene comprehension. Scene graph representation encodes objects, pairwise relationships, and attributes in a scene, providing a richer and more comprehensive understanding of visual data. Scene graph representation enables tasks such as predicting object relationships, zero-shot learning, and even visual storytelling and captioning.

Exploring Spatial-Temporal Relationships and Multi-Object Activities:
Computer vision has expanded beyond static images, delving into recognizing multi-object activities and relationships in dynamic scenes. Scene graph representation has been extended to spatial-temporal space, allowing algorithms to recognize movements, interactions, and relationships between multiple objects. This has led to advancements in recognizing complex activities and interactions, such as sports games and human-object interactions.

Inspiration from Brain Science and Human Cognition:
Fei-Fei Li stresses the importance of brain science and human cognitive science in inspiring AI development. Understanding how the human brain processes and interprets visual information can provide valuable insights for designing AI algorithms that mimic human visual capabilities.

Fine-Grained Object Categorization: Seeing Beyond Human Visual Ability:
Computer vision algorithms have surpassed human visual abilities in fine-grained object categorization, such as distinguishing between thousands of bird species or car types. This demonstrates the potential of AI to see and understand the world in ways that humans cannot.

Computer Vision and Insights into Neighborhoods:
Computer vision can analyze visual data from sources like Craigslist and Google Street View to learn about car types and neighborhood characteristics. It can recognize 2,800 car types since the 1970s and correlate this information with census data, such as education level, income, and voting patterns. Computer vision acts as a lens to not only see the world but also understand human society and sociology.

Limitations of Human Vision:
The Stroop test highlights the limitations of human vision, where reading the color of words (e.g., “red” in green) can be challenging due to conflicting information. Change blindness demonstrates the limited attention resources of humans, as changes in alternating pictures may not be detected as quickly as a computer program can.

Medical Errors and Visual Attention:
Medical errors are the third leading cause of death in the American healthcare system. Visual attention is crucial in medical scenarios, such as accounting for surgical instruments during surgeries. Manual tabulation and counting of surgical instruments by nurses and doctors slow down surgeries and can lead to errors.

Conclusion:
Computer vision has the potential to enhance our understanding of neighborhoods and human society, but human vision has limitations, particularly in multitasking and change detection. These limitations can have severe consequences, such as medical errors, highlighting the need for further research and technological advancements in computer vision.

Challenges of AI-Based Computer Vision:
AI algorithms, like computer vision, can amplify human bias, especially when trained on biased data. AI can also lead to privacy concerns as computer vision algorithms can be used to collect and analyze data without individuals’ consent.

Addressing Bias in AI:
Researchers are actively working on addressing bias in AI algorithms, such as facial recognition software, to ensure fair and equitable outcomes for all.

Seeing What Humans Don’t See:
Computer vision AI can be used to identify and analyze data that humans may not be able to perceive, such as surgical instruments in a surgical room or patterns in medical images.

Visual Illusions and Bias:
Visual illusions demonstrate how our prior knowledge and expectations can influence our perception of visual information, leading to misinterpretations. This highlights the importance of considering human cognitive biases when developing AI algorithms.

The Impact of AI on Society:
AI has the potential to impact society in profound ways, both positive and negative. It is crucial for technologists and other stakeholders to work together to study, forecast, and guide AI’s impact on people and society in a multidisciplinary approach.

Privacy-Aware Computer Vision Computing:
Researchers are developing technical approaches to privacy computing in computer vision, such as face blurring, dimensionality reduction, body masking, federated learning, and homographic encryption. These techniques aim to balance the need for computer vision technology to recognize human gestures and activities while protecting individual privacy.

Balancing Privacy and Functionality:
One example is a privacy-aware computer vision system that uses a lens to distort the input, making it difficult to identify individuals, while still allowing the algorithm to recognize relevant human activities.

Multidisciplinary Approach to AI Development:
To ensure the responsible development and deployment of AI, a multidisciplinary approach is necessary, involving technologists, policymakers, ethicists, and other stakeholders.

Labor Threat vs. Labor Shortage:
The focus on AI’s labor threat overlooks the current labor shortage in many industries, such as healthcare, where there is a shortage of nurses and overworked healthcare workers.

Balancing AI’s Labor Threat:
AI should be used to augment human capabilities, not replace them.

AI’s Augmentation in Healthcare:
Medical error is a leading cause of death in the American healthcare system. Ambient intelligence for healthcare uses smart sensors and ML algorithms to glean health-critical insights for patients and clinicians.

Hand Hygiene Practice:
Proper hand hygiene is crucial in reducing hospital-acquired infections. Traditional methods like human auditors or RFID technology are ineffective or impractical.

AI-Enabled Hand Hygiene Monitoring:
AI-based systems can monitor hand hygiene compliance more effectively and provide real-time feedback. A study in a hospital showed a significant improvement in hand hygiene compliance after implementing an AI-based system.

AI’s Potential in Augmenting Human Capabilities:
AI can help tackle labor shortages, improve safety, and enhance human capabilities across various industries. The focus should be on collaboration between humans and AI, not replacement.

AI-Driven Hand Hygiene Monitoring:
Implemented smart sensors in hospitals to monitor hand hygiene compliance. Developed algorithms to recognize proper hand sanitization techniques. System outperforms human observers in accuracy.

AI-Enabled Patient Mobility Assessment:
Developed AI algorithms to monitor patient mobility in ICUs. System recognizes four types of mobility activities. Algorithm accuracy is on par with human-labeled ground truth.

AI for Aging in Place:
Explored the potential of AI in senior homes and aging in place settings. AI can predict and prevent unsafe events, monitor patients, and manage chronic conditions. AI can address the labor shortage of caretakers.

Robotics in Healthcare:
Envisioned a future where robots assist seniors with daily tasks. Robots can supplement human caregivers, not replace them. Current robots are limited in capabilities and operate in structured environments.

Challenges in Robotics Research:
Most robotic research focuses on short-horizon tasks in simple environments. Real-world applications require robots to handle complex, dynamic, and uncertain situations. Lack of standardized benchmarks and metrics hinders progress.

Bridging the Gap:
Drawing lessons from the deep learning revolution, the field needs to: Collect large, diverse datasets. Develop robust algorithms. Establish standardized benchmarks and metrics. Foster collaboration among researchers and practitioners.

Benchmarking Robotic Algorithms:
To benchmark current robotic algorithms, three activities from Behavior Project were chosen: store decoration, collecting trash, and cleaning a table. Without providing artificially privileged information, today’s state-of-the-art algorithms performed poorly. Relaxing the conditions and giving privileged information, such as action primitives and artificial memory, improved the performance.

Creating a Diverse Dataset:
A large-scale user study with 1,400 participants was conducted to understand human needs and preferences for robotic assistance. The top-ranked 1,000 household activities from the study were selected as the baseline for the robotic learning project.

Simulation Environment:
Behavior Project collaborated with NVIDIA’s Omniverse team to create a realistic simulation environment for training robots in physics, perception, and interaction. The environment includes realistic physics, lighting and reflection effects, fluid, deformability, and transparency. A user study showed that Behavior’s environment outperformed other simulation environments in terms of perceptual realism.

Comparison with Other Simulation Environments:
Behavior Project’s environment is more diverse, larger-scale, and complex compared to other simulation environments. It offers a more challenging benchmark for robotic learning algorithms.

Conclusion:
Behavior Project provides a comprehensive benchmark for everyday household activities in virtual, interactive, and ecological environments. The project’s simulation environment is a valuable tool for training robots in physics, perception, and interaction. Behavior Project has the potential to drive progress in robotic learning and embodied AI.

AI for Household Activities:
Fei-Fei Li introduces the “Behavior” project, aiming to train robotic learning algorithms for various household tasks. The project involves creating simulation environments and a real-world apartment with a robot called Marvin. Marvin is tasked with completing tasks like picking up a bottle and placing it in a trash bin, but it often encounters challenges due to complexity and errors. The goal is to develop household robots that augment human capabilities rather than replacing them.

AI to Augment Humans:
AI should seek to augment and enhance human abilities rather than replace them. AI technologies should be inspired by human intelligence and brain sciences. Stanford University launched the Human-Centered AI Institute (HAI) with a focus on these principles.

HAI Initiatives:
HAI launched the Digital Economy Lab to study the impact of digital technologies on society, labor markets, and the economy. HAI has also collaborated with the Stanford Law School to address ethical and legal issues related to AI. HAI promotes interdisciplinary research and collaboration among faculty and students from various schools.

Stanford’s Center for Research on Foundation Models:
Fei-Fei Li has launched a center dedicated to studying the profound ethical, societal, and technological implications of large foundation models like GPT-3 and DALI arts. The center brings together professors, faculty, and researchers to investigate these models.

Ethics and Society Review of AI Research Grants:
Stanford has pioneered a novel review process called Ethics and Society Review for AI research grants, inspired by the Human Subject IRB review. This review is conducted by a multidisciplinary team including political scientists, humanists, and computer scientists.

Education and Outreach:
Stanford integrates ethics education into computer science classes for undergraduate and graduate students. AI education is also extended beyond Stanford to professional audiences such as business executives, journalists, lawyers, judges, and policymakers.

Lobbying for National AI Research Resources:
Stanford played a leading role in the successful lobbying efforts for the National AI Research Resource bill. Fei-Fei Li served on the 12-person task force appointed by the Biden administration to rejuvenate America’s public and higher education sectors’ capabilities in AI innovation and education.

Human Lessons and Values in the Journey of AI:
Fei-Fei Li emphasizes that the technical journey of AI has taught us more about human lessons and values than machine-learned lessons. This understanding is only at the beginning of its exploration.