A Chance Encounter:
Jensen Huang’s career in chip design began during a campus company day at Oregon State University. Inspired by a poster of a 29,000 32-bit CPU from AMD, Huang was drawn to the idea of chip design.

Software-Based Chip Design:
Huang had the opportunity to join a startup that pioneered chip design using software, an innovative approach at the time. Despite his interest in this new technology, Huang ultimately chose to join AMD to pursue his passion for microprocessor design.

Shifting to LSI Logic:
Huang’s lab partner from AMD joined LSI Logic, a company at the forefront of the EDA industry and chip design using computers. Impressed by Huang’s potential, the LSI team reached out to him, leading him to join their ranks.

Early Influences and Collaborations:
Huang’s time at LSI Logic exposed him to a range of talented computer architects, including the founders of Sun and Apple. These collaborations shaped his understanding of computer architecture and chip design.

Founding NVIDIA:
Along with Chris Malkowski and Curtis Priem, Huang co-founded NVIDIA, a company that would revolutionize the graphics processing unit (GPU) industry. Huang’s expertise in chip design and his collaborations with industry leaders laid the foundation for NVIDIA’s success.

Founding NVIDIA:
Jensen Huang initially worked at LSI Logic and was content with his career path. His co-founders, Chris and Curtis, wanted to leave Sun and convinced Huang to join them in starting a company focused on accelerated computing.

Accelerated Computing:
In the early days, the computer design industry primarily focused on general-purpose computing, with only a small minority believing in the potential of accelerated computing. NVIDIA’s mission became to solve problems that traditional computers couldn’t handle, leading them to explore applications in self-driving cars, robotics, climate science, digital biology, and artificial intelligence.

CUDA and General-Purpose Computing:
NVIDIA expanded the flexibility of its accelerators to become more general-purpose, creating a new computing model called CUDA. This allowed NVIDIA to address applications beyond graphics, including image processing, physics processing, and molecular dynamic simulation.

Balancing General-Purpose and Acceleration:
The challenge was to strike a balance between general purposeness and acceleration, as becoming too general-purpose would reduce the acceleration benefits. NVIDIA’s strategy was to find niche applications where a small R&D investment could yield significant acceleration, then gradually expand the application space.

Architecture Compatibility:
To attract developers and foster a platform for innovation, NVIDIA maintained architecture compatibility across its chips, ensuring that all NVIDIA chips performed the same way. This long-term commitment to CUDA compatibility initially resulted in low gross margins, but it laid the foundation for NVIDIA’s success in the AI era.

Conclusion:
Jensen Huang’s vision of accelerated computing and the development of CUDA laid the groundwork for NVIDIA’s success in artificial intelligence and other demanding applications. The company’s commitment to architecture compatibility and expanding the aperture of its applications has positioned it as a leader in the field of computing.

Early Applications of GPUs:
NVIDIA initially targeted specific applications for its GPUs, such as NAMD (a molecular dynamics simulation tool) and seismic processing. These applications involved complex calculations that could benefit from the parallel processing capabilities of GPUs.

The Emergence of AI:
Around 2012, NVIDIA began to recognize the potential of AI, particularly deep learning, for various applications. Andrew Ng approached NVIDIA to explore using GPUs for neural network training, aiming to reduce the computational requirements compared to traditional CPU-based approaches. Simultaneously, Geoffrey Hinton and Yann LeCun were also exploring the use of GPUs for deep learning in their respective labs.

ImageNet and the Deep Learning Revolution:
The release of AlexNet in 2012 marked a significant moment in the field of deep learning. AlexNet’s success in the ImageNet competition demonstrated the potential of deep learning for image classification tasks. This event drew widespread attention to the field and led to a surge of interest in deep learning and AI research.

NVIDIA’s Role in the AI Revolution:
NVIDIA’s GPUs played a crucial role in enabling the deep learning revolution by providing the necessary computational power for training and running deep learning models. The company’s GeForce gaming cards, initially designed for gaming applications, were repurposed for deep learning tasks due to their powerful parallel processing capabilities.

Introduction to AI’s Beginnings in NVIDIA:
Sarah Guo’s dialogue with Jensen Huang, NVIDIA’s CEO, starts with the early use of gaming cards in AI labs. Huang explains that NVIDIA’s approach is to enhance applications once they find utility. This led to their involvement in deep learning, initially focused on computer vision applications like self-driving cars and robotics. They recognized deep learning’s potential to revolutionize software development and its impact on various aspects of technology, such as chip design and system software.

Deep Learning’s Breakthroughs:
Huang highlights the transformative power of deep learning, especially in computer vision. He notes its superior effectiveness compared to decades of prior algorithms, raising questions about its scalability and broader implications for computer science. This led to exploring the limits of this ‘universal function approximator’ in solving complex problems with high dimensionality.

AI’s Broad Influence in Computing:
The discussion progresses to AI’s impact over the past decade, with the development of new models like CNNs, ResNets, RNNs, LSTMs, and significant advancements in perception models. Sarah Guo mentions the importance of Transformers and BERT, prompting Huang to discuss how these models, with their parallel training capabilities and effectiveness in understanding spatial and sequential data, have marked a significant shift in the industry.

Transformers and Language Models:
Huang expresses excitement about Transformers overcoming previous limitations in learning sequential data. He also touches on the integration of reinforcement learning, human feedback, and retrieval models, leading to advanced applications like ChatGPT. This marks a pivotal moment in computing, where programming languages shift towards natural, human language, making computer programming accessible to a broader audience.

Democratization of Coding with AI:
Huang reflects on the democratizing effect of AI tools like ChatGPT and Copilot. He shares an anecdote where people who previously couldn’t code started coding with the help of AI. Huang emphasizes the unique ability of AI to reason through problems and even write programs to solve them, underscoring the profound implications of AI in the future of computing and potentially leading towards machine sentience.

NVIDIA’s Computing Platform:
NVIDIA is a computing platform company that enables developers to utilize its platform for various purposes.

Evolution of Developers:
Initially, developers had control over their operating systems, requiring NVIDIA’s involvement only at the device driver level. In domains like scientific computing, developers use solvers and algorithms that need to be accelerated for optimal performance.

Algorithm Development:
NVIDIA recognized the need to develop algorithms themselves, particularly in multi-domain physics problems. The relationship between problem-solving algorithms and underlying computer architecture necessitated this approach.

Domain-Specific Languages:
NVIDIA’s expertise extends to designing computer algorithms for different domains, including particle physics, fluid dynamics, and deep learning. cuDNN, for example, serves as a domain-specific language for accelerated deep learning.

RTX and Ray Tracing:
NVIDIA has also developed domain-specific libraries for computer graphics, such as RTX for ray tracing.

Future Developers:

The question arises: who is a developer in the future?

Developers and Large Language Models:
Jensen Huang envisions a future where developers primarily engage with large language models or foundation models. He encourages the use of models from OpenAI (like ChatGPT), Microsoft, and Google, highlighting their effectiveness. However, Huang acknowledges the need for proprietary models in niche domains such as proteins, chemicals, climate science, or multi-physics, recognizing their significant market potential despite not being universally applicable.

Foundation Models for Specific Domains:
Huang discusses NVIDIA’s potential focus on creating foundation models for specific areas like 3D graphics, virtual worlds, and robotics. These areas align with NVIDIA’s core competencies. The strategy is to develop these models to a necessary extent but not aim to become an AI model company per se. NVIDIA’s goal is to assist industries and developers in creating their own AI models.

Efficiency and Principles of NVIDIA:
In an interesting twist, Huang describes NVIDIA’s philosophy as striving to be as “lazy” as possible – doing as little as necessary and as much as necessary. This approach is rooted in the first principles of computer science: rejecting and deferring work efficiently. This principle, while seemingly counterintuitive, aligns with efficient and strategic resource allocation.

Long-term Commitments vs. Short-term Pressures:
Sarah Guo probes into how NVIDIA balances long-term commitments like CUDA with the immediate pressures of being a large public company. Huang responds that investing in the future and being sustainable in the present are not conflicting goals. The challenge for CEOs is to adhere to the core beliefs of their company while ensuring the financial feasibility of their long-term visions. This balance is crucial in maintaining innovation and growth in a competitive market.

Conviction and Skill in Making Money:
Jensen Huang emphasizes that making money is not a matter of conviction but a learnable skill that he acquired over time. He acknowledges that it took him 20 years to fully grasp this skill and stresses the importance of developing it within the company. Huang believes that the purpose of a company should be driven by genuine conviction and a singular pursuit of its beliefs.

Accelerated Computing as the Core Belief:
NVIDIA’s core belief lies in singularly advancing the new computing model of accelerated computing. Huang expresses excitement about the potential of accelerated computing to solve problems beyond the capabilities of normal computers, leading to discoveries in fields like digital biology, climate change, robotics, and self-driving cars. He emphasizes that NVIDIA’s focus on tackling impossible applications has led to groundbreaking discoveries, including artificial intelligence and large language models.

The Future of Accelerated Computing:
Huang firmly believes that everything will eventually be accelerated due to the limitations of general-purpose computing. While CPUs will always be needed, acceleration is seen as the optimal approach for the types of applications that will be prevalent in the future. This belief has been at the core of NVIDIA since its inception 30 years ago, driven by genuine conviction.

Huang’s Confidence in Accelerated Computing:
Jensen Huang believes in the crucial role of accelerated computing in solving complex problems.

The Inevitability of Accelerated Computing:
He sees accelerated computing as the only approach for solving problems that are impossible with traditional computing.

Challenges with CPU Limitations:
Huang mentions that certain applications outgrew CPU capabilities 12 years ago, necessitating the shift to accelerated computing.

Examples of Early Adoption:
He highlights pioneers like Geoffrey Hinton, Yann LeCun, and Andrew Ang who recognized the need for accelerated computing in computer graphics and AI.

Company Adaptation to AI’s Rise:
The company has aligned aspects of its business with the growing prominence of AI.

Management in a Rapidly Changing Environment:
Huang emphasizes the need for constant adaptation and reassessment of management approaches in this fast-paced environment.

Company Architecture:
Huang emphasizes that a company’s architecture should not be generic but tailored to its specific purpose, function, and leadership style. NVIDIA’s structure allows for focused work in key areas while enabling innovation and discovery at the senior level.

Innovation:
Huang highlights the importance of understanding computer architecture and having a disciplined approach to it. NVIDIA maintains a single instruction set and computer architecture, enabling focused efforts and innovation. Skunkworks-like teams are used to explore long-term, experimental projects with potential for future impact.

H100 GPU Innovations:
Quantization, particularly 8-bit floating point, is a key breakthrough, enabling significant performance improvements. The H100’s transformer engine is designed specifically for learning and inferencing transformers, addressing their widespread utility. The H100 represents the largest, fastest, and most energy-efficient chip ever made, utilizing the world’s fastest memories.

Agile Projects and Impossible Applications:
NVIDIA focuses on developing agile projects that aim to achieve seemingly impossible applications that are anticipated to become important in the future. Some of these projects may not be successful at the moment, but the company remains confident in their potential.

Autonomous Driving and Robotic Foundation Models:
Jensen Huang expresses confidence in the progress of autonomous driving and believes that a robotic foundation model will be discovered. Through natural language expression, it will be possible to control a robotic system with various limbs and agility to perform specific tasks.

Blockers to Robotic Foundation Models:
The blockers to the development of robotic foundation models are currently unknown. It requires a process of discovery and exploration to find the path forward.

Learning Structure from Unstructured Information:
NVIDIA has expertise in learning structure from unstructured information, including language, images, and videos. Learning the structure from videos may provide insights into articulation and lead to the development of an articulation system for robots.

Timeline for Robotic Developments:
Jensen Huang estimates that significant progress in robotics could occur within the next 5-10 years. The company expects to see remarkable advancements in robotics during this period.

Palmy from Google:
The recent introduction of Palmy by Google is seen as a step in the direction of robotic advancements. Palmy utilizes transformer architecture, which is a foundational approach in AI.

Other Architectural Developments:
Derivatives of transformers, collectively referred to as transformers, are being refined and improved. NVIDIA has also made significant contributions to the field of generative adversarial networks (GANs), style transfer, and high-resolution image generation. This work has led to developments in variational autoencoders and diffusion models.

Learning and Generating Content:
The ability to learn structure from vast amounts of data, including video and multi-modality, is crucial. NVIDIA emphasizes the importance of generating content, including images, 2D and 3D images, proteins, chemicals, and various other types of content.

Jensen Huang’s Advice for Entrepreneurs:
Building a company is incredibly rewarding, but also painful. Ignorance is a superpower for entrepreneurs, but it’s important to stay agile and learn along the way. Be determined to stay with your conviction, but not so stubborn that you can’t adapt. Have resilience and forget the pain of failure to move on.

Jensen Huang’s Hopes for the Future:
He wants NVIDIA to make a significant contribution to healthcare and drug discovery by using AI to understand the language and meaning of proteins and chemicals. He hopes NVIDIA can create a foundation model for multi-physics for climate science to answer complex questions about the Earth’s future. He believes that AI has the potential to reduce the computational requirements for climate science by a billion to 10 billion times.