Jensen Huang (Nvidia Co-founder) – No Priors (Apr 2023)


Chapters

00:00:00 Journey of a Chip Designer: From LSI to Starting NVIDIA
00:02:55 Evolution of NVIDIA's Accelerated Computing Platform
00:09:59 Rise of AI and Adoption by Tech Giants
00:12:53 The Grand Shift to AI Platforms: Unveiling the Implications of Generative Models
00:19:31 Domains and Algorithms for Accelerated Computing
00:22:43 AI Foundation Models: A Customized Approach
00:26:07 Accelerated Computing: A Purpose-Driven Enterprise
00:29:25 Challenges and Shifts in Leadership and Management in an Era of Accelerated Computing
00:31:31 Optimizing Company Structure for Innovation and Efficiency
00:38:39 Future Directions of AI: Autonomous Driving, Robotic Foundation Models, and Generative Content
00:42:33 Entrepreneurship Advice from NVIDIA's Jensen Huang

Abstract

Article “NVIDIA’s Transformative Journey: From Chip Design to AI Pioneers”

Jensen Huang’s journey, from his initial interest in chip design at a campus company day to his leadership role in NVIDIA, encapsulates a remarkable story of innovation and strategic foresight. This article delves into NVIDIA’s evolution, highlighting its foundational years, the strategic pivot towards accelerated computing, and its recent groundbreaking advancements in AI and deep learning. Alongside the technical achievements, we explore Huang’s philosophical approach to business and leadership, which has been pivotal in steering the company through rapid technological changes and establishing it as a leader in AI and computing.

The Genesis of NVIDIA and Accelerated Computing

Jensen Huang’s career in chip design began during a campus company day at Oregon State University. Inspired by a poster of a 29,000 32-bit CPU from AMD, Huang was drawn to the idea of chip design. He had the opportunity to join a startup that pioneered chip design using software, an innovative approach at the time. Despite his interest in this new technology, Huang ultimately chose to join AMD to pursue his passion for microprocessor design. Huang’s lab partner from AMD joined LSI Logic, a company at the forefront of the EDA industry and chip design using computers. Impressed by Huang’s potential, the LSI team reached out to him, leading him to join their ranks. Huang’s time at LSI Logic exposed him to a range of talented computer architects, including the founders of Sun and Apple. These collaborations shaped his understanding of computer architecture and chip design. Along with Chris Malkowski and Curtis Priem, Huang co-founded NVIDIA, a company that would revolutionize the graphics processing unit (GPU) industry. Huang’s expertise in chip design and his collaborations with industry leaders laid the foundation for NVIDIA’s success.

In the early days, the computer design industry primarily focused on general-purpose computing, with only a small minority believing in the potential of accelerated computing. NVIDIA’s mission became to solve problems that traditional computers couldn’t handle, leading them to explore applications in self-driving cars, robotics, climate science, digital biology, and artificial intelligence. NVIDIA expanded the flexibility of its accelerators to become more general-purpose, creating a new computing model called CUDA. This allowed NVIDIA to address applications beyond graphics, including image processing, physics processing, and molecular dynamic simulation. The challenge was to strike a balance between general purposeness and acceleration, as becoming too general-purpose would reduce the acceleration benefits. NVIDIA’s strategy was to find niche applications where a small R&D investment could yield significant acceleration, then gradually expand the application space. To attract developers and foster a platform for innovation, NVIDIA maintained architecture compatibility across its chips, ensuring that all NVIDIA chips performed the same way. This long-term commitment to CUDA compatibility initially resulted in low gross margins, but it laid the foundation for NVIDIA’s success in the AI era.

NVIDIA’s Foray into AI and Its Impact

NVIDIA initially targeted specific applications for its GPUs, such as NAMD (a molecular dynamics simulation tool) and seismic processing. These applications involved complex calculations that could benefit from the parallel processing capabilities of GPUs. Around 2012, NVIDIA began to recognize the potential of AI, particularly deep learning, for various applications. Andrew Ng approached NVIDIA to explore using GPUs for neural network training, aiming to reduce the computational requirements compared to traditional CPU-based approaches. Simultaneously, Geoffrey Hinton and Yann LeCun were also exploring the use of GPUs for deep learning in their respective labs. The release of AlexNet in 2012 marked a significant moment in the field of deep learning. AlexNet’s success in the ImageNet competition demonstrated the potential of deep learning for image classification tasks. This event drew widespread attention to the field and led to a surge of interest in deep learning and AI research. NVIDIA’s GPUs played a crucial role in enabling the deep learning revolution by providing the necessary computational power for training and running deep learning models. The company’s GeForce gaming cards, initially designed for gaming applications, were repurposed for deep learning tasks due to their powerful parallel processing capabilities.

NVIDIA’s journey into AI, notably beginning around 2012 through collaborations with AI luminaries like Andrew Ng, Geoffrey Hinton, and Yann LeCun, coincided with the rise of neural networks and the ImageNet challenge. Initially designed for gaming, NVIDIA’s GPUs found a new purpose in AI due to their suitability for linear algebra operations. Sarah Guo’s dialogue with Jensen Huang reveals NVIDIA’s evolutionary strategy in AI, emphasizing application enhancement and recognizing deep learning’s potential beyond computer vision, in areas such as self-driving cars and robotics. Huang points out the surprising effectiveness of ImageNet in computer vision, a breakthrough that prompted questions about deep learning’s scalability and implications for computer science.

In conversations with Sarah Guo, NVIDIA CEO Jensen Huang reflects on the early use of their gaming cards in AI labs, emphasizing the company’s approach to enhancing applications with proven utility. This focus led them into the realm of deep learning, initially targeting computer vision applications like self-driving cars and robotics. Deep learning’s effectiveness, particularly in computer vision, was transformative, outperforming decades of prior algorithms. This led NVIDIA to explore the limits of deep learning as a ‘universal function approximator’ in solving complex, high-dimensional problems. AI’s influence over the past decade has been notable, with the development of new models like CNNs, ResNets, RNNs, LSTMs, and substantial advancements in perception models. Guo also brings up the importance of Transformers and BERT, leading Huang to discuss how these models, effective in understanding spatial and sequential data, have marked a significant shift in the industry.

Huang expresses enthusiasm about Transformers and their ability to overcome previous limitations in learning sequential data. He also discusses the integration of reinforcement learning, human feedback, and retrieval models, leading to advanced applications like ChatGPT. This represents a pivotal moment in computing, transitioning programming languages towards natural, human language, and making computer programming more accessible. Huang reflects on the democratizing effect of AI tools like ChatGPT and Copilot, sharing anecdotes of people who started coding with AI assistance. He emphasizes AI’s unique ability to reason through problems and even write programs to solve them, pointing to profound implications for the future of computing and the potential for machine sentience.

Conviction and Skill in Making Money and Accelerating Computing

Jensen Huang underscores that making money is a skill that can be learned, a realization that took him 20 years to fully grasp. He advocates developing this skill within the company, emphasizing that a company’s purpose should be rooted in genuine conviction and a singular pursuit of its beliefs. NVIDIA’s core belief in advancing accelerated computing has driven its focus on solving problems beyond the capabilities of normal computers, leading to discoveries in fields like digital biology, climate change, robotics, and self-driving cars. Huang believes that everything will eventually require acceleration due to the limitations of general-purpose computing, maintaining that CPUs will always be necessary but acceleration is the future for prevalent applications. This belief has been foundational to NVIDIA for three decades, driven by genuine conviction.

Company Architecture, Innovation, and the H100 GPU

Huang stresses the importance of a company’s architecture being specifically tailored to its purpose, function, and leadership style. NVIDIA’s structure facilitates focused work in key areas while fostering innovation and discovery at the senior level. He highlights the significance of understanding computer architecture and a disciplined approach to it. NVIDIA’s commitment to a single instruction set and computer architecture has enabled focused efforts and innovation. The company

employs skunkworks-like teams for long-term, experimental projects with future potential. The H100 GPU represents a major breakthrough with its transformer engine designed for learning and inferencing transformers, making it the largest, fastest, and most energy-efficient chip ever made, using the world’s fastest memories.

The Platform Shift and Democratization of Coding

Huang and Guo’s conversation delves into the broader impact of AI on computer science, with NVIDIA playing a significant role in AI models, especially perception models. Huang discusses the role of Transformers and BERT in overcoming learning sequential data limitations and his insights into ChatGPT, which signify a paradigm shift in programming. The democratization of coding, facilitated by AI tools like ChatGPT and Copilot, represents a profound change in problem-solving and programming methodologies. Huang, while uncertain about the definition of “sentience,” acknowledges the reasoning and problem-solving capabilities of advanced software.

NVIDIA’s Future Outlook and Development Strategy

NVIDIA’s future strategy involves supporting specialized domains like drug discovery or climate science through niche foundation models. Huang envisions future developers heavily relying on large language models. NVIDIA aims to aid industries in creating their own AI models rather than being an AI model company per se. Its minimalist approach balances long-term commitments with immediate demands, adhering to a principle of ‘laziness’ that ensures sustainability while sticking to core beliefs.

Jensen Huang’s Vision and Advice for Entrepreneurs

Huang envisions the realization of autonomous driving and robotic foundation models, emphasizing the importance of learning from unstructured information for developing robotic systems. He sees generative models playing a critical role in various domains. For entrepreneurs, he advises embracing ignorance as a strength, underscoring resilience, adaptability, and conviction. He expresses excitement about AI’s potential in healthcare, drug discovery, and climate science, with NVIDIA’s Earth2 system and Clara medical platform poised to make significant contributions.

NVIDIA focuses on developing agile projects that aim to achieve seemingly impossible applications, anticipating their importance in the future. While some projects may not currently be successful, the company remains confident in their potential. Huang expresses confidence in the progress of autonomous driving and believes that a robotic foundation model will be discovered, allowing control of robotic systems through natural language. The path to developing these models involves discovery and exploration, with NVIDIA leveraging its expertise in learning structure from unstructured information, including language, images, and videos. Significant progress in robotics is expected within the next 5-10 years, with Google’s Palmy, utilizing transformer architecture, seen as a step towards this advancement. NVIDIA has also contributed to the field of generative adversarial networks (GANs), style transfer, and high-resolution image generation, emphasizing the importance of generating content across various modalities.

Conclusion

Jensen Huang’s journey and NVIDIA’s evolution encapsulate a saga of technological foresight, strategic innovation, and philosophical insights into leadership and development. From its inception in chip design to its pioneering role in AI and accelerated computing, NVIDIA under Huang’s leadership has not only transformed itself but also significantly influenced the broader landscape of technology and AI. This journey, marked by conviction, innovation, and adaptability, offers valuable lessons and insights for both the tech industry and aspiring entrepreneurs.


Notes by: MythicNeutron