Programmable Shading to RTX:
Jensen Huang reflected on NVIDIA’s journey from introducing the world’s first programmable shading GPU two decades ago to launching RTX at SIGGRAPH 2018. RTX aimed to make real-time ray tracing, primarily an offline film rendering technique, a reality. Achieving this required NVIDIA to completely redesign the GPU, incorporate ray tracing accelerators, reimagine rendering software, and overhaul established algorithms.

Combining Graphics and AI:
To realize the RTX vision, NVIDIA had to bridge computer graphics and artificial intelligence. The debut demonstration of the first RTX GPU, named Turing, showcased this blend. This demo, titled “Star Wars Reflection,” utilized both rasterization and ray tracing, with DLSS super resolution enhancing the output to 4K.

Progress in Five Years:
Within five years, significant strides were made. The newer demo, Racer RTX, boasted 250 million polygons, a monumental increase from the previous 2.5 million. The entire scene utilized path tracing without rasterization, rendered at 1080p and then upscaled to 4K using DLSS. This progress highlights the immense advancement in computer graphics and the rewards of NVIDIA’s risk-taking.

Reinventing the GPU for AI:
While NVIDIA was revolutionizing computer graphics with AI, they also transformed the GPU’s architecture to better cater to AI tasks. From the Turing GPU, which many remembered from five years prior, GPUs evolved into more complex entities. The third generation, the Hopper GPU, is a testament to this evolution, with one trillion transistors, 35,000 components, and manufactured with precision akin to electric cars.

The Future: Generative AI Era:
After dedicating over a decade to AI, NVIDIA heralded the onset of the generative AI era, emphasizing its significance and potential impact.

Jensen Huang’s discussion underscores the rapid evolution of GPUs, the integration of AI and computer graphics, and NVIDIA’s continued dedication to pushing technological boundaries.

AI’s iPhone Moment: Huang draws a parallel to the iPhone’s revolutionary integration of technology, suggesting that we’ve reached a similar pivotal moment with AI. The convergence of different AI technologies has made it ubiquitous in various applications.

Transformers and Data: The transformative capability of the transformer model is emphasized. It can learn from vast amounts of data, understanding patterns and relationships, thus enabling the representation of numerous structured items like language, audio, 3D models, DNA, and more in a mathematical form.

Generative AI Era: The melding of large language models with generative models signifies the dawn of the Generative AI era. This has led to extensive research, funding, and global attention because of the potential to produce intelligent information, amplifying human capabilities.

AI Applications: Huang provides examples of groundbreaking applications, including Adobe Firefly’s outpainting, MOVE Ai’s motion capture, Vizcom’s sketch-to-image conversion, and Wonder Dynamics’ actor replacement with CG characters. He underscores the growing number of generative AI startups, suggesting a significant trend.

Generative AI Significance: The profound impact of generative AI is tied to its democratizing potential. Human language becomes a new programming paradigm, making everyone potential “programmers.” Large language models emerge as the new computing platform, with generative AI as the ultimate application, spurring unprecedented excitement.

Evolution of Computing: Huang outlines the emergence of a new computing platform, similar in significance to milestones like the PC and the Internet. Unique to this new platform, however, is its capacity to benefit prior computing systems. Huang envisions generative AI integration into familiar tools, enhancing productivity.

Grace Hopper Processor: Addressing the need for reinventing computer systems, NVIDIA introduces the Grace Hopper processor, tailored for the generative AI era. Designed for large-scale cloud operations, it replaces generic CPUs with specialized components, boasting impressive features like the world’s fastest memory, HBM3e, and notable link speeds.

Scalability of Systems: Huang demonstrates how Grace Hopper can be connected to create even more powerful systems, emphasizing its scalability. He unveils a massive GPU system, humorously hinting at its incredible capabilities.
By highlighting these advancements and the potential of generative AI, Huang paints a picture of a future where AI not only aids in various tasks but reshapes the very fabric of computing.

Evolution of AI Models:
Jensen Huang notes that what were once considered frontier AI models like GPT-3, GPT-4, and Llama have now become mainstream within just a few years. These models, once complex and costly to train, have paved the way for a broad range of applications.

Transition from General-Purpose to Accelerated Computing:
Historically, general-purpose computing, as exemplified by systems like the IBM System/360, dominated the landscape for over 60 years. Huang asserts that this traditional approach is giving way to accelerated computing tailored for AI applications.

Language Models as Intermediaries:
In the near future, users will often first interact with large language models when using any digital application or database. These models ascertain users’ intentions based on context and facilitate smart searches and queries, streamlining user experiences.

Model Integration for Better Efficiency:
Huang presents the combination of the Llama 2 language model, a vector database, and a generative model called Stable Diffusion XL as state-of-the-art. Together, these components efficiently guide user-computer interactions.

Cost-Efficiency of Accelerated Computing:
An ISO budget comparison reveals that using accelerated computing, specifically with Grace Hopper, yields far greater energy and cost efficiency than traditional CPUs. For the same budget, Grace Hopper utilizes 20x less energy and offers 20x the performance. Similarly, for an equal workload, accelerated computing drastically reduces both cost and power consumption.

Future of Data Centers:
Given its clear advantages, Huang reasons that accelerated computing is the undeniable future for data centers globally. Emphasizing the economic wisdom of “the more you buy, the more you save”, he identifies this shift towards accelerated computing as the most pivotal takeaway from his talk.

Backdrop for Future Innovations:
Huang hints at the union of real-time ray tracing and AI as the future of computer graphics. Setting the stage with accelerated computing and generative AI, he teases the introduction of new, groundbreaking developments.

Interdependence of Graphics and AI:
Graphics and artificial intelligence (AI) are closely intertwined. For AI to mirror human common sense, it needs to understand physical principles such as object permanence and the effects of gravity, which are essentially graphical concepts. Most current AI models lack this innate human understanding.

Omniverse: A Virtual Playground for AI:
NVIDIA’s Omniverse offers a virtual, physics-simulated world for AI. Here, AI can learn to perceive its environment and understand the consequences of its actions. By creating these simulated realities, AI can achieve a sense of physical grounding.

Applications and Industries:
The implications of AI training in virtual worlds like Omniverse are vast. Major industries, especially those grounded in physical principles, can benefit. One primary application is to train AI in virtual settings before real-world application.

AI’s Role in Creating Virtual Realities:
Beyond learning in virtual worlds, AI can assist in generating these worlds. For instance, users can query a large language model to find specific objects (like a car) and then instruct the AI to design a fitting environment for that object. Omniverse can then integrate and render this design.

Case Study: WPP and BYD:
WPP, the world’s leading ad agency, and BYD, a top electric vehicle manufacturer, utilize Omniverse and generative AI for their projects. WPP designs advanced car configurators for BYD, using Omniverse to create a highly detailed digital twin of a car. This allows for real-time rendering of different car configurations, streamlining the design and marketing processes.

Generative AI in Design and Industries:
Generative AI models are revolutionizing various fields. For example, NVIDIA’s Hopper GPU, which boasts high performance, was designed with the aid of AI and generative models. These models not only enhance design capabilities but also significantly boost team productivity.

The Future of AI in Software Development:
AI is shaping the future of software and computing. It’s expected to be integrated into every application and run on every computing platform, from data centers to personal devices. However, current complexities in software stacks and integration have limited its widespread deployment outside of cloud environments.

Democratizing AI:
To truly harness AI’s potential, it must be democratized – made accessible and usable for all. This requires optimized, unified software stacks that can operate on virtually any device. NVIDIA aims to simplify these stacks and expand AI capabilities beyond the cloud, bringing its power to every computing touchpoint.

Hugging Face’s Dominance:
Jensen Huang emphasizes the role of Hugging Face in the AI landscape, identifying it as the world’s largest AI community. With its vast user base of 50,000 companies and 2 million individual users, Hugging Face boasts 275,000 models and 50,000 datasets. Consequently, anyone looking to share an AI model within the community typically opts for Hugging Face.

New Hugging Face-NVIDIA Partnership:
A significant announcement is that Hugging Face will introduce a new service allowing users to train directly on NVIDIA’s DGX Cloud. This cloud solution is acknowledged as the optimal way to train models, and is expanding its reach to platforms like Azure, OCI, Oracle Cloud, and GCP. The seamless integration between Hugging Face’s portal and NVIDIA’s DGX Cloud is set to connect the world’s most extensive AI community with state-of-the-art AI training infrastructure.

Introducing NVIDIA AI Workbench:
The NVIDIA AI Workbench is another significant unveiling. This suite of tools facilitates users in automatically assembling required runtimes, libraries, and acceleration components to fine-tune and optimize large language models. Moreover, it ensures ease in deployment, allowing users to target various platforms, from PCs and workstations to data centers. With a single click, users can migrate projects across these platforms, simplifying the entire AI development process.

Workbench in Practice:
Highlighting the practicality of the NVIDIA AI Workbench, an example demonstrated its capability on a GeForce RTX 4090 laptop. As the AI project scaled, it needed more computational resources, prompting a shift to a workstation with four NVIDIA RTX 6000 Ada-generation GPUs. The Workbench effortlessly handled project environment creation, integrating tools like Jupyter for a seamless experience. Through the Workbench, users can fine-tune models, as evidenced by adjusting a model with images of “Toy Jensen” for improved accuracy. This scaling and customizing flexibility extends even to larger models, like the Llama 2 70B, reinforcing the Workbench’s versatility.

User Accessibility:
Underlining the Workbench’s user-friendliness, Huang states that anyone can use it by simply visiting NVIDIA’s website and downloading the tool. Its design helps creators set up necessary libraries and runtimes and facilitates fine-tuning of models. Additionally, users can effortlessly migrate projects across different platforms, ensuring consistent functionality and compatibility.

Personal Touch:
In a lighter note, Huang jests about his Swedish name, “Jensen,” making a playful reference to the “Toy Jensen” demonstration. This adds a personal touch to the discussion, connecting with the audience on a relatable level.

Democratization of AI:
Jensen Huang introduces the idea that everyone can now engage with generative AI, hinting at the democratization of this technology.

NVIDIA AI Enterprise:
This software is described as the operating system for modern data science and AI. It manages the workflow from data processing, which can make up to 60% of the computation work, through training and inference to deployment. NVIDIA AI Enterprise includes a vast library of 4,500 packages with 10,000 dependencies, a culmination of NVIDIA’s three decades of expertise.

Compatibility and Integration:
The NVIDIA AI Enterprise is designed to be exceptionally compatible. It works with every version of NVIDIA’s GPU, is integrated with major cloud services, and is incorporated into key operating systems like Linux and Windows. The system is also optimized for CUDA, supporting multi-GPU in multi-node environments. There’s notable collaboration with VMware, making it CUDA aware and retaining enterprise-level benefits.

AI Ecosystem & Workbench:
AI Enterprise paves the way for NVIDIA AI Workbench, supported by models from Hugging Face that are open-sourced and pre-trained. This comprehensive ecosystem promises support for virtually any application one might wish to run.

Hardware Announcements:
The Ada Lovelace GPU is introduced as the most powerful GPU by NVIDIA, intended for workstations. Workstations equipped with this GPU can handle tasks like real-time ray tracing for Omniverse and training large language models for generative AI. The new workstations are in production and available from major manufacturers.

Performance and Application:
With NVIDIA AI Enterprise on Windows 11 and WSL2, these workstations offer incredible AI capabilities. For example, they can fine-tune the 40 billion parameter GPT-3 model in about 15 hours on almost a billion tokens. This speed and power can be leveraged for a variety of applications, from data-driven software management to generative AI tasks.

Server Solutions:
Alongside workstation offerings, NVIDIA also introduces powerful server configurations. These servers, equipped with the L40S Ada Lovelace GPUs, aren’t meant for training the largest models like GPT-4 or GPT-5 but are optimized for mainstream models available today. Fine-tuning tasks on these servers are faster, with an L40S performing 1.5 times faster than the previous generation A100 GPU.

Conclusion:
Jensen underscores the sentiment of “generative AI for everyone” made possible through tools and platforms like Hugging Face, NVIDIA AI Workbench, and NVIDIA AI Enterprise, combined with their new enterprise-grade hardware offerings.

OpenUSD Introduction:
Jensen Huang emphasized the importance and potential of OpenUSD at SIGGRAPH 2023. OpenUSD stands as a groundbreaking framework offering a universal interchange for constructing 3D worlds. It covers various aspects including description, compositing, simulation, and collaboration on 3D projects. By aiming to unify the world under a single standard 3D interchange, its potential is likened to what HTML accomplished for the 2D web.

Challenges of 3D Workflow:
The 3D production pipeline is intricate, comprising numerous specialized and interrelated tasks like modeling, texturing, animation, and scene composition. Tools to handle these tasks are largely developed by different companies, resulting in incompatibilities. This necessitates frequent data conversion between tools, leading to a serialized workflow prone to errors and inefficiencies. Such complexities contribute to the high costs and lengthy timelines associated with creating high-quality 3D content.

OpenUSD’s Vision:
One of the primary objectives of OpenUSD is to centralize data. By ensuring every tool is natively compatible with USD, data becomes the focal point, promoting a parallel working structure. This “spoke and hub” model reduces the need for data interchange and conversion, streamlining the overall process. This revolutionary approach has garnered significant industry attention, with 50 tools already natively compatible and a growing community of contributors and adopters spanning diverse sectors.

Omniverse and OpenUSD:
Five years prior, in collaboration with Pixar, Omniverse was developed with OpenUSD as its foundational core. Designed as a comprehensive platform constructed entirely with USD in mind, Omniverse doesn’t function as a tool itself but rather serves as a connector of tools. Its aim is not final production but enabling efficient collaboration, interchange, and real-time shared work, further underscoring the transformative potential of OpenUSD.

OpenUSD Vision: Jensen Huang introduced the vision of OpenUSD: a single dataset that is seamlessly integrated across a variety of tools, including Adobe Stager, Houdini, Maya, Omniverse, Blender, RenderMan, and Unreal Engine. Although the rendering quality varies from tool to tool, the foundational dataset is consistent and usable by every tool.

USD Evolution and NVIDIA’s Involvement: Over the past five years, NVIDIA has heavily invested in USD. They’ve worked on extending USD for real-time, physics-based systems meant for industrial applications. This includes bringing RTX, introducing schemas for physics (both real-time and offline), integrating CAD, understanding geospatial data, incorporating AI frameworks for generative AI like deep search, and more. NVIDIA has ensured that USD assets are physics-accurate and hyperscale-ready, making them suitable for applications like robotics.

Omniverse Platform: Omniverse, a platform developed in collaboration with the industry, facilitates the implementation of the OpenUSD vision. Jensen emphasized that Omniverse is not just a tool but a platform for tools. He revealed that many reference applications within Omniverse are open-sourced. One standout application is Isaac Sim, a virtual environment for training robots. It’s so physically accurate that robots can theoretically learn to be a robot within Omniverse, minimizing the simulation-to-reality gap.

Omniverse Availability: Omniverse is being made more widely available. While enthusiasts and designers can download Omniverse for PCs or workstations, enterprises have the option to license it. Additionally, NVIDIA is launching Omniverse Cloud, expanding its accessibility and capabilities.

API Introductions: Several new APIs are being introduced at SIGGRAPH. One such API, the Avatar Cloud Engine (ACE), allows for voice recognition and animation of facial movements based on speech. Another key API, runUSD, lets users send USD data to the cloud and receive beautifully rendered and interactive USD content.

ChatUSD – Generative AI for USD Workflows: ChatUSD is an innovative tool that acts as a co-pilot for USD development. Built using NVIDIA AI Workbench and the NeMo framework, ChatUSD simplifies complex tasks, such as scene scaling and organization. It can provide general knowledge, generate code for repetitive tasks, and help users build intricate virtual scenes faster. Available on the Omniverse Cloud, ChatUSD aims to significantly boost productivity in the virtual world creation process.
In summary, Jensen Huang’s presentation shed light on the impressive strides NVIDIA has taken with OpenUSD and Omniverse, revealing the numerous innovations and tools they’ve developed to improve the world of 3D content and virtual reality.

Opportunity in Heavy Industries:
Jensen Huang highlights the immense opportunity that IT, software, and AI have in transforming the world’s heavy industries. There exists a potential to mitigate waste in the massive $50 trillion industry, which encompasses a large number of factories, including the new ones for electric vehicles (EVs) and chips.

The Desire for Digital Transformation:
Despite these industries’ inherent physical nature, there is a keen interest in adopting digital practices. They want the advantages offered by the digital world but have been traditionally restrained due to their reliance on physical processes.

Omniverse & Generative AI:
Omniverse, coupled with artificial and generative AI, offers a solution to help these industries digitize their workflow. Much like how different departments collaborate in the creation of animated projects, heavy industries can utilize a similar interconnected approach for their processes, from design to operation.

OpenUSD’s Vision:
OpenUSD has the potential to integrate and digitize the entire flow of heavy industries. By enhancing it with real-time and physics simulation capabilities, and connecting every tool to Omniverse, industries can undergo a complete digital transformation. This approach can lead to enhanced productivity, reduced energy consumption, decreased waste, and minimized mistakes in the digital realm before anything physical is produced.

Real-world Applications:
Companies like Mercedes and BMW are already leveraging Omniverse to digitalize their manufacturing processes, simulate autonomous vehicles, and even plan new factories. The complexity of planning, designing, and operating a factory can be streamlined through digital simulations, paving the way for software-defined and robotic factories.

Robots & AI:
While many robots today are not entirely autonomous and can be expensive to program, the vision is for AI to become central to self-programming these machines. Techman’s use of Omniverse for testing and simulating cobots (collaborative robots) and automating optical inspection exemplifies this direction.

Breaking Silos with Omniverse:
Hexagon’s use of Omniverse showcases its capability to unify different teams that were previously siloed due to incompatible tools. This integration not only enhances productivity but also addresses organizational challenges.

The Future of Robot Programming:
Jensen envisions a future where robot programming is as simple as giving prompts and showing a few examples. Generative AI models will then be able to generalize from those examples and perform the task.

Digital Twins & Real-time Simulation:
Omniverse can be employed to create “digital twins”, accurate digital replicas of physical entities. NVIDIA, for instance, is working on a digital twin of Earth’s climate system, while Deutsche Bahn is creating one for their entire railway network. These simulations demand real-time operation, emphasizing Omniverse’s importance in the future of digital transformation.

Collaborative AI Design:
Jensen Huang highlighted the synergy between human designers and generative AI models from various applications and companies. Together, they can expedite industrial digitalization.

NVIDIA Omniverse and Generative AI:
Omniverse, powered by NVIDIA, combined with generative AI, offers a transformative toolset for digital industrial planning. For instance, a 2D CAD floor plan can be converted into a 3D OpenUSD model with the SyncTwin Omniverse extension. The space can then be populated with SimReady OpenUSD assets, made searchable via AI. This workflow can also generate accurate lighting with Blender GPT, realistic flooring with Adobe Firefly, and an HDRI Skydome using Blockade Labs. Users can view the virtual space alongside their existing factory using Cesium, then share the concept via Omniverse Cloud GDN.

From 2D to Spatial 3D:
Emphasizing the technological leap, Huang pointed out that projects now transform from basic 2D PowerPoint slides to interactive spatial 3D models. The transition from “PowerPoint to a virtual factory” underscores the revolutionary capabilities introduced by these tools.

Importance of OpenUSD and Omniverse:
OpenUSD and NVIDIA’s Omniverse play foundational roles in this new era of digital industrialization. Huang expressed NVIDIA’s satisfaction with adopting OpenUSD and their ongoing efforts to integrate it into real-time, physics-based applications.

Alliance for OpenUSD:
Further accelerating the OpenUSD momentum, an alliance has been established featuring major companies like Pixar, Apple, Adobe, Autodesk, and NVIDIA. Their mission is to advance and standardize OpenUSD, boosting its integration into various industries.

NVIDIA’s Contributions at SIGGRAPH 2023:
SIGGRAPH 2023 marks a pivotal moment in computing history for NVIDIA. Huang delineated the transition to a novel computing model, one which hasn’t seen a fundamental redesign in decades. This includes a change in architecture, from the processor to the data center and middleware, to accommodate the rise of AI algorithms and their ensuing applications. For this new era, NVIDIA introduced the Grace Hopper processor, termed GH200, as well as NVIDIA AI Workbench for engaging generative AI. NVIDIA Omniverse also saw significant updates, particularly in support for OpenUSD.

Legacy and Importance of Computer Graphics:
Huang paid homage to the field of computer graphics, emphasizing its importance to NVIDIA. The company has dedicated three decades to advancing this sector, reflecting its deep commitment. He credited the SIGGRAPH community for enabling innovations like AI, Omniverse, and OpenUSD.

Closing Sentiments:
Wrapping up, Huang left the audience with a whimsical note about the cost-saving advantages of accelerated computing and generative AI.