Jensen Huang Channels ‘Doctor Strange’ to Make Physical AI a Reality

• Jensen Huang has unveiled a platform called Cosmos to simulate scenarios to train real-world robots.
• Huang likened it to Marvel superhero Doctor Strange simulating millions of versions of the future.
• The Nvidia boss said at CES that physical AI is the “next frontier” of artificial intelligence.

In “Avengers: Infinity War,” Marvel superhero Doctor Strange looks into the future to see over 14 million different outcomes of the galactic battle against supervillain Thanos. Jensen Huang thinks it’s the kind of power needed to reach “the next frontier of AI.”

In a keynote address at CES in Las Vegas on Monday, the Nvidia CEO introduced Cosmos, a platform that aims to make “physical AI” a reality by simulating endless real-world scenarios for robots and autonomous vehicles to study and gain a deeper understanding of their environment.

According to Huang, the path to this next frontier — in which autonomous hardware becomes a common sight in daily life — has been limited until now because of data availability. As he put it, “Physical world data is costly to capture, curate, and label.”

That’s where Nvidia Cosmos comes in, for Huang at least. “You could have it generate multiple physically-based, physically plausible scenarios of the future,” he told the Las Vegas audience. “Basically, do a Doctor Strange.”

Nvidia’s next frontier is coming

Here’s how it works. Cosmos ingests text, image, or video prompts to generate videos with virtual renderings of real-world environments, lighting, and more.

Developers of robots and autonomous vehicles can then use these virtual creations to provide their technology with synthetic data for reinforcement learning — a research technique used to teach AI models — as well as test and validate the models behind the physical AI.

According to an Nvidia blog post, Cosmos can also be used along with Omniverse, the company’s platform for creating 3D graphics and metaverses, to “generate every possible future outcome an AI model could take to help it select the best and most accurate path.”

Cosmos itself starts with a strong, foundational understanding of real-world environments. It has been trained on 20 million hours of video focusing on everything from humans walking and “dynamic nature” to camera movements, Nvidia said.

If robots and autonomous vehicles are to become a widespread reality, as other industry leaders like Elon Musk think, they’ll need a highly sophisticated understanding of these kinds of scenarios.

“It’s really about teaching the AI, not about generating creative content, but teaching the AI to understand the physical world,” Huang said.

There’s a good reason Huang is talking up physical AI. While Nvidia has grown by roughly $3.3 trillion since the start of the generative AI boom, thanks to high demand for its chips needed to train AI models, the business isn’t completely free of threats.

Some of Nvidia’s Big Tech customers, such as Amazon and Google, are developing chips of their own to reduce their dependence on Nvidia. The company made 87.7% of its $35.1 billion revenue last quarter from its chip and data center business.

As Business Insider’s Emma Cosgrove also notes, the semiconductor industry has historically been brutal. Companies typically experience boom and bust cycles as interest in niche chips can come in waves. There is an incentive then for Huang to diversify Nvidia’s sources of income.

Time will tell if Cosmos can offer the path forward to Nvidia’s next frontier. Development of robots that can navigate complex world environments has taken shape slowly, despite companies like Google, Boston Dynamics and Figure AI deploying increasing amounts of capital on developing these technologies.

Huang himself noted during his CES keynote that he expects autonomous vehicles to represent the “first multi-trillion dollar robotics industry.”

With autonomous cars already on the road in certain locations from companies like Waymo and Cruise, this could be the case. During CES, Huang shared that Nvidia had struck a new partnership with Toyota to help power its autonomous vehicle ambitions.

Getting to a world where robots roam freely among humans will take considerably more effort, however. Huang will hope that Cosmos starts to provide the superpowers needed to pull off such a feat.