- AI leaders are rethinking data-heavy training for large language models.
- Traditional models scale linearly with data, but this approach may hit a dead end.
- Smaller, more efficient models and new training methods are gaining industry support.
For years, tech companies like OpenAI, Meta, and Google have focused on amassing tons of data, assuming that more training material would lead to smarter, more powerful models.
Now, AI leaders are rethinking the conventional wisdom about how to train large language models.
The focus on training data arises from research showing that transformers, the neural networks behind large language models, have a one-to-one relationship with the amount of data they’re given. Transformer models “scale quite linearly with the amount of data and compute they’re given,” Alex Voica, a consultant at the Mohamed bin Zayed University of Artificial Intelligence, previously told Business Insider.
However, executives are starting to worry that this approach can only go so far, and they’re exploring alternatives for advancing the technology.
The money going into AI has largely hung on the idea that this scaling law “would hold,” Scale AI CEO Alexandr Wang said at the Cerebral Valley conference this week, tech newsletter Command Line reported. It’s now “the biggest question in the industry.”
Some executives say the problem with the approach is that it’s a little mindless. “It’s definitely true that if you throw more compute at the model, if you make the model bigger, it’ll get better,” Aidan Gomez, the CEO of Cohere, said on the 20VC podcast. “It’s kind of like it’s the most trustworthy way to improve models. It’s also the dumbest.”
Gomez advocates smaller, more efficient models, which are gaining industry support for being cost-effective.
Others worry this approach won’t reach artificial general intelligence — a theoretical form of AI that matches or surpasses human intelligence — even though many of the world’s largest AI companies are banking on it.
Large language models are trained simply to “predict the next token, given the previous set of tokens,” Richard Socher, a former Salesforce executive and CEO of AI-powered search engine You.com, told Business Insider. The more effective way to train them is to “force” these models to translate questions into computer code and generate an answer based on the output of that code, he said. This will reduce hallucinations in quantitative questions and enhance their abilities.
Not all industry leaders are sold that AI has hit a scaling wall, however.
“Despite what other people think, we’re not at diminishing marginal returns on scale-up,” Microsoft chief technology officer Kevin Scott said in July in an interview with Sequoia Capital’s Training Data podcast.
Companies like OpenAI are also seeking to improve on existing LLMs.
OpenAI’s o1, released in September, still relies on the token prediction mechanism Socher refers to. Still, the model is specialized to better handle quantitative questions, including areas like coding and mathematics — compared to ChatGPT, which is considered a more general-purpose model.
Part of the difference between o1 and ChatGPT is that o1 spends more time on inference or “thinking” before it answers a question.
“To summarize, if we were to anthropomorphize, gpt-4 is like your super know-it-all friend who when you ask them a question starts talking stream-of-consciousness, forcing you to sift through what they’re saying for the gems,” Waleed Kadous, a former engineer lead at Uber and former Google principal software engineer, wrote in a blog post. “o1 is more like the friend who listens carefully to what you have to say, scratches their chin for a few moments, and then shares a couple of sentences that hit the nail on the head.”
One of o1’s trade-offs, however, is that it requires much more computational power, making it slower and costlier, according to Artificial Analysis, an independent AI benchmarking website.
