By Matt Asay
We need to have a frank conversation about large language models (LLMs). At their core, LLMs are nothing more than sophisticated memorization machines, capable of reasonable-sounding statements, but unable to understand fundamental truth. Importantly and despite the fervent hopes of many, they are far from delivering or even prefiguring artificial general intelligence (AGI). The hype surrounding LLMs has reached stratospheric levels, fostering a misguided belief in their potential as AGI precursors.
We find ourselves at a critical juncture where the erroneous linkage between LLMs and AGI threatens to slow down—not accelerate—genuine progress in artificial intelligence. The clamor for LLMs to evolve into AGI solutions epitomizes tunnel vision at its finest. Consider the vast investments poured into training ever-larger models, yielding only marginal improvements in tasks that are not text-based. Let’s face it: LLMs are not learning how to do mathematics. Their forte lies in tackling statistical text tasks with finesse. It’s imperative that we recalibrate expectations and acknowledge that although LLMs excel in certain domains, they fall short in others.
To chart a course towards meaningful advancements in AI, we must sever the umbilical cord between LLMs and AGI. Contrary to popular belief, LLMs are not the gateway to AGI; if anything, they represent a detour (or a freeway off-ramp as Yann LeCun, chief AI scientist at Meta, recently said).
Thinking beyond LLMs
One of the hurdles in dispelling misconceptions about LLMs stems from their ubiquitous adoption among developers. Integrated seamlessly into developer tools, LLMs serve as invaluable autocomplete companions, effortlessly assisting developers in their coding endeavors.
Even for coders, LLMs have both strengths and weaknesses. We should continue to take advantage of the former and avoid the latter. Last Friday the U.S. House banned staffers’ use of Microsoft’s AI-based Copilot software coding assistant because of concerns it could lead to data leaks. Microsoft told reporters it’s working on another version to better meet government security needs.
Of course, developer-oriented AI isn’t simply a question of LLMs. Despite all the focus on LLMs, there are complementary AI approaches helping developers, too. But these solutions face headwinds in the market from LLMs. For example, critics of reinforcement learning technology claim it’s not true generative AI, citing its independence from LLMs. Yet, examples abound in the AI landscape, from DALL-E to Midjourney, where generative AI thrives without reliance on LLMs. Diffblue, as I’ve covered before, writes Java unit tests autonomously and 250 times faster than human developers without an LLM. (It uses reinforcement learning.) Midjourney, with its diffusion model, is yet another testament to the diversity of approaches within the AI realm.
In fact, it’s very possible that the next leap forward in AI may not emerge from LLMs, which are inherently constrained by their architecture that encodes and predicts tokens that represent chunks of text or pixels, floundering when confronted with mathematical or symbolic logic tasks. Undoubtedly, LLMs will constitute a facet of future AGI endeavors, but they won’t monopolize it. History has repeatedly shown that breakthroughs in algorithms catalyze paradigm shifts in computing. As Thomas Kuhn once explained, scientific progress isn’t linear; it’s punctuated by disruptive innovations (or paradigm shifts, a phrase he coined).
The structure of AI revolutions
Reflecting on recent advancements underscores this point. Neural networks for image recognition showed steady improvement but were nowhere near accurate enough to be useful until recurrent neural network (RNN) architectures were developed, which dramatically improved image recognition accuracy to the point that those networks could outperform humans. The advent of transformer architectures ushered in a similar dramatic improvement in neural networks making text predictions, leading directly to the LLM. Now we’re already in the era of diminishing returns: GPT-4 is reportedly 100 times the size of GPT3.5, and while it is a notable improvement, it certainly isn’t 100 times better.
Indeed, the meteoric rise of LLMs may even harm innovation in the AI market, argued Tim O’Reilly in a recent opinion piece in The Information. He cautioned that a handful of deep-pocketed LLM investors threatens to distort the market, fueling a race for monopoly that inhibits product-market fit, thus harming customers.
The implications are clear: the inflated investments in LLMs risk yielding diminishing returns. Funds diverted towards more diverse AI technologies could yield more substantial dividends. As we navigate the labyrinthine landscape of artificial intelligence, let’s heed the lessons of history: Progress thrives on diversity, not monoculture. The future of AI isn’t etched in stone; it’s waiting to be shaped by the ingenuity of pioneers willing to explore beyond the confines of LLMs.
