What is the dominant strategy in AI?

Much has been written about who will capture value in this new AI era. Startups or incumbents? Builders of foundational models, or builders of vertical software that leverage those models in their products?

For my part, I always thought that a vertically integrated player — that is, a firm that owns everything from the foundational model all the way up the stack to the software itself — would dominate.

This conclusion stems from Clayton Christensen’s theory of low-end disruption, which he detailed in Disruption, disintegration and the dissipation of differentiability (and expanded upon in his book The Innovator’s Solution). The theory is simple. In markets that are technically demanding (or nascent), an integrated approach will initially win, because integration enables firms to produce superior products.

Over time, modularized products get to be “good enough” and the dominant strategy shifts from vertical to horizontal. But it can take awhile! Just look at Intel and its decades of vertically integrated dominance.

This theory is famously flawed — Christensen predicted Apple’s demise again and again to no avail — but it still has a lot of explanatory power in the B2B world. And given the speed at which OpenAI achieved dominance, I thought perhaps we were witnessing the next, great vertically integrated behemoth taking shape.

But it seems the dominant strategy may have already shifted.

Just about two months ago, Meta open sourced the code to LLaMA, its large language model. One week later, the weights were leaked as well, effectively open sourcing the inner workings of the model itself. And since then, the technical advances achieved by the open source community have been nothing short of astounding, putting both OpenAI and Google on their heels.

Last week, an internal Google memo titled “We Have No Moat, And Neither Does OpenAI” was leaked. And according to the author, the technical advances from the open source movement have fundamentally changed the competitive landscape.

Specifically, the benefits to building your own foundational models have all but disappeared, due to two shifts:

• 1) Small models can now outperform large models for a fraction of the cost. “LoRA updates are very cheap to produce (~$100) for the most popular model sizes. This means that almost anyone with an idea can generate one and distribute it. Training times under a day are the norm. At that pace, it doesn’t take long before the cumulative effect of all of these fine-tunings overcomes starting off at a size disadvantage. Indeed, in terms of engineer-hours, the pace of improvement from these models vastly outstrips what we can do with our largest variants, and the best are already largely indistinguishable from ChatGPT.”
• 2) Data quality scales better than data quantity. “Many of these projects are saving time by training on small, highly curated datasets. This suggests there is some flexibility in data scaling laws. The existence of such datasets follows from the line of thinking in Data Doesn’t Do What You Think, and they are rapidly becoming the standard way to do training outside Google. These datasets are built using synthetic methods (e.g. filtering the best responses from an existing model) and scavenging from other projects, neither of which is dominant at Google.”

It seems the era of vertical integration in AI may already be over.

Here’s another analogy for you. If OpenAI is Intel, the open source community is the Taiwan Semiconductor Manufacturing Co (TSMC). Intel benefited from vertical integration for decades. But TSMC was able to wedge into the market by realizing that there were many chip designers targeting niche markets who couldn’t afford to build their own fab. In working with all sorts of chip designers and suppliers, and focusing exclusively on manufacturing excellence, TSMC eventually surpassed Intel’s manufacturing capabilities, fundamentally disrupting Intel’s business.

The open source community, in this somewhat tortured analogy, achieved in weeks what TSMC achieved in years. What just happened is like if TSMCs manufacturing capabilities just got open sourced and made available to everyone for free. (The benefit of playing with bits instead of atoms!).

In other words, it’s as if we’re all chip designers now with our own state-of-the-art fabs running in the cloud.

So what are you going to build?