AI Is Becoming a Systems Problem

For the last few years, the model has been the story. Bigger models, better models, and new capabilities arriving faster than most of us expected. That’s been fascinating to watch.

But lately I’ve found myself thinking less about the model itself and more about everything around it. Where does the work happen? What stays on the device, and what goes to the cloud? What happens when memory, power or connectivity are limited? And who manages context when several parts of a system want to use the same underlying capabilities?

These can sound like implementation details. I don’t think they are. They may end up determining which products actually work.

This feels familiar

I’ve spent much of my career building computing platforms, and I’ve seen a version of this movie before. A new technology arrives and, understandably, everyone focuses on the new capability. Then it gets good enough, and the conversation changes. It becomes less about the capability itself and more about what you can build with it.

Mobile went through this. Connectivity did too. The smartphone wasn’t the result of one breakthrough. It came from getting a lot of things to work together: compute, radios, sensors, multimedia, operating systems, applications, power management and eventually cloud services.

None of those pieces could really be designed in isolation.

The system became the product.

I wonder if we’re approaching a similar point again.

Where should intelligence live?

One question I keep coming back to is where intelligence should live. In the cloud? On the device? Somewhere in between? The answer, frustratingly, is probably all three.

The cloud gives you scale. The device gives you immediacy, privacy and local context. Each comes with different constraints. So the interesting question isn’t which one wins. It’s how the system decides.

Once you start pulling on that thread, you quickly get into architecture. Latency affects experience. Memory and compute affect what can run locally. Power affects how often it can run. Privacy affects where data can go. Cost affects what can scale.

These aren’t independent decisions. Change one and you often move another. I’ve spent enough time making these kinds of tradeoffs to know there is rarely a perfect architecture.

There are choices. And consequences.

Something else may change

Today, we mostly think about intelligence as something an application invokes. I’m curious whether that remains true.

If many applications need models, context, memory, permissions and access to specialized hardware, does it make sense for each of them to solve those problems independently? Probably not. Some of that responsibility may move into the platform itself.

And that’s where things get interesting.

Operating systems have always managed shared resources: compute, memory, storage, networking and power. What happens when they also have to manage intelligence?

I don’t know exactly what that architecture looks like yet. But I suspect it will matter.

There’s an organizational problem hiding in here too

Large systems have an annoying property: the hardest problems tend to live between teams. The model can be owned by one group, runtime by another, silicon by another, cloud by another, and product by someone else.

The user doesn’t care. They experience one system.

That’s something engineering organizations sometimes learn the hard way.

The architecture can be distributed. Accountability can’t be.

As these systems become more complex, I think this becomes as much a leadership question as an architecture question. Who is looking across the whole system? Who makes the tradeoff when improving one layer makes another worse? Who owns the experience rather than the component?

Those questions aren’t new. Which is partly why this moment feels familiar.

What I’m curious about now

Models will keep improving. I’m sure we’ll continue to be surprised by what they can do. But I’m increasingly watching something else: what gets built around them.

I’m watching how intelligence gets divided between device and cloud, what moves into the operating system, and how hardware and software evolve together. I’m also curious whether the companies that build the most useful products will necessarily have the best individual technology—or whether they’ll simply be better at making the whole system work.

I have a hunch it will be the latter.

The last few years have been about asking:

What can these models do?

I’m more interested now in:

What can we build when we stop thinking about the model in isolation?

That feels like the beginning of a different conversation.