Does the model fit in memory? That is now the wrong question

There was a moment, over the last two or three years, when Apple seemed to have almost by chance the perfect machine for running language models locally, and the reason was simple, unified memory. While the PC world kept living with the separation between system RAM and GPU VRAM, Macs with Ultra chips could make enormous amounts of memory available to the GPU compared with ordinary consumer cards. For anyone who wanted to load models with tens or hundreds of billions of parameters it was a huge advantage, and for a while the main question was almost trivial: does the model fit in memory or not?

Today that question is no longer enough. With the arrival of the M5 Ultra, of NVIDIA's DGX Spark and Station and of the first Chinese experiments like the Xiaomi AI Cube, the problem of local inference is changing in nature. It is no longer only about how much memory you can put inside a machine, it is about how fast that memory can be used, how much compute is available and, above all, how well hardware and software have been designed to work together. And this is probably the most interesting point of the whole new generation of AI hardware.

Apple's historical advantage

The new M5 Ultra continues to be an impressive machine when it comes to memory, because you can reach up to 512 GB of unified memory and about 1.2 TB/s of bandwidth. To put the number in perspective, the M3 Ultra stopped at around 819 GB/s: it is therefore an important jump, even if not as revolutionary as some might have expected.

For LLM inference, though, bandwidth matters a great deal. When a model generates an answer token after token, especially in a single session, much of the work consists of continuously reading the weights from memory, and this is one of the reasons why Ultra Macs have always managed to get interesting results despite not having GPUs comparable, in raw power, to NVIDIA's big accelerators. In this kind of work Apple still has a characteristic that is very hard to ignore, that is a great deal of very high bandwidth memory available in a single machine. But the problem is that modern inference is not made of generation alone.

The new bottleneck is called prefill

Before a model starts writing an answer it has to read the prompt. If we give it a ten line question, no big deal, but if we ask it to analyze a repository, a document of hundreds of pages or hundreds of thousands of tokens of context, the picture changes completely. That phase is called prefill, and this is where compute starts to carry an enormous weight. Simplifying a lot, token generation is often limited by memory bandwidth, while prefill depends much more on raw compute.

This becomes even more important when we stop thinking of an LLM as a chatbot and start thinking of it as an agent. A coding agent does not read a prompt once and then produce five hundred tokens, it reads files, analyzes code, calls tools, receives new results, reads more files, runs another prefill, produces code, checks the output and starts again. The ratio between generation and compute changes completely, and this is where you understand why looking only at memory bandwidth today risks being misleading.

The M5 Ultra is not simply a faster M3 Ultra

Apple seems to have understood the problem perfectly, because with the M5 Ultra it added neural accelerators directly inside the GPU and claims a very significant increase in AI performance compared with the M3 Ultra. On paper, then, it would be wrong to dismiss the M5 Ultra as a simple bandwidth upgrade. The real question is another one, that is how much of this new power can actually be used by the software stacks we use today for local inference. MLX, llama.cpp, LM Studio and the other runtimes will have to show how well they can exploit the new architecture, and it is probably on this ground that the real value of the machine will be decided, because a workstation costing more than ten thousand euros is not judged only by whether it can load an enormous model, but by whether it can run it at a useful speed.

The terabyte of RAM paradox

One of the most frequent criticisms of the new Mac Studio concerns exactly the absence of a 1 TB configuration. At first sight it looks like an obvious limitation, but the interesting question is what that terabyte would actually be for. Imagine being able to load an enormous Mixture of Experts with hundreds of gigabytes of weights: the fact that the model fits in memory does not automatically mean it can be used well, because if compute does not grow in proportion we end up with a machine able to host a gigantic model but forced to run it too slowly. It is a bit like building a twenty lane motorway and then putting a toll booth every hundred meters. This is why Apple's choice to stop at 512 GB might be less absurd than it seems, and perhaps the optimal point of the current architecture is exactly there.

NVIDIA is building a different kind of machine

The DGX Spark starts from an almost opposite philosophy, because it has much less memory than an M5 Ultra and clearly lower bandwidth, but it is built around Blackwell Tensor Cores and a software ecosystem developed over years for machine learning. CUDA, TensorRT, vLLM, SGLang and the model specific kernels are all designed for one very precise thing, making neural networks work. A Mac Studio has to be at the same time a video workstation, a development computer, an audio machine, a general purpose desktop and an AI platform, while the DGX Spark is not, it is a box built for AI.

This difference becomes even clearer with the DGX Station, where NVIDIA combines enormous amounts of memory with very high bandwidth HBM and an amount of compute that completely changes the scale of the problem. At that point we are no longer talking about a machine on which "you can run LLMs", we are talking about a personal AI server, and the difference is not only semantic.

Local AI is becoming multi user

There is then another thing that is often underestimated. The future of local inference is probably not a single person opening a chatbot on their own computer, because a truly powerful machine can serve several users, several agents and several applications at the same time. A team of developers could have a local coding agent, a company could have internal documents analyzed without sending them to cloud services, a research system could digest hundreds of thousands of tokens in a few seconds and an agent could stay operational twenty four hours a day. At that point the interesting metric is no longer just the number of tokens per second in a chat, but the overall throughput of the system, and it is an important change of perspective.

And then comes Xiaomi

And this is where the story becomes even more interesting, because while Apple and NVIDIA are evolving products that already exist, Xiaomi seems to want to build the whole stack from scratch. In recent months the Chinese company has started to show its AI strategy clearly: on one side there is MiMo, the family of proprietary foundation models, on the other there are the XRING chips, and in between hardware specifically designed for local inference is starting to appear.

The most interesting prototype is the Xiaomi AI Cube, a machine that combines several XRING chips, unified memory and a dedicated AI accelerator called O100. The interesting aspect is not even the product itself, which for now still has everything to prove, but the way Xiaomi is building the system: model, quantization, speculative decoding, runtime, chip, hardware, operating system and devices, all under the same roof.

The real advantage could be co-design

This is probably the most important step, because the future of local AI could reward those who manage to design model and hardware at the same time. Xiaomi can optimize MiMo knowing exactly which accelerators it will have to run on, can design the accelerator knowing how its own models are structured, can modify the runtime, work on the quantization, optimize the speculative decoding and decide how to distribute the load between smartphone, local server, home and car. It is a level of vertical integration that resembles Apple more than a traditional Android manufacturer, with one fundamental difference, that is that the starting point is not the personal computer, it is AI.

The home could become the real AI personal computer

And this is where the Xiaomi AI Cube starts to take on a different meaning. Let us think of it for a moment not as a workstation but as a home server: it stays on, it hosts MiMo, it manages the user's personal memory, it talks to smartphones and tablets, it speaks with AI glasses, it controls Xiaomi Home devices, it interacts with the car, it receives images from the cameras, it runs vision models and it manages agents. At that point we no longer have a PC with a chatbot inside, but a sort of local brain of the ecosystem, perfectly coherent with the "Human, Car, Home" strategy that Xiaomi has been pursuing for a while.

Apple was ahead, but the advantage is shrinking

Apple started with an advantage, because unified memory was perfect for a world that suddenly needed to load enormous models, and for a few years that was almost all it took. Today, however, NVIDIA, Xiaomi and other manufacturers are building systems specifically optimized for these workloads, and the competition is shifting. It is no longer enough to say you have 512 GB of memory, you have to ask how fast the prefill is, how much FP4 compute there is, how the speculative decoding works, how efficient the runtime is, how well the Mixture of Experts is exploited, how many simultaneous sessions you can serve, how much each token produced costs, how much the machine consumes and how well you can connect more than one of them. This is where the next phase of AI hardware will be played out.

We are no longer building computers with AI

The most interesting thing about all this is probably also the simplest: for years we added AI accelerators to computers, now the opposite is happening, we are starting to build computers around AI. The DGX Spark is one of the first clear examples, the DGX Station takes the concept to a much higher scale, the Xiaomi AI Cube shows that consumer manufacturers too are thinking in the same direction, and Apple, with the M5 Ultra, is trying to gradually turn the Mac Studio into something that can stay competitive in this new market.

It is not yet clear which approach will win, whether a general purpose workstation with enormous unified memory, a small highly specialized NVIDIA server or a vertical system like the one Xiaomi is trying to build. There will probably be room for all three. But one thing seems by now fairly evident: the first generation of local AI was obsessed with one question, "can I fit the model in memory?", while the next one will be obsessed with another, "once I have it in memory, can I make it reason fast enough to become genuinely useful?". And that is exactly where the most interesting part begins.