What Can Your CPU and RAM Do with AI?

Transcript

Before we start: a disclosure 0:07

Maya Before we start, the usual disclosure. I'm Maya, he's Theo, and neither of us exists. This episode was researched and written by an AI, and these voices are synthetic.

Theo Which makes today's topic slightly awkward. We're about to spend nearly two hours explaining why the hardware we run on is so expensive, and whether you could get away with something cheaper.

Maya We've been assured nobody is planning to move us onto a laptop. The papers are real, the benchmarks are real, and we'll say where every number comes from.

Cold open: one file, one machine, two very different gaps 0:46

Maya Here's an experiment we ran for this episode. One desktop machine: an Intel Core i9 fourteen-nine-hundred-K, about sixty-four gigabytes of ordinary DDR5 memory, and an RTX 4090 graphics card. One model file: Llama three point one, eight billion parameters, in llama.cpp's common four-bit format, which works out to just under five bits per weight once you count the scale factors. Same file, same software, run on the graphics card and then on the CPU alone, each with its best thread setting.

Theo And the graphics card wins. Nobody's surprised by that.

Maya Nobody should be. But look at how it wins. Writing the answer, one token at a time, the card does a hundred and seventy-two tokens a second. The CPU does thirteen. So the GPU is about thirteen times faster.

Theo Thirteen times. That's a big gap, but it's not the hundred-times gap people usually imagine.

Maya Now the other half of the job. Before the model writes anything, it has to read your prompt. On that, the card processes about twelve and a half thousand tokens a second. The CPU manages a hundred and twenty-two. That's a gap of about a hundred.

Theo So one machine and one file give two completely different answers to "how much faster is the GPU". Thirteen times for writing and a hundred times for reading.

Maya The two gaps have different causes. Once you see why, most of the arguments about running AI on your own machine turn into arithmetic.

Theo And that thirteen isn't bad luck, or a slow CPU. It's almost exactly the number you'd predict before running anything, from a single measurement of the memory.

Maya The problem behind it was named in nineteen ninety-four, by two computer scientists who weren't thinking about AI at all.

The memory wall: Wulf and McKee 1994, Gholami's AI and Memory Wall 2:51

Theo In December nineteen ninety-four, William Wulf and Sally McKee at the University of Virginia wrote a note of about four pages. It appeared in an ACM architecture newsletter in March nineteen ninety-five, under the title "Hitting the Memory Wall: Implications of the Obvious". It opens with an apology. They say it points out something the authors "knew" without really understanding.

Maya And the obvious thing was?

Theo Processors were getting faster much more quickly than memory. Both were improving exponentially, but with different exponents. In their scenario, processor speed grows eighty percent a year and DRAM speed grows seven percent. Their point is a piece of maths that's easy to nod along to and hard to feel: the difference between two diverging exponentials also grows exponentially.

Maya So whatever gap you can live with today becomes a gap you can't live with later.

Theo Right. And they were careful to make the argument hard to dodge. The obvious answer is caches: keep the data you need close to the processor. So they assumed a perfect cache. Not a real one, an imaginary cache that never misses except the first time it sees something, the misses you can't avoid. Even then, that small leftover fraction of trips to main memory gets more expensive every year, measured in processor cycles. One line I like: "although one minus p is small, it isn't zero."

Maya Give me the number.

Theo With a ninety-nine point eight percent hit rate, in their model, you hit the wall in eleven to twelve years. "The wall" meaning the average memory access costs more than five instructions' worth of time. And they projected the average cost of a memory access rising to one and a half cycles in two thousand, about eight in two thousand and five, and nearly a hundred by two thousand and ten.

Maya Did that happen?

Theo Not those numbers, and they said it wouldn't. The last part of the note says every such prediction in the past was wrong because of unstated assumptions, and then: "Our prediction of the memory wall is probably wrong too." What they wanted was for people to stop assuming the problem would solve itself. They dismissed prefetching outright, because in their model the memory bus was already fully used. Other tricks, they said, provide one-time boosts to either bandwidth or latency. None of them changes the two exponents.

Maya Thirty years later, the same idea comes back under the same name. In twenty twenty-four Amir Gholami and colleagues at Berkeley published "AI and Memory Wall", an extended version of an IEEE Micro article. They cite Wulf and McKee directly. They took twenty years of server hardware and fitted trend lines. Peak compute grew by about sixty thousand times. DRAM bandwidth grew by about a hundred times, and the bandwidth of the links between chips by about thirty.

Theo Sixty thousand against a hundred.

Maya Per two years, that's compute roughly tripling while memory bandwidth grows about one point six times. These are fitted lines through a chosen set of chips, not laws of physics. But the direction is the point. The arithmetic units have been sprinting and the roads into them have been walking.

Theo And the paper has a small experiment that I think is the real start of this episode.

Maya It's the one with BERT and GPT-two. They compare two transformer models with essentially the same shape and the same total arithmetic. BERT-base is an encoder: it reads a whole sequence at once. GPT-two is a decoder: it produces text one token at a time. They profile both on an Intel Xeon Gold server CPU at batch size one, timing BERT reading a sequence and GPT-two generating a sequence of the same length, token by token. GPT-two's latency is, in their words, significantly longer, even though it does the same number of floating-point operations, or fewer.

Theo Same amount of maths and much slower. So the maths isn't what you're waiting for.

Maya Exactly. When BERT reads a sequence, each weight it loads gets used for every token in that sequence. It's matrix times matrix. When GPT-two generates, each new token is a single vector, so every weight is loaded, used once, and thrown away. It's matrix times vector. The paper's term for the difference is arithmetic intensity: how many operations you get to do per byte you fetch from memory. Their line is that a model with higher arithmetic intensity "can run faster with the same or possibly even more FLOPs".

Theo So, more arithmetic, finished sooner. That's backwards from how most people think about speed.

Maya It is, and it's the whole puzzle from the cold open in miniature. One caution: the same paper later says CPUs have "much better performance than GPUs for bandwidth-bound problems". They don't test that claim, and as we'll see, it's too broad. What they did show is the mechanism. Generating text one token at a time is a memory problem before it's a compute problem.

Roofline and the two jobs: prefill, decode, Splitwise, and the ridge point of a desktop 8:30

Theo If arithmetic intensity is the key idea, the tool for using it is sixteen years older than AI and Memory Wall. In two thousand and eight Samuel Williams, Andrew Waterman and David Patterson at Berkeley wrote up something they called the Roofline model. It was published in Communications of the ACM the following year.

Maya Patterson as in the computer architecture textbooks.

Theo The same. And the model fits in one line. The speed you can attain is the smaller of two things: the peak arithmetic rate of the chip, or the memory bandwidth multiplied by the arithmetic intensity of your code.

Maya Say it as a picture.

Theo Put arithmetic intensity on the horizontal axis, operations per byte fetched from main memory. Put achieved performance on the vertical axis. You get a roof. On the left it slopes upward: there you're limited by bandwidth, and every extra operation per byte buys you more speed. Then it flattens out: there you're limited by the arithmetic units and extra intensity buys you nothing. The corner where the two meet they call the ridge point. It's the minimum intensity you need before the chip's peak compute even becomes relevant.

Maya And a detail people skip: the bytes they count are bytes that actually reach DRAM, after the caches have done their work. Not the loads you see in the source code.

Theo And the bandwidth is what you can actually sustain, measured with benchmarks, "not the pin bandwidth of the DRAM chips". Their worked example is a dual-socket Opteron server with seventeen point six billion floating-point operations a second of peak compute and fifteen gigabytes a second of measured bandwidth.

Maya There's a hook in that paper I didn't expect. People look at cache hit rates and think, ninety-eight percent hits, so we must be fine on memory. The authors point out that two of their kernels had hit rates of at least ninety-four and ninety-eight percent and were still not compute-bound. A high hit rate tells you the cache is working. It doesn't tell you the memory system is keeping up.

Theo Now put a language model on that roof. The survey "LLM Inference Unveiled", by Zhihang Yuan's group, builds a tool that does exactly that, on paper, operation by operation. These are analytical placements on the roof, not timings. Their worked example is Llama two, seven billion parameters, on an Nvidia A6000, with a two-thousand-token sequence and batch size one. In the prefill stage, reading the prompt, most operations land on the flat part of the roof: compute-bound. In the decode stage, in their words, "all computations are memory-bound".

Maya That lines up with the two gaps from the cold open. The GPU's hundred-times advantage should show up where the work is limited by arithmetic, and the thirteen-times advantage where it's limited by memory.

Theo With two caveats from the same survey. First, a short prompt doesn't give prefill much to reuse, so short prefills are memory-bound too. Second, if you batch many users together during decode, every weight you load now serves many tokens. Intensity rises and you move right along the roof. At large batch sizes their analysis says shrinking the weights from four bits to two or one "does not lead to a decrease in the inference time". You've stopped being limited by bytes.

Maya So "LLMs are memory-bound" is only true with qualifiers: decode, small batch, one user. Which happens to describe you, running a model on your own machine.

Theo One more from the survey: for Llama two thirteen B, beyond about fifty thousand tokens of context, the KV cache, the model's running notes on the conversation so far, takes more memory than the model itself.

Maya The clearest demonstration that these two jobs are different comes from Microsoft. Pratyush Patel and colleagues, with the University of Washington, published Splitwise in late twenty twenty-three. They profiled real serving traces from Azure on eight H100 GPUs. Running BLOOM, a hundred and seventy-six billion parameters, processing a fifteen-hundred-token prompt took the same time as generating six output tokens.

Theo Fifteen hundred tokens in, in the time it takes to write six.

Maya And then something even more telling. They lowered the power cap on the GPUs from seven hundred watts to three hundred and fifty. Prompt processing slowed down. Token generation showed almost no change.

Theo Halve the power and the generation speed doesn't move. That tells you how little of the chip's arithmetic decode was using.

Maya Their traces also show why the two jobs need separate numbers. In Azure's coding service, the median request had fifteen hundred tokens of prompt and thirteen tokens of output. In the conversation service, about a thousand in and a hundred and thirty out. So the paper reports two latencies separately: time to first token, which is the prompt, and time between tokens, which is the generation.

Theo And batching hits them differently. Running sixty-four requests together only doubled the time between tokens, because decode had so much idle compute to share. That's the batching economics cloud APIs rely on. On your own machine, with one user, you don't get that discount.

Maya Their conclusion is to put the two phases on different machines: prompt machines and token machines. They even suggest that token machines could use cheaper hardware, possibly CPUs with high-bandwidth memory. But they're clear that they couldn't test that and leave it to future work.

Theo We can do a small version of that test ourselves. If decode is limited by memory, there's a back-of-the-envelope rule. For a dense model like this Llama, each generated token has to stream every weight through the processor once. So tokens per second is roughly usable memory bandwidth divided by the size of the weights.

Maya On Sam's machine we measured the bandwidth first with STREAM, the standard memory benchmark. The best result was about sixty-four gigabytes a second, using eight threads on the eight performance cores. The Llama file is four point nine gigabytes of weights. Sixty-four divided by four point nine is about thirteen.

Theo And we measured twelve point seven with those same eight threads, and thirteen point four, the best result, when eight efficiency cores joined in.

Maya At its best, the CPU was behaving as if it pulled about sixty-six gigabytes a second out of memory, within a few percent of what STREAM says the memory can do. The eight-bit version of the same model is a useful check in the other direction. About seventy percent more bytes, about forty percent slower, and an implied seventy-two gigabytes a second. That's above what STREAM measured.

Theo Above the limit? That should be impossible.

Maya It's a reminder that the implied number is an estimate, not a reading of the memory bus. Not every row of the embedding table is touched for every token, caches help a little, and STREAM counts a different mix of reads and writes. But when a rule of thumb lands within a few percent for one file and about ten percent for the other, the CPU is mostly waiting on RAM.

Theo And the graphics card? Its memory is rated at about a thousand gigabytes a second. The same calculation on its hundred and seventy-two tokens gives about eight hundred and forty-five, so it gets about eighty-five percent of its rating. A thousand against sixty-four is about sixteen times. We measured thirteen.

Maya Why does the card fall further short of its rating than the CPU does?

Theo A token on the card takes under six milliseconds, so fixed costs, like launching kernels and reading the attention cache, are a bigger share of it. And at batch size one the matrix-vector kernels don't stream memory quite as perfectly as a benchmark does. That's a plausible explanation, not one we measured. But the shape holds: the decode gap is mostly the memory bandwidth gap.

Maya And we can find the other half from the same runs. During prefill, the CPU did about two trillion operations a second, counting a multiply and an add for each weight and token. The card did about two hundred trillion. That hundred-times ratio is the prefill gap. So the prefill gap is the compute gap, by construction. The interesting part is where the two regimes meet.

Theo The ridge point, where extra work per byte stops being free.

Maya Right. Divide each processor's achieved compute by its memory bandwidth. The CPU comes out at about thirty operations per byte, the card at about two hundred, using its rated bandwidth. Decoding one token does about three operations per byte of four-bit weights. So the CPU is far below its ridge, the card even further below. Process a batch of tokens together and the intensity multiplies by the batch size. So on the CPU, about ten tokens processed together is enough to stop being memory-bound. On the card it takes about sixty.

Theo So a three-word prompt is still a memory problem, even on the CPU, and a paragraph is a compute problem.

Maya Which is why Yuan's survey says short prefills are memory-bound too, and why a GPU needs big batches before its compute is used.

Why the GPU wins, and the 100X myth: Lee et al. 2010, Mind the Memory Gap, AVX-512 and AMX 18:38

Maya So the decode gap is mostly about memory. But there's a version of "GPUs are faster" that's about raw compute, and it has an interesting history. Around two thousand and eight and two thousand and nine, papers kept reporting that GPUs ran some workload a hundred times faster than CPUs, sometimes more.

Theo And in twenty ten, at ISCA, the big computer architecture conference, a team led by Victor Lee published "Debunking the one hundred X GPU versus CPU Myth".

Maya With an important disclosure: they all worked at Intel.

Theo Yes, so read it as a CPU company's argument, but a well-made one. They took fourteen throughput kernels: dense matrix multiply, sparse matrix-vector, sorting, FFT, a collision-detection kernel, Monte Carlo pricing and so on. They tuned every one carefully for both a four-core Core i7 nine-sixty and an Nvidia GTX two-eighty. Their finding: after tuning both, the GPU was on average about two and a half times faster. Not a hundred.

Maya Where did the hundreds come from, then?

Theo Mostly from comparing carefully optimised GPU code with CPU code nobody had optimised. In their words: "Many studies compare optimized GPU code to unoptimized CPU code." They say an untuned CPU implementation can be orders of magnitude below what the hardware can do.

Maya And they found cases where the CPU won outright.

Theo Two. Sorting: two hundred and fifty million elements a second on the CPU, a hundred and ninety-eight on the GPU. And a constraint solver for physics, a hundred and three frames a second against fifty-two. The sorting case is a nice illustration of how cache changes the algorithm. With a big cache, the CPU could sort thirty-two-bit keys in four passes over the data. The GPU needed eight.

Maya And where did the GPU win big?

Theo Exactly where you'd now expect. Dense matrix multiply: ninety-four billion operations a second on the CPU against three hundred and sixty-four on the GPU. And anything that streams memory, because the GPU had a hundred and forty-one gigabytes a second of peak bandwidth against the CPU's thirty-two. One more caveat: they excluded the time to copy data to the GPU, so they called the GPU figures an upper bound for real applications.

Maya In twenty twenty-three the authors wrote a short retrospective on the paper. Its main lesson is about method, not hardware. If you compare architectures, tune the CPU baseline too. Be willing to use a different algorithm on each, because identical code can make the comparison less fair, not more. And check each result against that machine's roofline.

Theo Which is the standard we should hold our own test to. We didn't write a careful CPU kernel. We used llama.cpp, which has had years of CPU tuning from a lot of people. That's about as fair a fight as a hobbyist gets.

Maya So what's changed since twenty ten? The bandwidth ratio has grown. In twenty ten the peak figures were a hundred and forty-one against thirty-two, about four and a half times. On Sam's machine it's the card's rated thousand against the sixty-four we measured on the CPU, about sixteen times. Not quite the same basis, but the trend is clear. Graphics cards moved to very wide, very fast memory soldered right next to the chip. Desktop CPUs still talk to two memory channels on sticks you can pull out.

Theo And the compute ratio has grown much more, because GPUs gained tensor cores: hardware built to do small matrix multiplies in low precision, many per clock. That's the hundred-times prefill gap we measured.

Maya Tensor cores explain the compute gap. They don't help while the card is waiting on memory, though.

Theo There's a structural reason the GPU copes better with memory than its bandwidth alone suggests. A CPU core runs a couple of threads and relies on caches and prediction to avoid waiting. A GPU keeps thousands of threads in flight. When some of them stall on memory, others are ready to do arithmetic. It hides latency with sheer numbers.

Maya Though even that has limits. A twenty twenty-five paper from the Barcelona Supercomputing Center and IBM, "Mind the Memory Gap", profiled an H100 serving small models at the largest batch sizes that fit. In the attention kernels, more than half of all cycles were spent idle, waiting for data. With one popular attention implementation, over eighty percent. Batching makes the weight multiplications more efficient. But each user's attention still reads that user's own cache, so its arithmetic intensity stays almost flat no matter how big the batch gets.

Theo How flat? Each request re-reads its own cache, so I'd expect batching to do nothing for it.

Maya For their smallest model, the attention kernels did between half an operation and one operation per byte, both at batch size one and at the maximum batch. And in their offline runs, prompt processing stayed under five percent of total time even at the largest batch. Generation dominated. Their practical suggestion follows from that. Pick a smaller batch at the knee of the curve. For one small model they kept eighty-three percent of the maximum throughput while using only about a sixth of the memory for the KV cache.

Theo So a flagship data-centre GPU also spends much of its time waiting on memory. It just has a much faster memory system to wait on.

Maya What about the CPU side? CPUs have their own matrix hardware now.

Theo They do, in stages. AVX-two gives you two-hundred-and-fifty-six-bit vector instructions. AVX five-twelve doubles the width. VNNI adds instructions that do eight-bit integer dot products in one step, which matters for quantized models. And Intel's server Xeons since Sapphire Rapids have AMX, tile registers with a dedicated matrix-multiply unit, which is the closest thing a CPU has to tensor cores.

Maya And Sam's fourteen-nine-hundred-K?

Theo Has AVX-two and AVXVNNI, but no AVX five-twelve and no AMX. Intel's hybrid desktop chips mix big performance cores with small efficiency cores. The small ones don't support AVX five-twelve, and Intel disables it on the whole chip rather than let the two kinds of core differ. The flagship gaming processor has narrower vector units than a server chip from several years earlier. And even where AMX exists, the spec sheet and the achieved speed are far apart, as we'll see with KTransformers.

More cores, slower: our thread sweep, Justine Tunney's llamafile, Intel's Xeon INT4 runtime 26:15

Maya Let's stay on the CPU and ask the obvious practical question. If you're running a model on your processor, how many threads should you give it?

Theo All of them, surely. More cores, more speed.

Maya That was our assumption too, so we measured it. Same Llama eight B file, four-and-a-half-bit quantization, generating a hundred and twenty-eight tokens, from one thread up to all thirty-two. With one thread: four tokens a second. Two threads: seven. Four threads: eleven. Eight threads, one on each performance core: twelve point seven.

Theo So it's levelling off already.

Maya Sixteen threads, eight performance cores plus eight efficiency cores: thirteen point four, the best result. Twenty-four threads: ten. And all thirty-two threads: two point seven tokens a second.

Theo Two point seven. Using the whole chip made it five times slower than using half of it.

Maya Now compare the memory benchmark. STREAM on one thread got about thirty-three gigabytes a second. Four threads got sixty-two, eight threads sixty-four, and adding more threads after that made it about fifteen percent worse. Decode speed tracks that curve. After four to eight cores, the memory is saturated. Extra cores have nothing to do except wait, and they get in each other's way while waiting.

Theo And reading the prompt, on the same sweep?

Maya Completely different shape. Prefill went from sixteen tokens a second on one thread to twenty-nine on two, fifty-four on four, eighty-nine on eight. Nearly linear. And it kept climbing with the efficiency cores, to a hundred and twenty-two at all thirty-two threads, exactly where generation had collapsed to under three.

Theo So more arithmetic units help reading and do nothing for writing. That's the roofline, measured on one chip.

Maya With one honest wrinkle: the eight-bit version's prefill didn't scale smoothly. It dipped at twelve and sixteen threads before recovering. We didn't chase down why.

Theo Why the collapse in generation at thirty-two, rather than just a plateau?

Maya We don't know for certain and didn't prove it. The likely ingredients are the mix of fast and slow cores, hyperthreads sharing a core, and how the threads synchronise. In llama.cpp every thread has to finish its share of a layer before anyone starts the next one. So the whole team waits for the slowest member.

Theo Justine Tunney made exactly that observation in twenty twenty-four, and on the same CPU. She wrote a blog post called "LLaMA Now Goes Faster on CPUs" about new matrix-multiply kernels she'd written for llamafile, the Mozilla-funded project that packages a model and the inference engine into one executable. Her benchmarks include an overclocked fourteen-nine-hundred-K. She says llamafile avoids the efficiency cores, because the worker threads run in lockstep.

Maya In lockstep, so the slowest core sets the pace for all of them.

Theo Right. But the more interesting part of her post is what her kernels sped up and what they didn't. Her own description is that prompt evaluation got thirty to five hundred percent faster. On the fourteen-nine-hundred-K, with the small TinyLlama model in sixteen-bit precision, prompt processing went from ninety tokens a second to four hundred and seven. Generation went from thirty-one to forty-two.

Maya A four-and-a-half-times speedup on reading, and about a third faster on writing. Though she also changed which cores the threads ran on in that release, so the kernels can't take all the credit for the writing gain.

Theo And on a Raspberry Pi five it's even starker. Prompt processing went from twenty-eight tokens a second to sixty-two. Generation stayed at five.

Maya Unchanged, because on that board generation was waiting on memory, not arithmetic.

Theo Exactly. Better kernels help where you're compute-bound. A prompt of any real length is compute-bound. Generating one token at a time for one user isn't. She explains it in the post: during generation the matrix multiply is really a matrix-vector multiply, while "chewing through prompts requires bona fide matrix-matrix multiplication". Her trick was mostly about reuse: unroll the outer loops so one value loaded into a register gets used for several outputs before it's thrown away.

Maya That's raising arithmetic intensity again, at the scale of registers instead of DRAM.

Theo She's also honest about limits. Her kernel beat Intel's MKL math library, eight hundred and ten billion operations a second against two hundred and ninety-five, for a five-hundred-and-twelve-sized matrix. Then she doubles the size and writes: "MKL wins once more." And on the Apple M-two Ultra, for one of the models, the new code was slower than the old.

Maya There's an example in the post that makes the prompt-versus-generation point useful in practice. A spam filter on a Raspberry Pi. You feed in an email and ask the model to answer with a single word, yes or no. It reads hundreds of tokens and writes one, so generation speed is irrelevant, as she says.

Theo Lots of real tasks look like that. Classifying, tagging, routing, extracting a field. They're almost all reading.

Maya Which matters, because reading is exactly where the CPU is weakest relative to a GPU. Hold that thought.

Theo Intel had already published on this a few months earlier. "Efficient LLM Inference on CPUs", by Haihao Shen and colleagues, late twenty twenty-three. They built a runtime with four-bit weights and CPU-specific kernels, on one socket of a fourth-generation Xeon. For Llama two seven B they report about twenty-two milliseconds per generated token. That's roughly forty-five tokens a second, with thirty-two tokens in and thirty-two out.

Maya Better than our desktop, which you'd expect from a server socket with many more memory channels.

Theo And they compared against a GGML-based implementation, the library behind llama.cpp. At the same quantization settings they were up to one point three times faster. The headline "up to one point six times" came from also changing how the weights were grouped. Two other details are worth knowing. They compute in eight-bit and sixteen-bit formats even though the weights are stored as four-bit. A "four-bit model" doesn't do four-bit arithmetic. And the size of the quantization groups changes the memory bill: three point eight gigabytes against four point nine for the same model at the same nominal four bits.

Maya And they're upfront that they tested a server, not a PC. Consumer PCs are listed as future work.

Fewer bits, smarter kernels: T-MAC, BitNet b1.58, bitnet.cpp's dummy 100B, BitNet 2B4T, ternary zeros 33:39

Theo If generating is just streaming bytes, the obvious move on a CPU is fewer bytes. We spent a whole episode last time on quantization. Here's the CPU-specific twist: every compressed weight has to be unpacked before you can use it, and on a CPU that unpacking competes for the same cores.

Maya The T-MAC paper puts a number on that. It's from Microsoft Research and two Chinese universities, Jianyu Wei and colleagues, twenty twenty-four, later at EuroSys. In their measurements, llama.cpp ran about fifteen percent slower at three bits than at four, "due to decoding overhead".

Theo Fewer bits, slower. Because three-bit values don't line up neatly in bytes, unpacking them costs extra instructions, and those instructions cost more time than the saved bytes bought back.

Maya T-MAC's answer is to stop multiplying altogether. Take a low-bit weight matrix and split it into one-bit layers. A one-bit weight only says add this activation or don't. So for a group of four weights there are only sixteen possible patterns. Precompute the sixteen possible sums of the four activations once, put them in a small table, and the weights become indexes into that table. In their words, the lookup table for groups of four "exactly fits into one register".

Theo So the core operation becomes a table lookup, which CPUs are very good at.

Maya Their headline: BitNet b one point five eight, a three-billion-parameter ternary model, at thirty tokens a second on a single core of an M-two Ultra, seventy-one on eight cores, and eleven tokens a second on a Raspberry Pi five. They also report up to sixty percent less energy for some configurations, measured with Apple's power-monitoring tool rather than at the wall socket.

Theo And the result they didn't put in the headline?

Maya On Nvidia's Jetson Orin board, with Llama two seven B at two bits: plain llama.cpp on the CPU got seven tokens a second, T-MAC on the CPU got sixteen, and the board's own GPU got twenty. T-MAC doubled the CPU, and the GPU still won. The authors say why: the rest of the model, everything that isn't these matrix-vector products, still runs at ordinary speed.

Theo You mentioned ternary. That's the BitNet line of work, which has some of the most overstated claims in this whole area. Worth going through carefully.

Maya Three papers. First, in early twenty twenty-four, Shuming Ma and colleagues at Microsoft published "The Era of One-bit LLMs". The weights in every linear layer take one of three values, minus one, zero or plus one. That's about one point five eight bits each, the logarithm of three, hence the name. The crucial point is that the model is trained that way from scratch. You don't take a normal model and round it.

Theo Trained from scratch on how much, though? A lab result at small scale isn't a model anyone uses.

Maya Fair, and at the sizes they trained it was competitive. At three billion parameters their ternary model matched their own reproduction of a sixteen-bit LLaMA, perplexity nine point nine against ten point oh. With three and a half times less memory and two point seven times lower latency per token. Two details people drop: the embedding layer stays at full precision, and all those speed measurements were on GPUs, not CPUs.

Theo So every efficiency number so far is a GPU number. Where does the famous CPU claim come from?

Maya The second paper, October twenty twenty-four: bitnet.cpp, a CPU inference engine for ternary models. This is where the famous line comes from: a hundred-billion-parameter model on a single CPU at human reading speed, five to seven tokens a second.

Theo Which I'd bet wasn't a laptop.

Maya It wasn't. Six point five eight tokens a second on a Mac Studio with an M-two Ultra and sixty-four gigabytes of memory, using all threads. With two threads, one point two seven. On the Intel laptop they tested, one point seven. And the hundred-billion-parameter model wasn't a trained model. The appendix says it plainly: "The tested models are dummy setups." It was the right shape, filled with weights that don't encode anything.

Theo So it was a speed test of the kernel on a model-shaped workload. Useful, but not an intelligent hundred-B model running on a laptop.

Maya Right. It shows that if someone trains a hundred-billion-parameter ternary model, this is roughly how fast it would generate. As far as we could find, nobody has released one. Two more details. Their speedup comparison runs llama.cpp with sixteen-bit weights, so of course moving roughly eight times fewer bytes is faster. And their claim of "lossless" inference means the output matched the full-precision engine token for token, on a small seven-hundred-million-parameter community model. That's a real and useful property, but it isn't a claim about accuracy on tasks.

Theo How does bitnet.cpp actually store a one-point-five-eight-bit weight? You can't put a fraction of a bit in memory.

Maya You can't, and it doesn't. It ships three kernels. The first stores each ternary weight in two bits, unpacks it and does an ordinary matrix-vector product. They recommend it when you have plenty of threads. The second packs pairs of weights into a four-bit index into a small lookup table, borrowing T-MAC's idea, for when threads are limited. The third packs three weights into five bits, which shrinks the model by another sixth, for when memory or bandwidth is the limit.

Theo Three kernels for three bottlenecks. Unpack and multiply when you have cores to spare, look up when cores are scarce, pack tighter when bytes are scarce.

Maya And it's the same lesson Yuan's survey states in general. If only the weights are eight-bit and the activations stay sixteen-bit, "the int-eight computational power cannot be utilized". All the operands have to be in the low format before the fast arithmetic units help. Weight-only quantization saves bytes. It doesn't save operations.

Theo Has anyone released a real one at any size?

Maya At small size, yes. That's the third paper. BitNet b one point five eight two B four T, April twenty twenty-five. Two billion parameters, trained on four trillion tokens, actually released. They timed it on a Surface laptop with a thirteenth-generation Core i7 running eight threads. Twenty-nine milliseconds per generated token. Llama three point two one B took forty-eight, Gemma three one B forty-one, Qwen two point five one point five B sixty-five. On their benchmark average it scored fifty-four, a little behind Qwen's fifty-five and ahead of the other two.

Theo So the fastest model in that comparison, and nearly the best.

Maya With caveats. The ternary model ran in bitnet.cpp, the others in llama.cpp, so model and engine both changed. The point four gigabytes of memory they quote excludes the embedding table. And the energy figures are estimates of arithmetic energy on a seven-nanometre chip, not measurements on the laptop.

Theo There's a twenty twenty-six follow-up from Intel that I think is the smartest paper in this cluster. Evangelos Georganas, Alexander Heinecke and Pradeep Dubey looked at the weights of twenty-nine published ternary models and counted. The three values aren't equally common. Up to fifty-one and a half percent of the weights are zero.

Maya So storing every weight as if minus one, zero and plus one were equally likely is wasteful.

Theo Their format is one bitmap saying "is this weight non-zero", plus a compact list of signs for just the non-zero ones. As they put it, "Zeros consume a bitmap bit and nothing else." On the sparsest model that gets the weights down to about one point four nine bits each, about one point six with the scaling factors included. In full decoding, on a sixty-four-core Xeon and on a desktop Core Ultra nine two-eighty-five-K, that was up to eighteen and fifteen percent faster respectively than a two-bit format.

Maya That's a small gain for a clever format. Does it hold everywhere?

Theo No, and that's the interesting part. On a Lunar Lake laptop chip, with fast LPDDR5-X memory and only eight cores, their format was slower than the two-bit reference at every density they tried. Their words: "Lunar Lake is instruction-bound on both core types."

Maya So with more bandwidth per core, unpacking became the bottleneck instead of memory.

Theo Yes. Compression is a trade between bytes and instructions, and the right answer depends on the ratio of bandwidth to compute on your particular chip. Same model, same format, opposite verdicts on two Intel processors.

Maya One last paper here, because it turns the rule of thumb into a tool. Xinyu Qiu and colleagues, twenty twenty-six: they tried to predict llama.cpp speed using nothing but the metadata inside a model file. For decode, a roofline-style formula that charges only the weights actually used per token predicted held-out models within about thirteen to fourteen percent on two Apple M-four Max machines, running on their graphics cores rather than the CPU. Charging all the weights in the file, the error was about fifty percent.

Theo The weights actually used per token. That only differs from the weights in the file if…

Maya If the model is a mixture of experts. In their words: "Capacity is controlled by stored parameters, whereas per-token weight traffic is closer to activated parameters." They also found the limits. On an RTX fifty-eighty the decode error was thirty-six percent. Prefill was much harder to predict, over a hundred percent error on that card. And the ranking of two low-bit formats flipped between Apple and Nvidia. In their words, "File compression alone therefore does not specify kernel efficiency."

Theo And the attention cache? That's traffic too, and it grows with the conversation.

Maya One useful detail. Some newer models, Gemma among them, use full attention only in some layers and a short sliding window in the rest. If you assume every layer keeps the whole conversation, you overestimate the cache traffic at sixteen thousand tokens by up to about fourteen times. Knowing the architecture matters for the cache, not just the weights.

Mixture of experts, the CPU's friend: 30B beats 8B, DeepSeek on servers, NUMA, Cascadia 45:21

Maya That idea, that what matters is the weights each token uses rather than the weights in the file, is the most important practical idea for anyone running models on a CPU. Here's how it looked on Sam's machine.

Theo We ran Qwen three thirty B A three B. Thirty billion parameters in total, about three billion active for each token. It's a mixture of experts. Each layer contains a hundred and twenty-eight small expert networks, and a router picks eight of them for each token.

Maya The file is eighteen and a half gigabytes, nearly four times the size of the dense Llama eight B. On the CPU alone, eight threads, it generated at twenty-six tokens a second.

Theo And the dense eight B managed thirteen.

Maya So the model with four times the bytes ran twice as fast. Because per token it only reads the experts the router picked plus the shared parts. That's about two gigabytes, not eighteen. Sixty-four gigabytes a second divided by about two gives a ceiling around thirty tokens a second. We got twenty-six.

Theo Which is the shape of the deal. On a GPU, the scarce thing is memory capacity: twenty-four gigabytes, that's it. On a CPU you can have sixty-four, a hundred and twenty-eight, half a terabyte. What's scarce there is bandwidth. A mixture-of-experts model needs lots of capacity and modest bandwidth per token, which is exactly what system RAM offers.

Maya How far does that go? This is where the evidence switches from papers to practitioners, so treat what follows as reports from individual machines, not controlled studies.

Theo The biggest open models today are mixtures of experts with hundreds of billions of parameters. DeepSeek R1, six hundred and seventy-one billion in total. Someone ran it with no GPU at all on a dual-socket Intel Xeon six-nine-eighty-P server, twenty-four sticks of DDR5-sixty-four-hundred, at eight-bit quantization. Using one socket: about five and a half tokens a second generating, and about fifty-two reading the prompt. Spreading the same hundred and twenty-eight threads across both sockets made it slower, about four tokens a second.

Maya Two sockets, more total bandwidth, and it got slower?

Theo Most likely because each socket has its own memory. When a thread on one socket reads weights stored in the other socket's memory, it goes over a slower link between them. Another report, on a different AMD server, saw a seventy-billion-parameter model go from two point four to four point three tokens a second just by changing how the weights' memory pages were first placed across the two sockets. Bandwidth sets the ceiling. Where the bytes live decides how close you get.

Maya Another report: two AMD EPYC nine-four-seventy-four-F processors with twenty-four sticks of DDR5-forty-eight-hundred. DeepSeek R1 at four bits: six and a half tokens a second generating, forty-four reading. Qwen three two-thirty-five B: eight generating, seventy-one reading.

Theo Six tokens a second is slow but usable for chatting. Roughly reading speed. But look at the other number.

Maya Forty-four tokens a second of prompt processing, for a five-hundred-token prompt. Paste in a ten-thousand-token document and at that rate you'd wait nearly four minutes before the first word, and in practice longer, because prompt processing slows as the context grows. On the same machine, with a four-thousand-token prompt, prompt processing fell to thirty-four tokens a second and generation to under five. Give it a coding agent with thirty thousand tokens of context and you're waiting a quarter of an hour.

Theo That's the hidden cost of CPU inference. Everyone quotes generation speed because it looks respectable. Prefill is what actually decides whether the machine is usable.

Maya There's a twenty twenty-six paper that pushes this as far as it goes. It's called Cascadia, from Tate Berenbaum and colleagues, with Intel co-authors and on Intel hardware. They ran a nine-hundred-and-seventy-five-billion-parameter mixture-of-experts model, forty-one billion active per token, on eleven laptop-class Intel AI PCs. Core Ultra processors with sixty-four gigabytes each, connected by ordinary gigabit Ethernet. Each machine holds six consecutive layers and passes activations to the next, a pipeline. The work runs on each machine's CPU and integrated graphics, not the NPU.

Theo Nearly a trillion parameters on a pile of laptops. How fast?

Maya With eighty-eight conversations running at once, sixty tokens a second in total. Which is about seven tenths of a token a second for each conversation.

Theo So the throughput number sounds impressive and each user waits more than a second per word.

Maya And the prompt side is brutal. A sixty-four-thousand-token prompt took a hundred and ten minutes to produce the first token. Then it generated at under one token a second.

Theo Nearly two hours before the first word.

Maya The paper is honest about it. Their own phrasing is roughly that contexts up to half a million tokens can fit in memory without being served at useful speed. That's the point of this section. Capacity lets you run the model. It doesn't make it fast.

Hybrid I: FlexGen, LLM in a Flash, PowerInfer on a 13900K and 4090, PowerInfer-2 51:16

Theo So the CPU has capacity and the GPU has speed. The obvious move is to use both. The research on how to do that well starts with a question that sounds trivial: when part of the model doesn't fit on the card, what do you move, and where do you do the arithmetic?

Maya The first answer was: maximise throughput and accept the latency. That's FlexGen, from Ying Sheng and colleagues at Stanford, Berkeley and elsewhere, twenty twenty-three. One Nvidia T-four with sixteen gigabytes of memory, two hundred and eight gigabytes of system RAM and a fast SSD. They wanted to run OPT one seventy-five B, a hundred and seventy-five billion parameters.

Theo A model more than twenty times bigger than the card's memory. Something has to give.

Maya What gives is latency. They treat the GPU, the RAM and the disk as one hierarchy, and solve an optimisation problem for where each piece lives: weights, activations and the KV cache. Then they process huge batches, so every weight brought onto the card gets used by many sequences before it's evicted. In their benchmark, a sixteen-bit OPT one seventy-five B generated about seven tenths of a token a second in aggregate. Compressed to four bits, so it fitted in RAM with no disk traffic, about one point one. The existing offloading systems managed one hundredth of a token a second.

Theo A hundred times better, from moving the same bytes more cleverly. But one point one tokens a second.

Maya Spread over a batch of a hundred and forty-four sequences, which took over an hour to finish. FlexGen says outright that it's for "latency-insensitive batch-processing tasks". It's the right tool for scoring a million documents overnight, and the wrong one for chat.

Theo Two details from FlexGen that come back later. First, they found it helps to compute attention on the CPU, where the KV cache already lives, so only small activations cross the bus. Turning CPU compute off cut their thirty-billion-parameter throughput from seven point three to four tokens a second. Second, their four-bit weights get converted back to sixteen-bit before the arithmetic, like Intel's CPU runtime earlier: the storage format isn't the compute format.

Maya Apple came at it from the opposite side: one user, small device. "LLM in a flash", Keivan Alizadeh and colleagues, December twenty twenty-three. The model is stored on flash, and only about half of it fits in DRAM. They exploit the fact that in some models most neurons in the feed-forward layers output zero for any given token. So they predict which ones will fire and load only those. They keep recently used neurons cached, and they lay out the file so each neuron's incoming and outgoing weights sit next to each other on flash.

Theo So each read is one contiguous chunk.

Maya And their most counterintuitive lesson: it's "worthwhile to read more than needed (but in larger chunks) and then discard". Flash is fast at large reads and slow at small scattered ones. On an RTX 4090 with a six-point-seven-billion-parameter OPT model under that memory restriction, they went from over two seconds per token, loading naively, to eighty-four milliseconds. With the caveat that the baseline is naive loading, not a model that simply fits in memory. And it isn't free in quality: for models that don't naturally produce those zeros, they convert them to ReLU activations first. For Microsoft's Phi-two, that took the MMLU score from fifty-seven to about fifty-four, and adding the neuron predictors took it to fifty-two.

Theo That's the graphics card. What about the CPU?

Maya On an M1 Max CPU, running in thirty-two-bit, the same restricted-memory setup went from over three seconds per token to about two thirds of a second. The data movement shows why. In a separate sixteen-bit measurement of the flash reads alone, the simple approach, half the model in memory and the other half loaded every token, moved six point seven gigabytes per token. Predicting neurons and keeping recent ones cached cut that to about nought point two gigabytes. And bundling each neuron's weights together cut the read time from a hundred and sixty-four milliseconds to eighty-seven.

Theo Now the paper I think is most relevant to anyone with a gaming PC. PowerInfer, from Yixin Song and colleagues at Shanghai Jiao Tong University, December twenty twenty-three. Their main test machine, which they call PC-High, is an Intel i9 thirteen-nine-hundred-K, an RTX 4090 with twenty-four gigabytes, and host memory they describe at sixty-seven gigabytes a second.

Maya That's practically Sam's machine. One generation newer CPU, the same card, and we measured sixty-four gigabytes a second.

Theo Which makes it a direct ancestor of our own experiments. Their observation is about which neurons fire. In OPT thirty B, about seventeen percent of the neurons account for eighty percent of all activations. A small set of hot neurons fires for almost every token, and a long tail of cold ones fires rarely.

Maya So put the hot neurons on the GPU.

Theo And the key decision: when a cold neuron does fire, don't copy its weights over to the GPU. Compute it on the CPU, where it already lives. Their words: "computing them on the CPU is faster than transferring them to the GPU." The CPU does a little work on data it already has, instead of the bus carrying a lot of data to a faster processor.

Maya How much did that buy on a machine like ours?

Theo On PC-High, with sixteen-bit models, an average of eight tokens a second and a peak of sixteen. On average about seven times faster than llama.cpp, which in that era offloaded whole layers. They also claim that for OPT thirty B on short prompts, the 4090 with PowerInfer was only about eighteen percent slower than an eighty-gigabyte A100 running v-LLM.

Maya That A100 comparison sounds too good to be general.

Theo It is, for two reasons. It needs models whose neurons switch off. They used models with ReLU activations, including versions of Falcon and Llama they'd converted to ReLU. Most modern models use activations that are rarely exactly zero, so the trick doesn't apply directly. And long prompts erase the advantage, because each token activates a different set of neurons and together they light up most of the network.

Maya So the advantage fades exactly where the CPU was already weakest, on long prompts.

Theo PowerInfer two took the idea to phones in twenty twenty-four. On a OnePlus twelve with twenty-four gigabytes, nineteen of them usable by apps, they ran a sparse forty-seven-billion-parameter Mixtral variant at about eleven and a half tokens a second. That model only activates about three billion parameters per token. They used the phone's NPU for the dense, predictable work and the CPU for the sparse, unpredictable neurons. Their reason: the NPU's sparse operations "perform slower than CPU implementations".

Maya So even with an accelerator sitting right there, the CPU kept the irregular work.

Hybrid II, experts: Fiddler, MoE-Lightning, HybriMoE, NEO, KTransformers, an 80B model on 2.5 GB of VRAM 59:26

Maya Mixture-of-experts models made the hybrid question much more practical, because the hot and cold parts are no longer something you have to predict. The architecture tells you. The attention layers and any shared expert run for every token. The routed experts each run occasionally.

Theo The paper that set the pattern is Fiddler, from Keisuke Kamahori and colleagues at the University of Washington, early twenty twenty-four. Mixtral eight by seven B at sixteen-bit precision on a GPU too small to hold it. The existing approach was to copy whichever expert the router picked over PCI-e onto the card, then run it there.

Maya And Fiddler's point is about sizes.

Theo One Mixtral expert is three matrices of about four thousand by fourteen thousand. That's over three hundred megabytes at sixteen bits. The activation it has to process, for one token, is a few kilobytes. So they measured both. Copying an expert's weights to the GPU took two to five times longer than the GPU then spent computing with them. Sending the activation to the CPU and back was under one percent of the latency.

Maya So you move the small thing to where the big thing already is.

Theo For small inputs, yes: compute the expert on the CPU. For large inputs, like a long prompt where many tokens hit the same expert, the weights get reused enough that copying them to the GPU pays off. Fiddler chooses per expert, per step. The result was about one point three times faster end to end than the best baseline, llama.cpp. A modest number, but the principle stuck.

Maya There's one detail in Fiddler I liked because it punctures an assumption. You'd think you should keep the popular experts on the GPU. They profiled which experts were popular and placed those. It beat random placement by only three to five percentage points of hit rate.

Theo So the routers spread the load fairly evenly. There's no small set of star experts to keep on the card.

Maya Then MoE Lightning, from Berkeley, twenty twenty-four: offline batch generation for MoE models on small GPUs, in the FlexGen tradition. Their memorable measurement: computing attention on the CPU was three to four times faster than just transferring that layer's KV cache to the GPU.

Theo Faster to do the work than to send the inputs somewhere else.

Maya HybriMoE, twenty twenty-five, adds scheduling. Sometimes an expert that's already sitting on the GPU should still be computed on the CPU, because the GPU is busy with something else at that moment. And NEO, also twenty twenty-four, moves attention and the KV cache for some users onto the CPU, so the GPU can fit a bigger batch. On a weak T-four it reported a huge throughput gain at the same latency. On H100s, about fourteen percent. And they note the CPU side is limited by "memory bandwidth, rather than computing power".

Theo The paper that brought this to people running the biggest open models at home is KTransformers. From Tsinghua University and the company Approaching AI. It was published at SOSP in twenty twenty-five, one of the top systems conferences. The setup in the paper: two Intel Xeon processors, each with one terabyte of DDR5, plus either an A100 or a consumer RTX forty-eighty with sixteen gigabytes.

Maya Two terabytes of RAM. Remember that whenever someone says "one consumer GPU".

Theo The design: attention, the shared expert and everything that runs for every token goes on the GPU. The routed experts, six hundred and fifty-four of the six hundred and seventy-one billion parameters in DeepSeek V-three, go in system memory, computed by the CPU with hand-written kernels. Earlier we mentioned AMX, Intel's matrix unit. Their custom AMX kernel reached twenty-one trillion operations a second against five point four for the standard library. But for decode-sized work, four or fewer tokens per expert, the plain AVX five-twelve vector path was faster than AMX.

Maya So the matrix engine only helps when there's enough matrix to fill it.

Theo A handful of tokens per expert is far too little reuse to reach the ridge point. Their headline is four point six to almost twenty times faster prefill and one point two five to four times faster decode than existing systems. But those comparisons are against Fiddler and a version of llama.cpp they modified themselves, so read them as ranges. The number I'd remember is more modest. The Fiddler-style baseline for DeepSeek V-three on their server generated four point seven tokens a second. Their optimised system, before any approximation, got five point nine.

Maya Before any approximation, meaning there's an optional extra. But first, where did the gain from four point seven to five point nine come from, if not from approximations?

Theo From several systems changes. The kernels we mentioned are one. A big one is treating the machine as what it is: two sockets, each with its own memory. They measured two hundred and twenty gigabytes a second within a socket and a hundred and twenty-five across sockets. Fiddler, which treats the machine as one big pool, ran one DeepSeek layer in six point nine milliseconds on one socket and five point eight on both. Twice the hardware, sixteen percent faster.

Maya The same thing we saw in the forum report, where two sockets were slower than one.

Theo KTransformers splits every expert's matrices across the two sockets, so each socket only reads its own slice from its own memory. That alone was up to one point six times faster for decode. Then there's the coordination cost. Each token goes back and forth between CPU and GPU for every layer, and each hand-off normally means launching small GPU jobs. They captured the whole decode step for one token in a single CUDA graph, a pre-recorded sequence the GPU replays, which was worth up to another one point two times.

Maya And after all that, the timeline showed the CPU busy seventy-four percent of the time and the GPU only twenty-eight. The two sides still waited on each other.

Theo That's what the optional extra attacks. It's called Expert Deferral. Normally each token waits for all eight of its chosen experts before moving to the next layer. They let some experts' results arrive a layer late, so the CPU and GPU overlap better. It changes the model's computation, so it's no longer exactly DeepSeek. Up to forty-five percent more throughput. But only up to a point: in one configuration, deferring a fourth expert added nothing, because the CPU was already saturated. With six of eight experts deferred, the average relative accuracy drop on their benchmark was half a percent. Just skipping those six experts instead cost about thirteen percent.

Maya And then the project's own tutorial, which is where most people met these numbers.

Theo Two Xeon Gold processors, one terabyte of DDR5-forty-eight-hundred, one RTX 4090-D. DeepSeek V-three at four bits, with all eight experts: eighty-three tokens a second reading, twelve generating. The tutorial's headline speedups, three times and nine times, and later "up to twenty-seven point seven nine times", come from comparing against llama.cpp while also changing the number of experts from eight to six, the prompt length, and in one case the precision.

Maya So: twelve tokens a second from a six-hundred-and-seventy-one-billion-parameter model on one consumer card, plus a terabyte of server memory. That's a real achievement. The twenty-eight times isn't.

Theo Now our version. Qwen three Next eighty B A three B. Eighty billion parameters, about three billion active per token, forty-eight and a half gigabytes at four-bit quantization. It can't fit in the 4090's twenty-four gigabytes. Three ways to run it on Sam's machine.

Maya CPU only: thirteen tokens a second generating, eighty reading.

Theo The traditional way: put as many whole layers on the GPU as fit, nineteen layers here, adding about nineteen and a half gigabytes of video memory use, with the rest on the CPU. Twenty point five tokens a second generating, three hundred and twenty-seven reading.

Maya And the expert split: one flag in llama.cpp that keeps every routed expert tensor in system RAM and puts everything else on the GPU. Thirty-nine tokens a second generating, two hundred and fifty-eight reading. Adding under three gigabytes of video memory use.

Theo Twice the generation speed of the layer split, using about a seventh of the video memory. That fits the mechanism: every token touches every layer's attention, which now runs on the fast side, while each token reads only a few experts from the slow side. But we moved several kinds of tensor at once, so this run doesn't tell us how much each move contributed.

Maya And don't read it as "a three-gigabyte card would do". The 4090's compute and bandwidth are still doing the attention.

Theo One puzzle, if you're applying our rule. This eighty-B model and the thirty-B one from earlier both have about three billion active parameters. On the CPU alone the thirty-B did twenty-six tokens a second and this one thirteen. Same active size, half the speed.

Maya Two differences. The thirty-B result used eight threads, and this one used twenty-four, which made the thirty-B slower too when we tried it. And Qwen three Next uses a newer hybrid attention design, which may simply be less optimised on CPUs so far. We didn't isolate which factor matters more. The bandwidth rule is an upper bound, not a forecast.

Theo Back to the eighty-B comparison. Notice the layer split still won on prompt processing, three twenty-seven against two fifty-eight, at our five-hundred-token prompt. A plausible reason: when many tokens hit each expert, having a large share of the experts on the card pays off.

Maya Same lesson as Fiddler: the right placement depends on which job you're doing.

Theo And all of this is one machine, one model, one set of runs. It's a demonstration, not a ranking.

Embeddings: Sentence-BERT, MiniLM, MTEB, static embeddings, Matryoshka, Qwen3 Embedding, and what quantization breaks 1:10:54

Maya Everything so far has been about generating text. But a lot of practical AI on ordinary machines doesn't generate anything. It turns text into vectors: search, retrieval for RAG, deduplication, clustering, classification. That's embeddings, and it's the area where "do I need a GPU?" has the most surprising answer.

Theo Start with why embeddings exist in their current form. Sentence-BERT, Nils Reimers and Iryna Gurevych, twenty nineteen. Before it, if you wanted to find the most similar pair among ten thousand sentences with BERT, you fed every pair through the model together. That's about fifty million forward passes, which they estimated at about sixty-five hours on a V-one-hundred.

Maya And with Sentence-BERT?

Theo You encode each sentence once into a vector, then compare vectors with cheap cosine similarity. About five seconds to embed, and a hundredth of a second to compare. Encode each item once and reuse the result, instead of re-reading pairs.

Maya And they measured CPU speed too, which makes a nice baseline for our own numbers.

Theo On a six-core Intel i7 from twenty fourteen, Sentence-BERT base embedded forty-four sentences a second. With what they called smart batching, grouping sentences of similar length so you don't waste work on padding, eighty-three. The V-one-hundred did fourteen hundred to two thousand.

Maya So in twenty nineteen the GPU was twenty-five to thirty times faster at this. The next step was making the models smaller. MiniLM, from Microsoft, twenty twenty: distil a big model into a small student by teaching it to copy the teacher's last-layer attention patterns. The six-layer version is the basis of all-MiniLM-L6-v2, one of the most downloaded embedding models there is. One telling detail from their size table: their smallest variant has five million parameters doing the actual transformer work and eleven point seven million in the word-embedding table.

Theo So the table of word vectors is twice the size of the network.

Maya Then the question of which model to pick. MTEB, the Massive Text Embedding Benchmark, Niklas Muennighoff and colleagues at Hugging Face, twenty twenty-two. Fifty-eight datasets, a hundred and twelve languages, eight kinds of task. Its headline finding is in the abstract: "no particular text embedding method dominates across all tasks." Models tuned for sentence similarity did badly at retrieval. And a model almost fifty times smaller than the leader matched it on clustering.

Theo So the leaderboard rank is not the thing to optimise. The task is.

Maya Now our measurements. Two thousand real Wikipedia paragraphs, a hundred to two hundred tokens each. Eight threads on the performance cores. MiniLM in plain thirty-two-bit PyTorch: two hundred and twelve passages a second. Through Intel's OpenVINO with eight-bit weights: four hundred and sixty-two. The GPU, in sixteen-bit: about three thousand seven hundred.

Theo So the GPU is eight times faster for bulk embedding. A smaller gap than in twenty nineteen, partly because the model is much smaller and the CPU much newer.

Maya Now the single query, which is what a search box actually does: one short question, embed it, look it up. MiniLM on the CPU with OpenVINO: one point one milliseconds. On the 4090: one point eight.

Theo The CPU wins.

Maya For a single query, yes. At that size the GPU almost certainly spends most of its time launching work and moving data rather than doing arithmetic. These timings include tokenising the text and getting the vector back, which is what a real search box pays. The same was true for BGE small, two milliseconds against two point eight.

Theo So for the query side of a search system, small embedders don't need a GPU at all.

Maya But the size of the model matters enormously. Qwen three Embedding point six B, six hundred million parameters and much stronger on benchmarks, embedded five passages a second on our CPU in thirty-two-bit, thirteen with OpenVINO. The GPU did three hundred and eighty-nine. Our two thousand passages took almost seven minutes through the CPU's thirty-two-bit path, about two and a half minutes through OpenVINO, and five seconds on the card.

Theo Is Qwen's embedder even the same kind of model as MiniLM?

Maya Not really, and its technical report, from Alibaba's Qwen team in twenty twenty-five, explains how. It's built from a Qwen three language model, a decoder that reads left to right, and it takes the internal state at the final end-of-text token as the embedding. It comes in point six, four and eight billion parameters, with vectors from a thousand to four thousand dimensions and up to thirty-two thousand tokens of context. Their ablations are worth knowing: for the point six B, training without their synthetic data cost about three points on their multilingual benchmark and five on English, and skipping their checkpoint merging cost about two. And their smallest reranker, the slower model that re-scores the top results, didn't help everywhere. On code it scored seventy-three against the embedder's seventy-five on its own.

Theo So bigger and more elaborate isn't automatically better, even inside one paper.

Maya And Han's quantization paper, which we'll come to, tested the point six B. At four bits it kept all of its retrieval quality, at three bits about ninety percent, and at two bits one percent. A cliff, not a slope.

Theo So "you need a GPU for RAG" is half right. For queries with small models, a CPU is plenty. For bulk indexing with a big embedder, the GPU saves a lot of time. Whether you need it depends on how big the collection is and how long you can wait.

Maya Then there's the extreme option: remove the transformer entirely. In January twenty twenty-five Tom Aarsen at Hugging Face published static embedding models. Each token gets a fixed vector from a lookup table, and you average them. No attention, no context. The author's own numbers: on CPU, about a hundred and seven thousand query embeddings a second, against two hundred and seventy for all-MPNet-base, the model it's compared with.

Theo And quality?

Maya Eighty-seven percent of MPNet's retrieval score on their benchmark. And in their test the static model was quicker on the CPU than on the GPU, a hundred and seven thousand against ninety-seven thousand. Our passage benchmark didn't reproduce that: about five thousand passages a second on the CPU against seven thousand on the GPU. But that's close for a CPU, and for single queries the CPU was slightly quicker. Our quality check was the SciFact retrieval task. Static scored nought point five nine, MiniLM nought point six four five, BGE small nought point seven.

Theo So about twenty-four times the throughput of plain PyTorch MiniLM on the CPU, or eleven times the faster OpenVINO version, at a real cost in quality.

Maya And the vectors themselves can be shrunk too. Matryoshka Representation Learning, Aditya Kusupati and colleagues, twenty twenty-two: train the model so the first few dimensions of its embedding are useful on their own. Then search with a short prefix and re-rank the finalists with the full vector. In their image-retrieval experiment, a shortlist of two hundred found using sixteen dimensions, then re-ranked using two thousand and forty-eight, was a hundred and twenty-eight times cheaper in theory. In practice, fourteen times faster on the same CPU search setup.

Theo Why the gap between a hundred and twenty-eight and fourteen?

Maya The theory counts only the arithmetic of comparing vectors. Real search also pays for the index, memory access and the re-ranking step. Hugging Face's embedding quantization post, twenty twenty-four, applies the same idea to precision. Turn each dimension of the stored vectors into a single bit. Thirty-two times less storage. Exact search on CPU was about twenty-five times faster. Retrieval quality kept about ninety-two and a half percent, and about ninety-six and a half once you re-score the top candidates with the full vectors. That's for the model they focused on. Another model in their table kept only three quarters of its quality with binary vectors, so check yours.

Theo Now the warnings, because quantizing the model, as opposed to the stored vectors, has subtler costs.

Maya The sharpest is a twenty twenty-six paper, "The Undetected Damage of Quantization on Retrieval", by Luca Zhou and colleagues. They show a model's benchmark score can barely move while its answers change underneath. In one RAG experiment, exact-match accuracy stayed at forty-two point seven percent after four-bit quantization. Five hundred and eighty-nine of two thousand two hundred and fifty-five answers changed anyway.

Theo Same score, a quarter of the answers different.

Maya Their explanation is neat. A classifier is trained to push the right answer far ahead of the second-best, so small rounding errors don't flip it. Retrieval has no such pressure. The top document and the runner-up can be nearly tied, and the top result is only guaranteed to survive quantization if the gap between them is more than twice the largest shift quantization makes to any score.

Theo And a companion paper, Hyojung Han's, also twenty twenty-six, tested whether the rules of thumb from LLM quantization carry over to embedders. They mostly don't. Protecting the embedding table, which you'd expect to matter, never turned out to be the top priority. At two bits, one model kept sixty-six percent of its retrieval quality, one kept forty-five, and three others kept one or two percent. And the score could barely change while thirteen to twenty-two percent of the top ten documents were replaced.

Maya And on the CPU specifically, eight-bit isn't automatically faster. Yuyeong Shin's paper "Is int-eight portable?", twenty twenty-six, ran the same quantized image model across devices. On CPUs with dot-product instructions, eight-bit was up to two point one times faster. On a Core i9 ten-nine-hundred-K, which lacks the relevant instructions, it was one point seven six times slower than thirty-two-bit. We saw a version of that ourselves. MiniLM quantized to eight bits through ONNX Runtime ran at a hundred and forty-three passages a second, slower than the two hundred and twelve in plain thirty-two-bit. The same model through OpenVINO's eight-bit path ran at four hundred and sixty-two. Same chip, two eight-bit deployment paths, more than three times apart. They differ in engine, quantization method and scheduling, so it's not one kernel choice. It's why you benchmark the whole software path.

Theo And on big server chips, Intel's own twenty twenty-six paper on eight-bit BERT in stock PyTorch is a good reminder to pick the right baseline. On a Granite Rapids Xeon with AMX, eight-bit BERT-large was five point eight times the throughput of thirty-two-bit. But only one point six times that of sixteen-bit, which the chip also accelerates. And for DistilBERT with dynamic quantization, the matrix multiplies got forty-five percent faster while the whole model ended up slightly slower than sixteen-bit.

Maya Faster matrix multiplication, slower application. The bottleneck moved somewhere else.

NPUs and unified memory: Gemma 3 on an AMD NPU, Panther Lake, Apple, Strix Halo 1:23:52

Theo Two more kinds of hardware come up whenever this topic does: NPUs, the neural processing units now in most laptop chips, and unified memory machines like Apple's Macs, where CPU and GPU share one large pool.

Maya NPUs first. Every laptop is now sold on its TOPS, trillions of operations per second. The best measured paper we found is from twenty twenty-six, by Shouyu Du and colleagues. They mapped Google's Gemma three onto the NPU in an AMD Ryzen AI seven three-fifty laptop chip and compared it with the same chip's CPU and integrated graphics.

Theo And the NPU wins?

Maya Depends on context length, and it's quite instructive. Gemma three four B with a short, one-thousand-token context: the NPU generated eighteen tokens a second, the integrated GPU about the same, the CPU fifteen. At a hundred and twenty-eight thousand tokens of context, the NPU held eleven, the CPU dropped to four, and the integrated GPU fell to under two.

Theo So the NPU's advantage is long context, where the other two run into trouble with the attention cache.

Maya With a caveat: the NPU ran the authors' own hand-written kernels, while the CPU and graphics ran Ollama and LM Studio. So it's chip against chip only roughly. And not everywhere. For the smaller one B model at thirty-two thousand tokens, the CPU beat the NPU, thirty-four against twenty-eight. Two other details: the four-bit weights get converted to sixteen-bit inside the NPU for the actual arithmetic. And their power-efficiency numbers leave out the memory controller and the rest of the board, so treat the energy claims as partial. They also note the CPU and integrated GPU ran at about a hundred degrees, while the NPU stayed below fifty.

Theo Which might matter more than speed on a laptop you're holding.

Maya And the paper explains why the NPU loses at short context. In their words: "The CPU and i-GPU get far more memory bandwidth, which explains their decoding speed advantage at short contexts." On that chip the NPU's share of memory bandwidth is capped below forty gigabytes a second. Short-context decode is a bandwidth contest, and the NPU enters it with a smaller pipe.

Theo And at long context?

Maya There, attention over the long cache is the cost, and their tiled kernel handles it better. Reading a thirty-two-thousand-token prompt into the one B model took seventeen seconds on the NPU, ninety-five on the integrated graphics and a hundred and sixty-five on the CPU. One caution: the paper's abstract claims bigger prefill speedups than its own tables show, so we'd quote the tables.

Theo Intel's side is thinner. A practitioner test of a Panther Lake NPU, using Intel's OpenVINO, got sixty-five tokens a second from the one-billion-parameter Llama three point two at four bits, on a short prompt. Sixteen- and thirty-two-thousand-token contexts failed to compile for the NPU at all.

Maya So, good at small models and short prompts on that chip, while the AMD study found the opposite strength. NPUs vary enormously, and the software matters as much as the silicon.

Theo That's the honest case for NPUs today. Similar speed for small models, cooler and quieter, better at long context in this one study. Not a replacement for a graphics card. Now a measurement that ties back to the start. A practitioner tested one of Intel's new Panther Lake laptop chips in September twenty twenty-six. Same model, same file, same llama.cpp build, once on the CPU and once on the integrated Arc graphics, which share the same memory. The model was a thirty-five-billion-parameter mixture of experts with three billion active.

Maya Same memory for both. So the decode gap should be small.

Theo The integrated GPU was six times faster at reading the prompt and only one point seven times faster at generating. With identical memory, the generating gap shrinks to under two, and the reading gap stays.

Maya That's the cold open with the memory difference taken out. The reading gap is about compute, and most of the writing gap was about memory. The leftover factor of one point seven is probably the CPU not quite saturating the shared memory while unpacking four-bit weights, the Lunar Lake effect from earlier. The blog's table is solid. Its own explanations, less so.

Theo Which is the real argument for unified memory. You get a GPU for the compute-heavy reading, and the whole memory pool for capacity. The numbers here are practitioner reports again. On an Apple M-three Ultra with five hundred and twelve gigabytes, one person reported DeepSeek V-three at four bits generating six tokens a second on the original release, and eleven on a later version with a newer attention setting and a shorter context. Its memory is rated at eight hundred and nineteen gigabytes a second. AMD's Strix Halo, the Ryzen AI Max plus three-ninety-five with a hundred and twenty-eight gigabytes, measured about two hundred and fifteen gigabytes a second against two hundred and fifty-six on paper, and runs a dense seventy-billion-parameter model at four and a half to five tokens a second. Nvidia's DGX Spark, also a hundred and twenty-eight gigabytes, measured around two hundred and thirty against two hundred and seventy-three on paper.

Maya On Macs, the rule runs into a limit. On the long-running llama.cpp benchmark thread, two Macs tested on the same old software build with the same small Llama two model: an M-four Max, rated at five hundred and forty-six gigabytes a second, and an M-three Ultra, rated at eight hundred and nineteen. The Ultra, with twice the graphics cores, read prompts about twice as fast. But it generated only about twelve percent faster, ninety-two tokens a second against eighty-two, despite half as much bandwidth again.

Theo So past a certain point, the kernels and the chip's ability to use its bandwidth set the fraction you get, not the rating.

Maya Strix Halo is a cleaner check of the bandwidth rule. A seventy B model at four bits is about forty gigabytes. Two hundred and fifteen divided by forty is about five and a half.

Theo And it measured four and a half to five, depending on which graphics backend was used. Close, and a reminder that software alone moved it by ten percent. These are CPU-and-GPU combinations with three to thirteen times a desktop's memory bandwidth and enough capacity for big models.

Memory generations: channels beat generations, latency, DDR6, "DDR7", HBM4 and the RAM price shock 1:31:15

Maya Which brings us to the question in the title: RAM. If bandwidth is everything, what do DDR3, four and five actually give you? My guess is that each generation roughly doubles it, and that's the whole story.

Theo The first half is right. A normal memory channel is sixty-four bits wide, eight bytes, and bandwidth is transfers per second times eight bytes. DDR3 sixteen hundred gives twelve point eight gigabytes a second per channel. DDR4 thirty-two hundred, twenty-five point six. DDR5 sixty-four hundred, fifty-one point two. One detail trips people up: each DDR5 stick is split internally into two thirty-two-bit sub-channels. That helps efficiency, but it doesn't double the width. A DDR5 stick is still sixty-four bits of data.

Maya And the second half, that the generation is the whole story?

Theo That's the wrong part. Almost every desktop and laptop has two channels, however many sticks you install. So a dual-channel DDR3 machine from around twenty twelve has about twenty-five gigabytes a second on paper. DDR4 thirty-two hundred, about fifty. Fast DDR5, about a hundred. Real systems get less. Sam's machine measured sixty-four. The Intel ternary paper from earlier measured ninety-eight on a Core Ultra desktop, though it doesn't say what speed the memory ran at, so don't read that as a percentage.

Maya In tokens, using our rule, for an eight-B model at four bits, about five gigabytes: roughly five tokens a second on the old DDR3 machine, ten on DDR4, twenty on fast DDR5. And for a seventy-B model, about forty gigabytes, under one, one and a quarter, two and a half. So no desktop memory generation makes a big dense model comfortable to chat with.

Theo Now a server. AMD's current EPYC has twelve DDR5 channels per socket, at up to sixty-four hundred. Twelve times fifty-one point two is about six hundred and fourteen gigabytes a second, per socket. The eight-B ceiling becomes about a hundred and twenty tokens a second and the seventy-B ceiling about fifteen. Against the DDR3 desktop that's twenty-four times the bandwidth. Four times comes from the generation and six times from having twelve channels instead of two.

Maya But we heard earlier that a Xeon with twelve channels per socket ran DeepSeek at five tokens a second. If the ceiling is six hundred gigabytes a second, why isn't it faster?

Theo Do the arithmetic first. DeepSeek's own report says it activates about thirty-seven billion parameters per token, so at eight bits each token reads nearly forty gigabytes. Six hundred divided by forty is a ceiling of about fifteen tokens a second, and that run got about a third of it. Part of the gap is threads, sockets and page placement, which we saw can swing results by almost a factor of two. And part is that big CPU servers need a lot of concurrency to keep twelve channels busy. The Roofline paper warned that sustained bandwidth drops when there isn't enough of it. More channels make more bandwidth available. Using it is a separate problem.

Maya Is there independent evidence that the channels themselves matter that much?

Theo Phoronix ran the same EPYC processor with eight channels populated and then twelve. On a memory-bound benchmark the twelve-channel configuration scored fifty percent higher. Not an AI benchmark, but the same mechanism. Intel's Xeon six adds MRDIMMs, multiplexed rank modules. A buffer on the stick runs two ranks of chips together, so the bus to the processor delivers eighty-eight hundred mega-transfers a second from chips that individually run slower. Against the sixty-four-hundred modules it replaces, that's about forty percent more bandwidth on paper. Phoronix measured a third more on a memory-bound benchmark, on a server Intel supplied. It's still DDR5, not a new generation. The same goes for CAMM-two, the flat module you may have seen in new laptops. It's a new shape for the module, carrying DDR5 or LPDDR5, not a new kind of memory.

Maya What about latency? I'd assume that's improved too.

Theo Hardly at all. Chips and Cheese, who measure this carefully, found DDR5-six-thousand on a Ryzen nine ninety-nine-fifty at just over seventy nanoseconds to reach memory. About the same as DDR3 on an Intel Haswell desktop from twenty thirteen. Bandwidth has multiplied several times since then, and the time for one trip hasn't really changed. Under heavy contention, another test found it going past four hundred nanoseconds.

Maya Which ought to hurt, if every fetch of a weight pays seventy nanoseconds.

Theo Less than you'd think for LLM decode. Decode reads weights in long sequential sweeps, so the hardware prefetchers see it coming and keep many requests in flight. It's close to the least latency-sensitive access pattern there is. Wulf and McKee's arithmetic was about latency, with saturated bandwidth as their starting condition. For a language model generating text, bandwidth is the wall.

Maya Now the question in Sam's brief: DDR6 and DDR7.

Theo DDR6 isn't a product yet. As of October twenty twenty-six, JEDEC, the standards body, lists it as "in development". Industry estimates put commercial availability around twenty twenty-eight or twenty twenty-nine. And there is no DDR7. What exists is GDDR7, graphics memory, the kind soldered onto an RTX 5090. Thirty-two gigabytes on a five-hundred-and-twelve-bit bus, about eighteen hundred gigabytes a second. Four times the bus width of a dual-channel desktop, at a much higher rate per pin. The seven means something completely different there.

Maya So is HBM just more of the same with more pins?

Theo It's the same basic DRAM cells, packaged very differently: stacks of chips placed right beside the processor with an enormously wide connection. The previous generation, HBM3-E, delivers a bit over one point two terabytes a second per stack, according to Micron and SK Hynix. The HBM4 standard allows up to two terabytes a second per stack. Nvidia's Vera Rubin GPU uses HBM4, up to two hundred and eighty-eight gigabytes and a quoted peak of twenty-two terabytes a second per GPU. About three hundred and forty times what we measured on Sam's desktop. On the low-power side, LPDDR6 was standardised in July twenty twenty-five, and the Chinese manufacturer CXMT reports it shipping in a Xiaomi foldable phone at about ten and a half thousand mega-transfers a second.

Maya And then the part nobody planned for. Buying more RAM got expensive.

Theo Very. Tom's Hardware's survey on the fourteenth of October twenty twenty-six found the cheapest thirty-two-gigabyte DDR5-six-thousand kit they tracked at four hundred and fifty dollars. In August they'd found a mainstream Corsair kit of that size and speed at four hundred and one dollars, against a historical low of around eighty.

Maya So about five times its old price.

Theo TrendForce, which tracks the memory market, attributes it mainly to supply. Memory makers are moving production capacity to HBM and server memory, cutting shipments for PCs and module makers, and winding down DDR4. By July twenty twenty-five, DDR4 contract prices had already risen by more than sixty to eighty-five percent, and eight-gigabyte DDR4 modules were selling for more than DDR5 ones. Their forecasts for this year were for conventional memory contract prices to rise by roughly ninety percent in the first quarter and sixty in the second. Those are forecasts made during negotiations, not measured prices.

Maya So the AI boom made the GPUs expensive, and then made the memory you'd use instead of GPUs expensive as well.

Theo There's a line near the end of Wulf and McKee's note that fits. They point out that DRAM keeps giving us plenty of space, and what's scarce is speed. Thirty years later the industry is spending enormous sums to buy speed, in the form of HBM, and plain capacity has got expensive as a side effect.

Back to your machine: eight rules, and what our measurements don't show 1:40:36

Maya Let's make this practical. Somebody has a desktop with a decent CPU, plenty of RAM, maybe a graphics card. What should they actually do differently?

Theo First, work out your ceiling before you download anything. Find your real memory bandwidth, from STREAM or a similar benchmark, not the number on the box. Divide it by the gigabytes of weights each token has to read. For a dense model that's the file size. For a mixture of experts, it's the shared part plus the experts that are active. That number is your best case for generation. On our machine the dense model came within a few percent of it, and the thirty-billion-parameter mixture of experts within about fifteen.

Maya Second, threads. Don't assume all of them. On a chip with performance and efficiency cores, start with one thread per performance core and test a few counts around that. On ours, four to sixteen threads were all within about a fifth of each other, and all thirty-two was five times slower.

Theo Third, on a CPU, prefer mixtures of experts. System RAM is cheap capacity and limited bandwidth, and a model with a large total and a small active part fits that exactly. Our thirty-billion-parameter MoE generated twice as fast as a dense eight B.

Maya Fourth, if you have a GPU that's too small for the model, don't split by layers by default. Keep attention and the shared weights on the card and the routed experts in system memory. In llama.cpp that's the tensor-override flag that sends expert tensors to the CPU, or its shorthand options for MoE offload. On our eighty-billion-parameter test that doubled generation speed and used about a seventh of the video memory. It's the same idea as Fiddler, PowerInfer and KTransformers: move the small activations, not the big weights.

Theo Fifth, watch prefill. It's the number the forum posts leave out, and the one that decides whether a CPU setup is pleasant. Long documents, RAG context and agents that re-send thirty thousand tokens every turn are where CPUs struggle. Prompt caching, which avoids re-reading a prefix you've already processed, helps a lot there. The documentation for one such system, LM Cache, shows a fifteen-thousand-token prompt taking six and a half seconds to first token cold and about a seventh of a second when the prefix was cached. That's their example, not our measurement. So does keeping prompts short.

Maya Sixth, embeddings. For the query side of search, small embedders on the CPU are as fast as a GPU or faster, around a millisecond. For bulk-indexing a large collection with a big, high-quality embedder, a GPU is worth it. In between there are levers: static embeddings when speed matters more than the last few points of quality, and Matryoshka truncation or binary vectors with re-scoring for cheaper search. And when you quantize an embedder, check whether the actual top results changed, not just the benchmark average.

Theo Seventh, never assume lower precision is faster on a CPU. It depends on the instructions your chip has and the kernels your software picks. We saw eight-bit run slower than thirty-two-bit on one runtime and twice as fast on another, on the same chip. Measure.

Maya And eighth, if you're buying hardware for this, think in channels and capacity. A twelve-channel server, a high-bandwidth unified-memory machine and a desktop with two DDR5 sticks belong to different categories, whatever generation the memory is. For big mixtures of experts, capacity decides what you can run and bandwidth decides how fast.

Theo And a word on our own numbers, since we've leaned on them. Every test was one run set on one machine. Prompt processing and generation were separate synthetic tests with random tokens, so generation started from an empty context rather than after a real prompt. We didn't lock clock speeds. And we couldn't read the memory's actual speed or channel layout, only measure what it delivered. They illustrate the mechanisms the papers describe. They don't rank hardware. So, to the question in the title. What can your CPU and RAM do with AI?

Maya More than most people assume. Generation at a respectable fraction of a GPU, limited by memory bandwidth you can calculate in advance. Mixtures of experts far bigger than any consumer card can hold. Search and embeddings at GPU-class latency. And as the other half of a hybrid, holding the experts while the card does attention.

Theo What it can't do is read quickly. The hundred-times prefill gap from the cold open is real, and nobody has a trick that removes it. Every clever system in this episode either avoids moving weights, or overlaps the work, or accepts a long wait before the first token.

Maya Which is what Wulf and McKee were telling everyone in nineteen ninety-four. Arithmetic keeps getting cheaper, and getting the data to it is the hard part. They ended their note by saying their prediction was probably wrong too. For general-purpose computing they may have been. For a language model generating one token at a time, they look to have been about right.

Theo That's the episode. The papers, and our own measurements on Sam's machine, are linked in the show notes.