Making a Chip Fall Out of the Workload

01 OpenAI built a weird inference chip

In June 2026, OpenAI and Broadcom announced a chip called Jalapeño, and at Hot Chips they showed just enough of it to drive me a little crazy. The headline numbers are enormous but honestly boring: PUBLIC FACT 13.4 petaflops, 216 GB of memory at 15.4 TB/s, 700 watts.¹ Every flagship accelerator has numbers like that.

What got me was the shape. The flagship AI accelerators are, with few exceptions, organized like GPUs: one big shared pool of memory, big caches, huge matrix engines, everything built for throughput. PUBLIC FACT Jalapeño is described as the opposite on nearly every axis: lots of small compute "slices," each hugging its own private slab of memory. No big shared hierarchy. Software tells every tensor exactly where to live. OpenAI even named their villains out loud: "contended unified memory subsystems" and "expensive global memory fences."²

And it apparently works: they published benchmarks claiming 1.7x lower latency and 1.9x better tokens per watt than NVIDIA's best. PUBLIC FACT Vendor numbers, sure. But specific ones.

The sharpest outside reading of the chip, and the main inspiration for this piece, is zartbot's teardown.³ Their one-line thesis stuck with me: architecture determines end-to-end latency. Not kernels, not clever software. The floor on how fast a token can arrive is set by the shape of the machine, and this machine's shape was chosen for exactly that floor.

Between the Hot Chips talk and that teardown, the picture is more detailed than you'd expect for a secret chip. Worth laying out, because every one of these becomes a plot point later:

So here was the question I couldn't put down. When a smart team breaks all the conventions and wins, there are two possibilities: either they know secrets, or the conventions were wrong for this problem and anyone optimizing honestly would land in the same place. Those two possibilities are distinguishable! You don't need their blueprints. You need to optimize a chip from scratch, without peeking, and see what falls out.

That's the whole project: try to make Jalapeño fall out of the workload. To spoil the ending just enough: I didn't reconstruct their chip. But one architectural choice kept reappearing, strongly enough that I ended up testing it all the way down in synthesizable Verilog. The evidence ladder this piece climbs: a calibrated analytical simulator, a 3,400-design blind search, 81 physical-cost regimes, 32 different front ends, an independent cycle-level simulator, and finally RTL.

02 Why GPUs are weird for batch-1

First, why would anyone build an anti-GPU? zartbot has the frame I keep coming back to: a CPU minimizes the latency of one thread, a GPU hides latency behind a crowd of threads. Both tricks are real, and single-user inference defeats both. It has one thread's worth of urgency and a supercomputer's worth of arithmetic, so it sits in an awkward middle that neither machine was built for.

The hiding trick is the one that fails interestingly. A GPU tolerates slow memory, slow launches and slow synchronization because it always has ten thousand other things to work on during the wait. Generating text for one user, there is nothing else to work on: tokens come out one at a time, and each token is a strict chain of steps. Every overhead the crowd used to absorb now lands directly on the time between your tokens.

It's worth being concrete about what that chain actually is, because everything later depends on its shape. Take gpt-oss-120b, the open model OpenAI used in its benchmarks. PUBLIC FACT One token means: for each of 36 layers, project the current state into queries, keys and values; run attention, which rereads the stored keys and values of every previous token in the conversation (the KV cache, and it grows as the chat gets longer); project back; then a router picks 4 of the layer's 128 experts and the token's state runs through those two-matmul experts. Do all of that, in order, roughly 330 matrix operations, and you have produced one token. Now do it again.

That expert detail matters more than it looks. The model's weights are enormous, but each token only touches about a thirtieth of them, and mostly a different thirtieth than the token before. There is no reliably small hot set of expert weights for a cache to hold. Which means the weights can't sit near the compute; they have to stream past it, from big memory, every single token.

And you can put a floor on the whole game with one division. gpt-oss-120b activates about 5.1 billion parameters per token PUBLIC FACT; at 4-bit weights that's roughly 2.6 GB that must cross the memory pipe per token. Divide by the pipe: 2.6 GB ÷ 15.4 TB/s ≈ 170 microseconds. That's the physics floor, before a single cycle of overhead; a perfect machine tops out near 5,900 tokens/s for one user. My simulated reconstruction reaches about 40% of that floor and a GPU-style design about 16%, and the rest of this piece is really about where those missing fractions go:

There's a second problem, and it ends up being the core of this whole investigation. The matrix units in these chips are built wide: a 128x128 engine wants 128 rows of input to fill every lane. A one-user model hands it a single row of activations per step. Feed one row to a unit whose natural tile is 128 rows wide and, that cycle, most of its row capacity has nothing useful to do. This mismatch between unit width and batch size is the variable everything later turns on.

The obvious rebuttal is: so batch. Stack many users' rows into a block and the wide unit fills up again. That works, and it is exactly how GPUs serve at scale. But it trades one person's speed for everyone's throughput, and my own simulations show the two philosophies splitting cleanly along that line:

03 The physics nobody gets to negotiate with

Before simulating anything, it's worth knowing which walls are load-bearing, because two pieces of physics explain most of this chip's personality.

First: arithmetic is nearly free, and moving data is not. In the energy model I use (Horowitz-style coefficients scaled to a 3 nm-class process MY MODEL), one FP8 multiply-accumulate costs about a quarter of a picojoule. Fetching one byte from HBM costs about forty. That's a 160:1 gap, and it turns power budgets into geometry: the joules go to distance traveled, not math done.

Run the two headline numbers against the 700-watt budget and something jarring falls out. Streaming memory at the full 15.4 TB/s costs roughly 15.4e12 bytes/s × 40 pJ, which is about 600 watts: using the memory system flat-out nearly is the power budget. Meanwhile 13.4 petaflops of FP4 at even a tenth of a picojoule per op would be around 2,000 watts, three times the envelope. So under this energy model, the brochure compute number isn't a simultaneously sustainable operating point; the real design target has to be what the chip can hold. zartbot makes the same point from the other side: this chip is designed for sustainable FLOPs at batch 1, not peak FLOPs on a slide. Decode is a power-bound data-movement problem; the compute is almost incidental.

Second: on a chip this size, "global" is a place, and it's far away. At 1.7 GHz a clock cycle is 588 picoseconds, and a signal in on-chip wiring covers roughly a millimeter or two in that time. A reticle-sized die is about 26 by 33 millimeters. Anything global, one shared cache, one hardware scheduler, one chip-wide fence, is physically many cycles away from most of the silicon, before any queueing even starts. Local memory and local synchronization aren't an aesthetic; they're what the speed of a signal in a wire leaves on the table. My own RTL later measured the miniature version of the same law: every doubling of a matrix unit's width added about four gate levels to its critical path, because bigger circuits are longer circuits.

Keep both laws in mind and the chip's whole silhouette starts looking less like invention and more like surrender, in the good sense: to move fewer bytes shorter distances, and to never make the whole chip wait on a signal crossing it.

04 Can you rediscover a chip?

You can't x-ray the chip. But you can x-ray the problem, and the problem is fully public: the models are open, their tensor shapes are known, the physics of memory bandwidth and silicon area are in textbooks. So the plan became a game with three rules.

Rule one: keep an honest ledger. Every fact about Jalapeño gets a tag, PUBLIC FACT, EXTERNAL INFERENCE, or my own guess, with a source. No quietly promoting a rumor into a spec. Rule two: build a simulator of imaginary chips. A model where I can dial every design knob, slice count, matrix unit size, memory organization, sync cost, and it estimates speed, power and area on real open models. MY MODEL Rule three, the important one: the search never gets to see the answer. An optimizer explores thousands of chip designs scored only on "make one user's tokens fast, cheaply, within a power budget." Whatever the winners look like, that's the data.

Does a homemade simulator deserve any trust? Only if it can predict chips that exist. Pointed at an H100 built from spec-sheet numbers, it predicted single-user speeds inside the published range. I even wrote down predictions for hardware I hadn't checked yet, committed them to git, and then looked up the real measurements. One prediction missed badly, in an instructive way I'll get to.

A confession about method: this whole project is one long exercise in trying not to fool myself. I wanted Jalapeño to be derivable. Wanting a result is exactly how you fabricate one by accident, so every stage below exists to attack the stage before it.

05 The first result looked suspiciously good

MY MODEL The first thing I checked: if I build a chip in the simulator using only what OpenAI disclosed, does it beat a GPU-style design with identical raw specs? It did, by 2.5x on speed and 1.85x on energy. OpenAI's own claims are 1.7x and 1.9x. I never tuned toward their numbers, and landing that close on the first try made me suspicious rather than happy. Models that agree with you immediately are usually flattering you.

Then the real event: I released the search. 3,400 candidate chips, scored blind; 2,090 fit the area and power budget. Here is the whole design space, one dot per chip:

Afterwards I checked the winners for Jalapeño's fingerprints: did the best designs choose local memory? Fine-grained compute? No big cache? Read the grey bars below as chance, what a random viable chip looks like, and the blue bars as what the winners actually chose:

The SRAM rejection stopped being a puzzle once I remembered the experts. On-die memory is a bet that some working set is small and hot, and a mixture-of-experts model at batch 1 refuses to have one: a different sliver of 216 GB streams past every token, so silicon spent remembering weights is silicon that mostly remembers the wrong ones. The search figured that out on its own and spent the area on compute instead. (Whatever the real chip's "large SRAM" is for, my model says single-user expert weights isn't it; staging traffic across a 128-chip pod is a better guess, and out of scope here.)

06 Trying to make it disappear

A result this convenient is exactly the kind you should try hardest to break. So I spent the next stretch attacking it: sweeping every guessed constant from pessimistic to optimistic, swapping in rival cost models, re-running the search each time.

Mostly it held. One experiment didn't. In my original model, choosing "local memory" automatically came bundled with cheap synchronization, because that's how OpenAI describes their machine. Bundled variables are dangerous: maybe the search never cared about local memory at all and was only ever selecting the fast sync attached to it. So I unbundled them. Cheap sync became its own design choice, priced in silicon, available to any memory style.

MY MODEL The local-memory preference collapsed on the spot. The winners kept buying fast synchronization, even at full price, and stopped caring about the memory layout it used to come wrapped in. The search never wanted Jalapeño's memory diagram. It was selecting cheap coordination the whole time; the memory layout just happened to be attached.

This was the low point of the project and also the moment it became worth writing about. Watching the search decline to want the thing I had built the whole story around felt bad for about an hour. Then it felt like the experiment working.

07 The thing that wouldn't die

Meanwhile, one result refused to die no matter what I threw at it: many small matrix units beat a few big ones. Every time. The cleanest way to see it is one-variable breakage: take the reconstruction, flip exactly one design choice back to the GPU way of doing it, and measure how much slower one user's tokens get:

Read that chart as a ranking of what actually matters. Swap the entire memory architecture: barely a ripple. Make the compute chunky or the coordination slow: catastrophic. And the bottom row is the punchline: going even finer than my own default made things faster.

zartbot's teardown reads the same lesson straight out of the hardware EXTERNAL INFERENCE: they infer matrix units built to stay efficient all the way down to a single row of work, where GPU tensor engines fall off a cliff below a batch of 64. A machine designed so that the smallest possible piece of work is still a first-class citizen.

08 Why not make arrays tiny?

Okay, but if small units are good, why not make them tiny? Thousands of 16x16s? Because every unit, however small, carries fixed costs that don't shrink with it: a control sequencer, operand wiring, accumulator ports, a slot in the instruction stream. Halve the arithmetic and those stay roughly constant, so their share of the silicon doubles. Somewhere below 32, a "matrix unit" is mostly plumbing.

I rebuilt the cost model around exactly those physical taxes and swept them across 81 combinations, deliberately including ranges hostile to my own result. MY MODEL Every single one produced a valley: tiny units lose, huge units lose, and the floor sits between 32x32 and 128x128, with 64x64 the most common winner. The teal cells are fast, the faded ones slow; the valley is easy to spot:

Why do the huge units lose? This one you can just see. Below, each grid is one matrix unit on one clock tick: blue rows are doing real work for your tokens, grey rows are idle silicon. Try the batch buttons:

And the tiny units lose to those fixed costs, plus one more: a chip carpeted in thousands of small units needs more instructions issued per cycle than any front end can supply. Which raised an uncomfortable thought. That issue rate was my modeling assumption. Was my favorite number, 64, just an echo of it?

09 Change the front end

There's only one honest test for whether a result depends on an assumption: delete the assumption and see if the result survives. I threw away the instruction front end entirely and rebuilt it four different ways: a single strict dispatcher, a crowd of independent dispatchers, a SIMD-style broadcast where one instruction drives a whole group of units at once, and a compiler-choreographed machine with almost no dispatching at all. Thirty-two configurations, full search re-run under each.

MY MODEL The valley survived all four. And 64x64 revealed a property I hadn't gone looking for: it's the safe bet. Whatever front end you pick, choosing 64 costs at most 4% versus that front end's true ideal. Strict dispatcher? 128 is ideal. Wide and parallel? 32. Don't know yet? Build 64.

Here's the part that gave me a small chill. zartbot's teardown, working from die photos and area accounting rather than simulation, estimates the real chip at 64 slices of 15-odd matrix units each, at, yes, 64x64. EXTERNAL INFERENCE Their number comes from reading the silicon; mine comes from a blind search that was never told what the silicon looks like. Two completely different roads, same intersection. That's not proof of anything, but it's the closest this project gets to a handshake with the actual chip.

The revised understanding: issue bandwidth isn't the explanation, it's one of three forces. Tile quantization pushes fine, issue supply and per-unit overhead push coarse, and the optimum is where they intersect.

10 What I think the chip actually looks like

Nobody outside OpenAI has published a floorplan, so everything in this chapter is triangulation. But the triangulation is unusually satisfying, because the outside guesses factor the disclosed totals almost perfectly. Watch:

zartbot's teardown reads the die as 64 slices with about 15 matrix units each, at 64x64. Take that seriously and multiply it out: 64 × 15 = 960 units, each doing 64 × 64 = 4,096 multiply-adds per cycle, times 2 operations per multiply-add, times the disclosed 1.7 GHz. That's 960 × 4,096 × 2 × 1.7e9 ≈ 13.4 petaflops, the exact number on OpenAI's slide. Do the same for memory: 15.4 TB/s split across 64 slices is 240 GB/s per slice, which is precisely the private-lane figure the teardown estimates from the HBM wiring. Two independent reads of the silicon, and both divide cleanly into the two disclosed headline numbers. Guesses that factor like that are usually close.

So here's the machine I picture. A reticle-limited die, roughly 26 by 33 millimeters, tiled with 64 slices the way a city is tiled with blocks. Each block is a small, complete computer: one out-of-order scalar core doing the bookkeeping, a SIMD engine for the nonlinear odds and ends, and around fifteen 64x64 matrix units, all of them reading and writing one shared 512 KB L1. That sharing detail is easy to skim past and shouldn't be: on a GPU, the tensor and vector engines pass data through explicit staging and barriers, and here they'd just... look at the same memory. Down each edge, three HBM4 stacks, with each slice column wired more or less straight into the nearest one, which is what makes the 240 GB/s lanes private rather than shared. A thin dedicated network threads between the slices for the collective operations. And on the package, next to the big die, an IO die and two blank spacer dies whose whole job is mechanical: the lid has to sit flat on something.

One area cross-check, again from the teardown: the matrix units come out around 180 mm², about 22% of the die, which back-solves to a die in the 800 mm² range, right at reticle. (My search used a 900 mm² budget and landed on 64-wide units in 64-plus slices before I'd read any of this; the same constraints squeeze from both ends.)

What would break this picture? A disclosed matrix-unit shape that isn't 64-wide, a real L2, a slice count that doesn't divide the bandwidth into those lanes. All of it dies the moment OpenAI publishes another slide, which is the healthiest thing you can say about a guess.

11 Build another simulator

By this point every result rested on the same analytical formulas, and formulas can be wrong together. So I wrote a second simulator that shares nothing with the first, no utilization formulas, no rooflines, just discrete events: chunks of work fetch their operands, wait on finite memory bandwidth, queue behind each other, and finish when they finish. Performance emerges from the traffic instead of being computed from a model of it.

Embarrassing disclosure that belongs in the story: the first version announced that big units win everything, by miles. So did the second. Both times the cause turned out to be a shortcut in my own code that starved small units in a way no real chip would, a missing standard trick here, an unrealistically dumb queue there. Fix the shortcuts and the valley reappeared, 64 back on top. I include this because it cuts both ways: if I'd been rooting against fine-grained compute, I could have stopped at version one and written that essay instead. The bugs you stop hunting are the ones that agree with you.

MY MODEL The independent model confirmed the valley, and, more valuably, caught my formulas in two genuine errors. Where memory bandwidth is the only bottleneck, unit size stops mattering entirely, a penalty I had been charging big units simply evaporates. And for prompt processing, the winner depends on how fast each unit can be fed: starve them all from one shared memory and bigger units win; give every slice its own private feed and 64 wins again. Private local feeds per slice is, notably, exactly what OpenAI says they built.

12 Fine, build the hardware

Two agreeing simulators are still just my opinions, formalized twice. But by now the claims were sharp enough to test in the language chips are actually designed in. So I wrote one matrix unit in Verilog: size adjustable, 32, 64 or 128, identical design at every size so nothing gets favored, with a throttleable input feed, backpressure, and counters on everything. Then I ran slices of the real models' actual matrix operations through it, ninety runs, each checked against a reference calculation, plus synthesis to measure silicon area and timing.

The simulators had made two crisp predictions. A unit of width D should starve at exactly D bytes per cycle of feed. And 64 should beat 128 at equal silicon. Drag the feed and watch the first one come true:

RTL RESULT The starvation knee landed exactly where predicted, at every size, in every run, and the timing law matched within 1%. When your napkin math survives contact with Verilog, that's about as good as a home lab gets. The hardware also corrected me twice. The fixed control cost of a small unit turned out far smaller than I'd assumed, meaning I had been unfair to small units while trying to be unfair to no one. And big units pay a speed penalty I'd never modeled: wider arrays have longer internal signal paths, so they literally cannot be clocked as fast.

RTL RESULT The hardware also surfaced a tax I hadn't appreciated: padding. Real model layers aren't sized in tidy powers of two; one real slice of the attention math is 80 columns wide. A 64- or 128-wide unit must pad that out to its own width and fetch the padding from memory, wasted bytes on the busiest resource the chip has. 60% waste, versus 20% at width 32. Bottom line at equal silicon: 64 beat 128 on everything, at every feed rate. And 32 actually edged out 64, riding the padding tax and those cheaper-than-expected operators. So the final showdown, 32 versus 64, ends in a question my tools can't answer: it hinges on wiring congestion, which only real physical chip layout reveals.

13 What I think is actually true

Here's the whole investigation on one spine, each stage and what it did to my beliefs:

So here's the claim I'll stand behind. For serving one impatient user, matrix units at or below 64x64 beat the 128-and-up engines that throughput chips are built around, and that held across three independent methodologies and everything I could throw at it. 64x64 specifically is the choice that loses least no matter how the rest of the machine turns out. Whether 32 sneaks past 64 comes down to wiring costs none of my tools can see, so that door stays open on purpose.

The other survivor is less glamorous but bigger in the data: ceremony is the real tax on single-user inference. Hundreds of tiny steps per token, each paying a startup and coordination fee, and the chip that makes those fees nearly free wins whatever its memory diagram looks like. That, I suspect, is the actual soul of Jalapeño.

14 What I got wrong

• NUMA plus explicit placement as a discovered preference. It was a coupled constant wearing a result's clothing. The search wants cheap synchronization; the memory topology was along for the ride.
• "Smaller is always better" for matrix units. Refuted by my own physical-cost sweep: 16x16 loses everywhere it differs.
• Issue bandwidth as the explanation of the optimum. Removing it entirely left the basin standing; it is one force of three.
• The analytical coarse-unit penalty on memory-bound dense decode. The cycle model showed split-K hides it; the landscape there is flat.
• My analytical control-area cost for small arrays. Synthesis measured it 3 to 6x smaller than I had assumed. I was unfair to fine units while trying to be unfair to no one.
• Two versions of the cycle simulator. Both quietly rigged against fine granularity through modeling shortcuts. The bugs were the finding.
• A100 single-stream throughput. Overpredicted 2.5x, because real serving stacks pay software overheads my simulator does not model. Ironically this miss supports the thesis: the measured gap is exactly the per-op overhead class Jalapeño's design attacks.

15 What this says about OpenAI's models

Did I reverse engineer the chip? No. I still don't know their matrix-unit dimensions, their memory sizes, their timing, and nothing here recovers a single proprietary number. But the whole premise of this project was that a chip is a confession. Silicon takes years and billions; you only harden choices into it that you're sure about. So if the architecture really does fall out of the workload, then reading the architecture backwards tells you what workload OpenAI is betting on.

And every arrow points the same direction. Everything my experiments found inevitable about this design, fine-grained compute, cheap coordination, a fast private lane from every slice to memory, buys exactly one thing: the speed of a single stream of tokens. Not throughput. Not serving more users per chip. The time between one token and the next, for one conversation. PUBLIC FACT OpenAI's own framing agrees: sub-millisecond token-to-token latency, benchmarks led by reasoning models like DeepSeek R1, explicit talk of ultra-low-batch speculative decoding.²

Why would one stream be worth a custom chip? Because of where the models are going. A chat model emits a paragraph and stops; nobody notices 20 milliseconds per token. A reasoning model thinks in tokens, thousands of them, serially, before you see a word, and an agent stacks those chains end to end all day. At that point token latency stops being a UX nicety and becomes the rate limit on how fast the system can think. Three-x on one stream is a rounding error for a chatbot and a generational gap for an agent.

There's a second tell in the public material that I find just as interesting. The obvious way to build inference hardware in 2026 would be specialized fleets: prefill chips, decode chips, maybe draft and verify chips for speculative decoding. OpenAI says they considered exactly that and refused, because idle accelerators still burn power and the mix between those phases swings too fast to provision for; they built one balanced chip and gate the silicon they aren't using. PUBLIC FACT zartbot's phase analysis spells out the mechanism EXTERNAL INFERENCE: speculative decoding's acceptance rate moves around at runtime, so the demand for drafting and verifying oscillates against each other, and fixed pools would always have the wrong shape. Together the two decisions describe one workload: long serial chains of thought, with speculative decoding important enough to shape the hardware, and a phase mix too volatile to hard-partition.

None of that is in a model card. It's in the chip, which is the more believable place for it: model roadmaps are marketing until they ship, but nobody tapes out a 700-watt bet they don't hold.

What I actually took away from a weekend of trying to kill my own results: the interesting part of this design was never what OpenAI added. It's what they were sure enough to delete. The shared cache, the hardware scheduler, the whole apparatus of flexibility, each one removed because the workload, read honestly, doesn't pay for it. My search kept deleting the same things for the same reasons, without ever being shown the chip.

Turns out you can learn a lot from a machine you've never seen, if you take its problem seriously enough.

Sources

1. zartbot, "Jalapeño architecture analysis": the teardown that most shaped this piece. ↩
2. Tom's Hardware, Hot Chips 2026 coverage; OpenAI announcement. ↩
3. ServeTheHome, "OpenAI Jalapeno Custom AI ASIC at Hot Chips 2026"; The Register. ↩