Disaggregation solves one problem and exposes another

Separating prefill and decode into independent compute pools eliminates the resource contention that plagues coupled architectures, lets each pool scale independently, and allows you to tune batch sizes and parallelism strategies for the actual workload profile rather than a compromise between two conflicting ones.

However, disaggregation in this sense is really just a scheduling and orchestration problem and solving it does not depend on changing the hardware technology under the hood. It separates the workloads logically into two phases with fundamentally different compute and memory characteristics — but that logical separation alone does not solve the problem. The next step is to tailor the underlying technology to each phase independently, all the way down to the type of physics each technology relies on.

The prefill compute profile

During prefill, the model ingests the full prompt in a single forward pass. Every input token is processed in parallel across all attention heads and feed-forward layers. For the largest models with hundreds of billions of parameters, and context length in the hundreds of thousands of tokens, this means executing a sequence of extremely large general matrix multiplications (GEMMs) against weight matrices that do not change between requests.

The arithmetic intensity is high and so prefill is firmly in the compute-bound regime, meaning the bottleneck to improving performance is processing throughput, not memory bandwidth (you are not waiting for data to arrive from memory, but for the compute units to finish the multiply-accumulate operations.)

The parallelism structure is also highly regular. In dense layers, the same weight matrices are applied to every token in the batch and tiling strategies are relatively straightforward. There is no sequential dependency within a single prefill pass - tokens do not need to wait for each other, so this is a workload that rewards hardware designed to execute large, dense matrix operations at maximum throughput with minimal overhead per operation.  

MoE models add a layer of complexity due to the headache of routing tokens between different experts. Provided expert layers are large and concurrency is high enough to keep each expert well-batched, load balancing techniques and batching across concurrent requests mean the same compute-bound arguments apply.

To summarize, prefill means high arithmetic intensity, which needs maximum operations-per-second, efficient large GEMM execution, and lowest energy per multiply-accumulate.

The decode compute profile: a different problem entirely

Decode is sequential by definition. Each token depends on the KV cache entries produced by all previous tokens in the sequence. You cannot parallelize across output tokens the way you parallelize across input tokens in prefill. Each forward pass through the model produces exactly one token, and the dominant cost is reading the accumulated KV cache from memory.

The arithmetic intensity of decode is low. For a long-context request with a large KV cache, the ratio of compute operations to memory bytes loaded is well below the roofline ‘ridge point.’ The hardware is almost always waiting on memory, not on compute, so throughput scales with memory bandwidth not raw compute speed. Adding more compute units to a decode-bound system achieves very little.

KV cache size also scales with sequence length and batch size. For large batches of long-context requests, the working set can reach hundreds of gigabytes. The hardware for decode phase needs large, fast memory capacity, high bandwidth interconnect between compute and memory, and efficient cache management.

Decode wants: maximum memory bandwidth, large & high-bandwidth memory capacity, efficient irregular memory access patterns. A chip optimized for prefill will be underutilizing its compute units for most of its runtime.

Why silicon cannot optimize for both

Silicon architectures are designed around a tradeoff between compute throughput and memory bandwidth. The tradeoff point is fixed at design time. A given chip has a specific ratio of TOPS to memory bandwidth, and that ratio determines where on the roofline model it lands.

Prefill wants that balance tilted towards compute: it needs high operations per second and can give up memory bandwidth. Decode wants it tilted the other way: it needs high memory bandwidth and gains little from extra compute.

These demands pull in opposite directions, and a silicon chip offers only one balance point to meet them. Run both phases on the same processor, and you are left with a single compromise that suits neither; disaggregate but fill both pools with that same silicon, and you simply get two fleets of identically mismatched hardware. Either way the limitation is the chip, not the topology.

The right answer is not to find a better silicon balance point. The right answer is to use different hardware for each pool, chosen for the specific compute profile of the workload it runs. This is the real power behind disaggregated inference systems.

What purpose-built prefill hardware looks like

A hardware architecture designed specifically for prefill optimizes one thing: executing large, dense matrix multiplications at the highest possible throughput per watt. The access pattern is predictable, the operands are static (model weights do not change), and the parallelism is embarrassingly regular. This is a workload profile that maps extremely well to optical compute.

The Lumai Iris server family uses light rather than electricity to perform matrix multiplications. Large GEMMs are decomposed into vector-matrix multiplication, each of which is completed in a single optical cycle. The energy scaling is structurally different from silicon: as matrix size increases, compute scales quadratically while energy grows at most linearly. This is not an incremental efficiency gain over a chip; it is a different position on the energy-compute tradeoff curve entirely.

Lumai Iris Server will deliver approximately 10x less energy per inference versus silicon-based equivalents. It supports INT4 and INT8 precision, deploys in existing air-cooled data center racks, and integrates with standard ML frameworks. It uses components derived from high-volume data center communications technology, meaning no exotic materials andno novel component supply chain risk.  

Critically, this is deployable technology, not a research project. It slots into the prefill pool of a disaggregated inference stack and handles the GEMM-heavy compute phase, freeing the decode pool to run on memory-optimized hardware without carrying the energy and cost overhead of prefill.

A disaggregated inference stack is the right architecture. The next step is putting the right hardware into each half. For prefill, that means hardware technology built from first principles for matrix multiplication at scale, not general-purpose silicon running a workload it was not designed for.

What this means for your stack

Continuing to run prefill and decode on the same silicon, whether coupled or split into disaggregated pools of identical hardware, leaves energy, throughput, and cost efficiency on the table at exactly the scale where those factors matter most.

Purpose-built prefill hardware is how you close that gap. Iris Nova, the first generation of the Lumai Iris server family, is available for evaluation now. If you are architecting or optimizing a disaggregated inference stack and want to assess what purpose-built prefill hardware changes about the system profile, we would love to talk.