Decode-side hardware is already commercialized. Prefill is not. That asymmetry is quietly breaking your inference stack.

The inference stack has a split personality

LLM inference is not one workload. It is two, and the industry has only built specialized hardware for one of them.

Disaggregated inference architectures make this clear. Prefill and decode run on separate hardware pools, each tuned for its own compute profile. The decode side has attracted serious investment: memory capacity and memory-bandwidth-optimized accelerators are commercially available and being deployed at scale across hyperscalers and neoclouds today.

Prefill has not received the same treatment. It runs on the same silicon clusters just as everything else, on hardware designed for a fundamentally different problem. That mismatch has a cost, and it compounds as context windows grow.

What prefill actually demands

Prefill is compute-bound. When a prompt arrives, the model processes every input token in parallel, through the attention and feed-forward layers - trillions of floating-point operations for a long prompt on a frontier mode. The dominant computation is matrix multiplication: large weight matrices applied to the full sequence at once, layer after layer, as fast as the chip can grind through them.

Decode is the opposite. It is memory-bandwidth-bound. Each new token is generated sequentially, and producing each one means streaming the entire model through the compute unit - every weight, every layer, plus the growing KV cache. This requires fast access to the weights and KV cache rather than raw arithmetic throughput - the performance ceiling is how quickly you can move data, not how quickly you can multiply.

These are different problems - compute-bound and memory-bandwidth-bound workloads do not optimize well on the same architecture. A chip designed to maximize memory access rates for decode is not the right chip for the matrix-heavy, parallelizable work of prefill. Running both phases on silicon hardware means neither runs on optimal infrastructure.

Why this matters at scale

At low inference volumes, the inefficiency is tolerable. At scale, it is an issue .

Frontier AI companies are projecting a roughly 1,000x increase in effective compute demand over the next five years. Delivering that on conventional digital accelerators requires approximately $100 trillion in infrastructure investment and around 1,000 GW of additional electrical capacity. Neither figure is achievable. The constraint is not transistor density. It is energy.

Prefill is where that energy problem is most acute. Matrix multiplication at hundreds of billion-parameter scale is relentlessly power-hungry on silicon. Every prefill request burns watts that could be avoided with a purpose-built architecture. In a disaggregated stack, prefill is not an afterthought. It is a discrete infrastructure decision, and right now that decision is being made by default rather than by design.

The architectural gap nobody has filled

Decode has its specialists. Memory-optimized accelerators, custom silicon for fast memory access, dedicated decode pools in production inference clusters. The ecosystem recognized the workload profile and built hardware for it.

Prefill does not have an equivalent. There is no commercially deployed, purpose-built prefill processor. The workload is simply run on whatever chip capacity is available, tolerated rather than designed for. For the architects building disaggregated inference infrastructure, this means one side of the stack is engineered and the other is improvised.

That gap exists because prefill has a specific requirement that general-purpose silicon cannot efficiently meet: the ability to execute very large matrix multiplications at high throughput, with low energy per operation, at data center scale. Closing it requires a different compute technology, not a faster version of the same one.

Light is the right answer to a matrix problem

Lumai was built on this premise. Our Lumai Iris Server uses light rather than electricity to perform matrix multiplications, the computation that defines the prefill workload. Each vector-matrix multiplication completes in a single optical cycle. The energy scaling is fundamentally different: as matrix size increases, compute scales quadratically while energy grows at most linearly.

The result is approximately 10x less energy per inference versus silicon-based equivalents. Iris Nova, the first generation, is validated on Llama 3 at billion-parameter scale and will be deployed as a cluster in existing air-cooled data center racks with no liquid cooling required. It supports INT4 and INT8 precision and is compatible with standard ML frameworks including PyTorch.

This is not a research chip or a co-processor. It is a full server deployment, built from the same technology platform already used at high volume in data center communications. The supply chain is established. The hardware is available for evaluation today.

The prefill processor addressing the challenge

Disaggregated inference is becoming the default architecture for production LLM deployments. As it does, the prefill phase shifts from an implicit silicon workload to an explicit infrastructure decision. The question is no longer whether to specialize prefill. It is what hardware you choose to do it with.

If you are building or optimizing a disaggregated inference stack and want to understand what purpose-built prefill hardware looks like in practice, we would like to talk. Lumai Iris Nova is available for evaluation today.

‍