Every request to a large language model runs through two very different phases: prefill and decode. Most infrastructure teams spend their time optimizing decode, the token-by-token generation everyone can see in a streaming response. Prefill gets less attention and for a growing share of workloads this is a mistake. Prefill is the compute-bound phase, and as context lengths and agentic workflows push more tokens into the prompt, it increasingly sets the ceiling on how fast, how cheap, and how scalable inference stacks can be.

Prefill is the step where the model reads your entire input, the prompt, the system message, any retrieved context, and turns it into the internal representation it needs before it can generate a single output token. It computes and stores the key and value tensors for every input token at every layer: the KV cache that the decode phase reads from. Unlike decode, which happens one token at a time, prefill processes every input token in parallel, in one large batched pass through the model.

This single pass is the reason why prefill determines time-to-first-token - until it finishes, the model has nothing to say.

Why prefill is a matrix-math problem

Underneath the transformer architecture, prefill is dominated by matrix multiplication. Most of it is the projection and feedforward layers, the learned weight matrices that hold the bulk of every model's parameters. On top of that sits the attention operation itself, which carries no parameters but grows quadratically with sequence length. Because prefill processes the whole prompt at once, these multiplications happen at full sequence length and full batch width simultaneously. The longer the context window, the larger the matrices, and the more compute-bound the phase becomes.

This is the opposite profile of decode. Decode generates one token per step, so each step multiplies a single vector by the weight matrix - its matrix operations are small and the bottleneck is memory bandwidth, moving the KV cache and weights fast enough to keep the compute units fed. Prefill has the data, it just needs to push enormous matrices through enormous multiplications, as fast as the hardware will allow.

Why this is breaking the old playbook

General-purpose GPUs were built to do many things reasonably well: training, prefill, decode, all on the same silicon. That flexibility carries a cost. Running matrix-heavy prefill workloads on hardware designed for a broader job means paying for capability you are not using, in power, in cooling, and in rack space, while the actual bottleneck, the matrix multiply itself, is limited by how much arithmetic throughput you can extract from a fixed power budget. Digital silicon achievesthroughput by packing more transistors into two dimensions, and that scaling is now bounded by power density.

That is why disaggregated inference, splitting prefill and decode onto separate, purpose-built hardware, is becoming the standard architecture for serving frontier models at scale. Decode needs high memory bandwidth and flexibility whether that is served by a GPU or by decode-specialist silicon. Prefill needs something else entirely: raw, efficient matrix throughput.

Where light changes the equation

Lumai Iris was built specifically for this gap. The Iris Server performs the matrix multiplications at the heart of prefill using light instead of electricity. Each vector-matrix multiplication completes in a single optical cycle, and because Iris uses 3Doptics, the compute scales quadratically with matrix size while the energy required grows at most linearly. This is the opposite of how digital accelerators scale, and it is why Iris delivers roughly 10x more compute per watt than GPU-based equivalents on prefill workloads.

Iris Nova, our first-generation system, is already validated on billion-parameter LLM scale, supports INT4/INT8 precision, and drops into existing air-cooled data center racks. No liquid cooling retrofit. No exotic supply chain. The optoelectronic components are derived from the same technology already running at high volume in data center communications.

The takeaway

Prefill is not a footnote in your inference stack. It is a distinct, matrix-heavy workload that deserves distinct, purpose-built hardware. As context windows grow and disaggregated architectures become the norm, the question is not whether to separate prefill from decode. It is what you run prefill on.

See what optical prefill looks like on your workloads.

Book a technical evaluation of Iris Nova and get a straight answer on where it fits in your inference architecture. Contact us at https://lumai.ai/contact-us

‍