Why Is Matrix Multiplication Breaking AI Inference?

Strip a transformer down to what actually consumes arithmetic and it is matrix multiplication: the attention projections and products, the feedforward layers, and the final vocabulary projection, accounting for well over 90 percent of the arithmetic in an inference call across dense and Mixture of Experts models in production today. In a conventional digital accelerator, compute and energy are coupled, so as matrices grow the power bill compounds, and you cannot shrink your way out of a coupling problem. Lumai Iris performs matrix multiplication optically, so energy grows at most linearly while compute grows quadratically, the decoupling behind roughly 10x lower energy per inference on the matrix-heavy prefill workloads it targets.

***

Frontier AI companies are projecting a roughly 1,000x increase in the effective compute needed for inference over the next five years. Delivered with conventional digital accelerators, that translates to around $100 trillion in new infrastructure and roughly 1,000 GW of additional electrical capacity. Neither number is achievable, not economically, not physically. The constraint is energy, and it starts with one operation.

One operation, most of the compute

Strip a transformer down to what actually consumes arithmetic, and it is matrix multiplication: the query, key, value and output projections in attention, the attention score and value products themselves, the up and down projections in the feedforward layers, and the final vocabulary projection. Across dense and Mixture of Experts models at every scale in production today, matrix multiplication accounts for well over 90 percent of the arithmetic in an inference call.

Whatever runs your matrix multiplication is what determines your cost per token, your power draw, and how far you can scale.

The scaling trap

In a conventional digital accelerator, compute and energy are coupled. As models grow, larger matrices mean compute requirements rise quadratically, and energy consumption rises right alongside it. Every increase in model size or context length compounds the power bill. Incremental silicon improvements, smaller nodes, denser packaging, are finite, non-recurring, and increasingly expensive to extract. You cannot shrink your way out of a coupling problem. You have to break the coupling.

Why GPUs cannot break it

GPUs move electrons through digital logic to perform matrix multiplication. That is fundamentally energy-intensive work, and it does not get meaningfully cheaper as matrices get larger. It becomes more expensive, in power, in cooling infrastructure, in the silicon itself. This is why the industry's current trajectory requires liquid cooling, new data centers, and gigawatts of capacity that do not yet exist. The hardware is not underperforming; digital compute was never going to do efficiently at this scale.

Decoupling compute from energy, with light

Lumai Iris performs matrix multiplication optically instead of electronically. So light multiplies, and each vector-matrix operation is completed in a single optical cycle. Because Iris servers use optics rather than just digital logic, the relationship between compute and energy changes fundamentally: matrix size increases compute quadratically, exactly as before, but instead  energy grows at most linearly.  This decoupling principle is the entire reason the Lumai Iris server delivers roughly 10x lower energy per inference than GPU-based equivalents on the matrix-heavy prefill workloads it targets.

This is not a research curiosity. Iris Nova, our first-generation system, runs billion-parameter LLMs scale, supports INT4/INT8 precision, and is compatible with standard frameworks like PyTorch. It deploys into existing air-cooled racks, no liquid cooling, no exotic materials, no new supply chain. The optoelectronic components come from the same technology platform already running at high volume in data center communications. Iris Nova is already available for evaluation and built by an Oxford spin-out.

The takeaway

If matrix multiplication is what is breaking your inference stack, more of the same hardware will not fix it. The industry needs a compute technology where energy does not scale in lockstep with compute. That is what optical matrix multiplication delivers, and it is available for evaluation today.

Frequently Asked Questions

How much of AI inference is matrix multiplication?

Well over 90 percent of the arithmetic in an inference call, across dense and Mixture of Experts models at every scale in production today: the query, key, value and output projections in attention, the attention score and value products, the up and down projections in the feedforward layers, and the final vocabulary projection.

What is the scaling trap?

In a conventional digital accelerator, compute and energy are coupled. As models grow, larger matrices raise compute quadratically and energy rises alongside it, while incremental silicon improvements are finite, non-recurring, and increasingly expensive to extract. You cannot shrink your way out of a coupling problem; you have to break the coupling.

How does Lumai Iris break the coupling?

By performing matrix multiplication optically: each vector-matrix operation completes in a single optical cycle, so matrix size still increases compute quadratically but energy grows at most linearly. That decoupling is the reason the Iris server delivers roughly 10x lower energy per inference than GPU-based equivalents on the matrix-heavy prefill workloads it targets.

Is this deployable today?

Iris Nova runs billion-parameter LLMs, supports INT4/INT8 precision, is compatible with standard frameworks like PyTorch, and deploys into existing air-cooled racks with no liquid cooling, no exotic materials, and no new supply chain. It is available for evaluation and built by an Oxford spin-out.

Iris Nova is available for evaluation today. Contact Lumai to arrange one.