Takeaways from the AI Hardware and Edge AI Summit 2024

Last week, our Head of Product, Phillip Burr, took part in a panel at the AI Hardware and Edge AI Summit 2024 – the summit’s debut appearance in Europe. The panel discussion covered topics ranging from hardware challenges for AI in datacentres to the necessity for new, novel technologies. 

Expertly moderated by Sally Ward-Foxton from EE Times, Phil was joined on the panel by: 

• Stelios Venieris, Senior Research Scientist, Samsung AI; 
• Roberto Mijat, Senior Director of Product Marketing and Strategy, Blaize;
• and Jake Kochnowicz, VP Product Management, Imagination Technologies.  

In this blog, we pick out three of Phil’s takeaways from the discussion.

• The biggest hardware bottlenecks in a datacentre today

Power. Today’s solutions use up to 1kW in power. This is an incredible amount. As Phil explains to his ‘non-techie’ friends, you could heat a sizeable room with a couple GPUs. Yet all this power has a large monetary cost and environmental cost to the planet. 

From an operational perspective, existing racks in datacentres typically don’t have the power or the cooling capacity to deal with such power. Consequently, there are racks with big gaps in them so that power or thermal densities are not exceeded. 

Total cost of ownership (TCO). When the hype dies down, TCO will be a bottleneck. Why so? GPUs are incredibly expensive both from a capital and operational perspective which means that the economic benefit threshold from the deployment of AI will be higher – thereby limiting deployment. It’s not surprising that GPUs are incredibly expensive. They are chasing the diminishing returns of performance increase, and when you are chasing diminishing returns the cost of increasing performance becomes very high. 

Development costs are also astronomical. Phil noted that the development of Blackwell amounted to a reported $10 billion! Layer on top the power and the datacentre infrastructure costs of delivering and cooling all that power, and the bill’s looking very expensive.

Memory bandwidth. It may be surprising that Phil, working for a 3D optics startup, called out memory bandwidth. But he is often asked whether solving the AI compute performance challenge would then just result in hitting a memory bandwidth bottleneck. Phil explained why this isn’t the case – the Lumai processor has the advantage that the memory can be distributed across the full width of the vector, increasing the memory bandwidth available without needing to use expensive HBM.

• Optical compute can solve datacentre challenges 

Sally was keen to ask why companies seem to be pivoting away from optical compute to optical interconnect. In response, Phil agreed that the companies looking at optical solutions using integrated photonics are making this pivot to interconnect/switching. The reason is because it is extremely challenging to get the performance needed using integrated photonics, due to the low component density, poor scalability and low compute precision.

Lumai has a very different approach – 3D optics. Performing matrix vector multiplication in light means that copying, multiplying and adding doesn’t use power. The advantage of Lumai’s approach is that the system becomes more power efficient as performance increases. This is because the power consumed increases at most linearly as the width of the input vector increases, but the number of mathematical operations performed scales quadratically.

A large matrix size therefore yields a significant increase in throughput without the associated energy cost. Compared to a GPU solution, Lumai’s accelerator only uses about 10% of the energy at the same performance.  

Crucially, there is plenty of runway to continue increasing performance.

• Optical compute isn’t a futuristic technology

The final part of the panel discussion looked to explore what barriers were present to adopting AI hardware technologies. In particular, Phil was asked what the biggest barriers were when it comes to even more futuristic and novel hardware like optical compute.

While certainly novel, Phil was keen to say that Lumai’s technology isn’t futuristic. Given the constraints of fully silicon solutions, it won’t be that long before the technology is in datacentres. There is wide recognition that AI compute can’t continue on its current trajectory of more power. 

The key thing for Lumai is to ensure that its processors look and connect like any other processor. Software developers never need to know that their AI model is running on a light-based processor – other than it runs much faster and the cost they are being charged is significantly less.

A year of progress

As well as sharing insights, it was a pleasure to also hear about the groundbreaking technologies building the future of AI hardware, a future so important for delivering AI deployment and adoption at scale. We hope to return next year to see how the industry has progressed even more! 

To keep up to date with the latest Lumai news, give us a follow on LinkedIn.