Inference at Scale: Building the Infrastructure for AI’s Next Phase

The bottleneck for real-time AI is power and proximity

Serving AI in real time is a different problem from serving it in batches, and latency sets the architecture. Voice agents have to answer inside 400 milliseconds. That pushes specialized silicon and puts GPUs physically close to users, and it splits models by stage of inference. The panel agrees inference is starved of capital relative to the trillions it will need, and that power and politics are the binding constraints.

Takeaways

• Voice agents must answer within 400 milliseconds. Hippocratic AI's patient-facing calling agent chains an ASR model, roughly 30 language models and text-to-speech across hundreds to thousands of B300 nodes. Thirty to 35 models must answer in under 400 milliseconds for billions of daily interactions.
• A 600 millisecond pause breaks caller trust. Datta says patients assume the line dropped when a voice agent goes quiet, so p99 matters as much as the average. A call runs hundreds of turns and every one has to clear the bar.
• Inference hardware is splitting into specialized chips. Jacobson says GPUs are fragmenting by stage of the inference chain, with prefill separated from decode so the decode layer can push out more tokens. Much of the latency sits in the KV cache, which grows with users and context.
• Inference is under-invested by trillions of dollars. Jacobson calls it the world's largest market and notes that wiring the internet took ten years of fiber. Bringing real-time AI to 9bn people needs far more power and GPUs, and power and politics are the two biggest problems.
• Compute has to sit close to its users. Matthews compares the moment to railroads, arguing not every data center can be the size of Manhattan. A data center in Arkansas cannot serve San Francisco well, though the CRO of Nebius told him small data centers are less economically valuable.
• GPUs are a health and safety question. Datta describes a health research institute that lost 1,700 patients to follow-up. Hippocratic AI reached 200, about 180 got lung CT scans, and nine would otherwise have returned with stage four cancer. HIPAA keeps data in the continental US while latency keeps GPUs close to users.

What comes next

• Taiuti says compute will become as essential as water and that cities without dedicated compute will not exist, because physics keeps data centers near users.
• Jacobson expects far more inference outside hyperscalers, shifting to laptops and phones as Apple takes the challenge seriously.
• Matthews notes the 2015 to 2018 edge networking wave failed for lack of use cases, and telecom data centers lack GPUs and water cooling.