Dylan Patel on Why AI's Biggest Infrastructure Gains Come From Co-Design

Patel rejects a hardware-only account of AI efficiency gains. He says model architecture, infrastructure software, kernels, and chips have all improved, and the largest jumps come when labs optimize the model, software stack, and hardware together.

Patel says inference, the serving of a trained model to users, should not be benchmarked as one generic workload. A batch document job can pack requests for lower cost, while an interactive coding or chat session may spend more compute for lower latency.

Patel says more data center capacity is coming online every quarter, but better models create valuable new tasks faster than supply can double or quadruple. He treats this as a demand-and-supply crunch, not only a delay in hardware buildout.

Patel says Nvidia's advantage is no longer only CUDA as a programming interface. Many open models are co-designed for Nvidia GPUs, while interconnect choices such as Nvidia NVLink and Google's ICI shape which model architectures run well on which hardware.

Patel expects more bifurcation between model architectures and chip architectures. Custom ASICs can be efficient for one workload, but labs may still need general-purpose AI compute because next year's model architecture is uncertain.

Patel says inference benchmarking has to be living and daily because models, drivers, PyTorch, vLLM, SGLang, kernels, and chip configurations keep changing. A point-in-time benchmark can go stale before it reflects real serving economics.

Patel says AI GPU clusters reward fast buildout, specialized networking, different contracting, and higher revenue per megawatt. That gives neoclouds room where traditional hyperscaler advantages in CPU cloud do not map cleanly onto AI infrastructure.

Patel says fast inference has a real market when the user values speed, but SRAM-heavy systems can face limits if frontier models become very large or require long context. The chip can be strong and still be constrained by the model shape it must serve.

Patel says equivalent-quality model cost has fallen roughly 60x and intelligence per watt roughly 40x, while computers remain many orders of magnitude away from the human brain. He adds that the comparison does not settle infrastructure strategy because computers can use much more power than brains.

Patel uses DeepSeek as a public example of co-design. Its expert shapes, network IO, collectives, and attention arithmetic were optimized for Hopper, Blackwell, and Huawei chips, so a strong TPU can still be a poor fit for that model shape.