Legendary engineer has a solid plan to make AI inference cheaper

Signals Inbox·July 7, 2026·AI Infrastructure

The cost of running AI models could change if model weights move from ultra-expensive HBM toward cheaper storage like NAND flash.

Q1What actually happened?

John Carmack argued that AI inference may not need normal random-access memory for model weights. His point is simple: when a model runs, a lot of its weight access is predictable. So instead of keeping everything in expensive HBM, future AI accelerators might stream weights from much cheaper memory, as long as the bandwidth is high enough.

Q2Why do people care when Carmack says this?

Because Carmack is not a random AI pundit. He is actually one of the engineers behind Doom, Quake, modern 3D game engines, Oculus, and a lot of low-level performance thinking in tech. When someone like that talks about memory access patterns, people listen because this is exactly his zone: making hardware do more than people expect.

Q3What is the real tension here?

AI companies keep buying massive GPU clusters, but the expensive part is not only the compute. It is also feeding the compute fast enough. HBM is extremely fast memory placed close to the GPU, but it is scarce, expensive, and hard to scale. Carmack is basically saying: maybe we are overpaying for the wrong kind of memory in some inference workloads.

Q4What does memory bottleneck mean?

A chip can only calculate if the data arrives fast enough. If the model weights are stuck waiting in memory, the expensive accelerator sits underused. That is painful because AI chips are priced like Ferraris. The nightmare is not having too little math power. It is buying the math power, then starving it because the memory system cannot deliver weights fast enough.

Q5Why mention NAND flash?

Because NAND flash is much cheaper per gigabyte than HBM. It is the kind of memory used in SSDs. The problem is that it is slower and not built for the same kind of access. Carmack’s angle is: for inference, maybe that trade-off is acceptable if the access pattern is predictable and the hardware is designed to stream weights continuously.

Q6Why is it for inference, not training?

Training is messy. You update weights, move gradients, coordinate many chips, and need huge memory bandwidth in many directions. Inference is cleaner. The model weights are mostly fixed, and you are repeatedly reading them to generate tokens. That makes it more plausible to use a deterministic memory path, especially for high-volume AI services serving the same model all day.

Q7Who could this threaten?

The obvious pressure point is the HBM supply chain. Nvidia, SK Hynix, Samsung, Micron, TSMC packaging, and hyperscaler capex are all tied into the idea that premium AI memory stays essential. If cheaper memory can handle more inference workloads, even partially, it could shift margins, chip design, and who captures value in AI infrastructure.

Q8Who could benefit?

Startups building inference-specific hardware could benefit, because they do not have to copy the GPU playbook. NAND makers could also benefit if AI becomes a new demand source for flash, not just SSDs and phones. Hyperscalers would care most because they spend billions running models. A small reduction in memory cost per token becomes huge at their scale.

Q9Is this already proven?

Not at full data-center scale. The idea is directionally credible, but the market proof would be real hardware running competitive models at competitive latency, throughput, power, and cost.

Q10So what is the market signal?

The signal is that AI infrastructure may be entering its memory architecture phase. The first wave was bigger models. The second was more GPUs. The next fight may be about how to feed inference cheaply enough for always-on AI products. Carmack’s post matters because it points to a very specific wedge: if memory gets redesigned, the cost curve of AI could move again.