Signals Inbox·July 28, 2026·AI Infrastructure

What is the best AI hardware now?

NVIDIA Blackwell Ultra is the best AI hardware overall right now, but the market has split: Rubin leads on technical potential, specialist accelerators can win inference, and local or edge buyers face a completely different contest.

Q1What does “best AI hardware” actually mean now?

Right now, NVIDIA Blackwell Ultra is the best AI hardware overall, although different workloads still produce different winners.

The word “best” only becomes useful once we decide what the buyer is trying to do. A frontier lab training a model across thousands of accelerators cares about scaling and network reliability. A cloud provider serving millions of prompts cares about tokens per megawatt, latency and utilization. A developer running a model at home may care almost entirely about memory capacity and price.

Raw FLOPS do not settle those questions. Vendors count different numerical formats, sometimes include sparsity and often describe peak performance that real models cannot sustain. Memory decides whether a model fits. Memory bandwidth controls how quickly weights can be read. Interconnect speed becomes critical once the job crosses several chips. Software determines how much of the hardware a team can actually use.

For this article, we call hardware “best” when it delivers the strongest useful performance at the required scale, with software and supply a serious buyer can actually access. Blackwell Ultra wins on that basis. Rubin has the higher ceiling, while AMD, Google, AWS, Cerebras, Groq and local systems win narrower contests.

Best AI hardware by workload

Q2Why can’t we choose the best AI chip from a specification sheet?

A specification sheet can narrow the field, but it cannot tell us which AI system will finish a real job fastest or cheapest.

Modern AI workloads expose different bottlenecks. During pretraining, thousands of chips repeatedly exchange gradients and model states. During interactive inference, a system may spend more time moving model weights and key-value cache data than performing arithmetic. Mixture-of-experts models add irregular routing, while video generation and recommendation models stress the hardware in different ways again.

The latest MLPerf rounds show how far the market has moved from single-chip comparisons. MLPerf Training v6.0 received 95 distinct system submissions using 13 accelerator types, and 60% of those systems used multiple nodes. In MLPerf Inference v6.0, the largest entry reached 72 nodes and 288 accelerators; six months earlier, the largest submission had only one-quarter as many nodes.

At this point, the network, server design and software stack shape the result alongside the accelerator. That is why two buyers can test the same chip and report completely different economics. One has a well-optimized model running at high utilization. The other loses half the advantage to communication, weak kernels or request sizes that do not match the system.

Q3Is NVIDIA Blackwell Ultra still the best AI hardware overall?

Yes. Blackwell Ultra is currently the best all-round choice for serious AI training and inference.

The biggest reason is breadth. Blackwell Ultra supports dense language models, mixture-of-experts systems, multimodal models, recommendation engines, image generation and video workloads through the same CUDA-based stack. CoreWeave already offers GB300 NVL72 and HGX B300 capacity, so buyers can deploy it now.

The latest MLPerf Training round strengthened the case. NVIDIA posted the fastest result and the best per-accelerator result on every benchmark, while remaining the only platform submitted across the full seven-test suite. The tests covered conventional models alongside the new DeepSeek-V3 and GPT-OSS mixture-of-experts workloads.

Blackwell Ultra also benefits from years of accumulated software work. CUDA, TensorRT-LLM, NCCL, Transformer Engine, optimized attention kernels and broad framework support reduce the odds that a new model will expose an unsupported operation. Large buyers can choose among several cloud providers and server vendors instead of relying on one tightly controlled environment.

The premium is substantial, and a stable workload may run more cheaply elsewhere after careful optimization. Still, when a team cannot afford a long porting project or a surprise compatibility problem, Blackwell Ultra is the safest high-performance answer.

Q4Has NVIDIA Vera Rubin already taken the lead from Blackwell Ultra?

Vera Rubin has taken the technical lead. Blackwell Ultra remains the better default purchase until Rubin capacity becomes broadly available.

Rubin is further along than a normal future product. NVIDIA says the platform’s seven main chips are in full production, and CoreWeave has already completed system-level validation of an operational Vera Rubin NVL72 rack. The system joins 72 Rubin GPUs and 36 Vera CPUs through a 260 TB/s NVLink fabric.

The expected jump is large. NVIDIA and CoreWeave describe up to ten times better inference per watt, one-quarter as many GPUs for some training jobs and one-tenth the cost per million tokens compared with Blackwell. Rubin also pairs GPUs with NVIDIA Groq 3 LPX accelerators for low-latency decoding, a more specialized design than the all-purpose GPU approach used previously.

Those are vendor projections, and one validated rack does not equal broad customer access. CoreWeave’s public materials still frame Rubin as infrastructure customers should prepare for, while Blackwell Ultra instances are already listed as available in selected regions. Independent benchmark coverage is also far deeper for Blackwell.

An early buyer with a production allocation and a strong infrastructure team should take Rubin. Everyone else should plan around it without pretending they can rent it everywhere tomorrow.

Q5What is the best hardware for training frontier AI models?

For frontier AI training today, Blackwell Ultra has the strongest proven record.

Frontier training rewards systems that keep thousands of accelerators busy at once. A fast chip can lose much of its advantage when workers wait for gradients, experts or model states to arrive. NVIDIA’s training stack combines the GPU, NVLink domains, InfiniBand or Spectrum-X networking, NCCL communication libraries and model-specific optimizations.

In MLPerf Training v6.0, NVIDIA partners scaled Blackwell systems to 8,192 GPUs across production cloud environments. The platform led every benchmark both at full scale and after performance was normalized per accelerator. The new DeepSeek-V3 test is especially relevant because it uses 671 billion total parameters and activates 37 billion per token, creating the irregular communication pattern common in modern mixture-of-experts models.

Software gains were visible within the same hardware generation. NVIDIA reported a 30% improvement in DeepSeek-V3 training throughput over three months through kernel fusion, better routing and more overlap between communication and computation. That can shorten an extremely expensive training run without replacing a single server.

AMD is now competitive on smaller standardized tests, and Google’s TPU 8t may become formidable for Google-native frontier work. For an outside buyer today, Blackwell Ultra has much stronger public evidence at large scale and is easier to source from several providers.

Q6What is the best hardware for large-scale AI inference?

For large-scale AI inference, Blackwell Ultra currently offers the best broad platform. Specialist chips can still win on latency or on one particular model.

Serving models is no longer one simple throughput contest. Providers must control time to first token, speed after the first token, tail latency, memory use and the number of simultaneous users. Larger batches can raise total throughput while making an individual user wait, so the winning setup depends on the service-level target.

MLPerf Inference v6.0 now includes GPT-OSS 120B, DeepSeek-R1 reasoning, text-to-video, vision-language processing and a modern recommendation model. MLCommons has already started adding agentic inference to the next round, which shows how quickly the target keeps moving. NVIDIA led across the widest range of current workloads, letting a cloud operator move capacity among chat, video, search and recommendation jobs without changing its main hardware platform.

AMD’s MI355X came much closer on language models. In AMD’s accepted submissions, one MI355X node exceeded comparable B200 results on GPT-OSS 120B and reached 91% of B300 offline performance and 82% of B300 server performance. Multi-node MI355X systems also passed one million tokens per second on several tests.

AMD is credible for carefully chosen deployments now. Blackwell Ultra keeps the lead because it combines strong latency, high throughput and wider workload coverage without forcing the buyer into one narrow serving pattern.

Q7Is AMD MI355X finally a real NVIDIA alternative?

Yes. MI355X has become a serious NVIDIA alternative, especially when the workload can be benchmarked before a large purchase.

MI355X is good enough that large buyers can use it to push back on NVIDIA pricing. Each GPU provides 288 GB of HBM3E and 8 TB/s of memory bandwidth, giving an eight-GPU server roughly 2.3 TB of accelerator memory for large models, long contexts and bigger key-value caches.

AMD’s old memory-capacity advantage needs a more precise reading now. MI355X carries far more memory than NVIDIA B200’s 180 GB, but Blackwell Ultra moves into roughly the same capacity class. Rubin goes further by pairing 288 GB with much higher bandwidth. AMD still offers unusually generous memory, although it no longer owns that comparison outright.

Recent benchmark results show a much smaller performance gap. MI355X came within 5% of B200 on Llama 2 70B fine-tuning and within 6% on Llama 3.1 8B pretraining. On selected inference tests, it matched or beat B200 and approached B300.

ROCm remains the deciding variable. PyTorch support, optimized kernels and cloud availability have improved quickly, but CUDA still has broader third-party testing and a larger engineering base. MI355X deserves serious consideration for high-volume workloads that have already been validated. Teams changing architectures every week will still find NVIDIA easier.

Q8Can Google’s new TPUs beat NVIDIA GPUs?

Google’s TPU 8t and TPU 8i can beat NVIDIA for workloads built around Google’s stack. NVIDIA remains the safer general-market choice today.

Google has moved beyond Ironwood with two eighth-generation designs. TPU 8t targets large-scale pretraining, while TPU 8i focuses on sampling, serving and reinforcement learning. The split reflects how different the bottlenecks have become: training wants sustained throughput across huge clusters, while reasoning inference wants low latency and fast communication among experts.

TPU 8t offers 216 GB of HBM, 12.6 petaflops of peak FP4 compute and a 9,600-chip superpod. TPU 8i raises memory to 288 GB, uses 384 MB of on-chip SRAM and cuts network diameter through a serving-oriented topology. Google claims up to 2.7 times better training performance per dollar for 8t and 80% better inference performance per dollar for 8i compared with Ironwood.

Access and proof are the weak points. Google says both systems will reach Cloud customers soon, so most external buyers still cannot compare mature production capacity with widely available Blackwell Ultra. Google’s internal teams also benefit from tight coordination among DeepMind models, JAX, Pathways, XLA and custom networking.

A team already committed to Google Cloud should benchmark these TPUs as soon as capacity opens. Buyers who need portability, broad tooling and several cloud choices will have an easier time with NVIDIA.

Q9Is AWS Trainium3 the cheapest serious AI option?

Trainium3 may offer the best economics for large AWS-native workloads. AWS has not yet proved a universal cost lead against current NVIDIA or AMD systems.

A Trn3 UltraServer can connect 144 Trainium3 chips, provide 362 petaflops of FP8 compute and scale through UltraClusters to hundreds of thousands of chips. AWS also supports PyTorch, JAX, Hugging Face and its own Neuron software across SageMaker, EKS and other services.

AWS claims up to 4.4 times the performance, 3.9 times the memory bandwidth and four times the performance per watt of Trainium2. In Amazon Bedrock, it reports up to three times higher performance and more than five times as many output tokens per megawatt at similar user latency.

Anthropic provides the clearest real-world proof of Trainium’s scale: almost one million Trainium2 chips are already training and serving Claude. AWS and Anthropic now have a huge base for tuning Trainium3, although an outside buyer may see very different savings on another model.

Trainium3 is attractive when the workload is large, predictable and unlikely to leave AWS. A team that wants multi-cloud portability should treat the lower claimed token cost as compensation for deeper platform dependence.

Q10Are Cerebras and Groq actually faster than GPUs?

On supported models, Cerebras and Groq can feel dramatically faster than GPU services because the user waits far less for each token.

Cerebras uses a wafer-scale processor to keep far more computation and memory movement within one giant device. Its public inference service has delivered more than 2,000 tokens per second on Llama 4 Scout and over 1,500 tokens per second on a DeepSeek-R1 Llama 70B model. At those speeds, a long response arrives in seconds instead of unfolding line by line.

Groq follows a different path with statically scheduled LPUs and large on-chip SRAM. The design produces consistent token latency, which fits voice interfaces, coding tools and agents that make many sequential model calls. NVIDIA’s decision to pair Groq 3 LPX racks with Vera Rubin shows that low-latency decoding has become important enough to deserve dedicated hardware inside a broader platform.

The limits appear as soon as the workload changes. Model coverage is narrower, training options are limited and custom operations can be harder to support. A GPU cluster can move from fine-tuning to image generation to a new research model. A specialist service earns its speed by doing fewer things extremely well.

Choose Cerebras or Groq when interactive speed directly improves the product and a supported model meets the quality target. Keep general-purpose infrastructure on GPUs or cloud-specific accelerators.

Q11Does AI software matter more than the chip now?

Software can easily decide the winner between two chips that look similar on paper.

CUDA’s advantage comes from accumulated coverage. New attention methods, quantization formats, distributed-training tools and inference engines are usually tested on NVIDIA first. When a model introduces an unusual operation, an NVIDIA user is more likely to find a working kernel, a documented workaround and an engineer who has already seen the problem.

The current software race gives us a few concrete examples. AMD now publishes reproducible ROCm containers and benchmark guides for MI355X. Google is adding native PyTorch support to its newest TPUs, although that support is still in preview. AWS has brought PyTorch, JAX and Hugging Face into Neuron rather than asking every customer to begin with proprietary tools.

All three alternatives are easier to use than they were two years ago. Compatibility still has layers, though. A model may launch successfully, run slowly because of a fallback kernel, or fall apart once it scales beyond one server.

The software gap matters most for teams experimenting with new architectures. A company serving the same model for a year can spend time optimizing another platform and recover that engineering cost through lower infrastructure bills. A research team changing models constantly will usually get more value from CUDA because fewer things break and more fixes already exist.

Q12Which AI hardware is the most power efficient now?

There is no credible universal power-efficiency winner across all AI workloads yet.

Efficiency depends on useful output rather than a chip’s wattage. A lower-power accelerator can consume more total energy if it takes much longer, while an extremely powerful rack may perform well only when requests keep it close to full utilization. Cooling, networking, host CPUs and idle capacity belong in the calculation too.

Rubin currently carries the strongest vendor claims. NVIDIA and CoreWeave describe up to ten times better inference per watt than Blackwell for Vera Rubin NVL72, while the Rubin-plus-LPX configuration is projected to reach up to 35 times more throughput per megawatt on trillion-parameter models. Google claims up to twice the performance per watt for TPU 8t and 8i over Ironwood. AWS reports more than five times as many output tokens per megawatt for Trainium3 versus Trainium2 at similar latency.

These comparisons use different models, baselines and system boundaries. They show rapid progress inside each product family, but they cannot be turned into a fair cross-vendor ranking. Current MLPerf results offer better discipline, although comparable power submissions still cover only part of the market.

A serious buyer should measure complete-system energy at the required latency and model quality. Any universal winner announced from peak TOPS per watt is mostly marketing.

Q13Should a startup rent AI hardware or buy its own servers?

Most startups should rent AI hardware and consider ownership only after one workload runs steadily enough to keep expensive servers busy.

Early-stage products change too quickly for a large hardware purchase. A team may start with API calls, move to an open model, add retrieval, switch its serving engine and later fine-tune a smaller model. A server selected for the first version can become a constraint before it has been paid off.

Cloud access also lets the startup test the actual application on several platforms. The team can compare Blackwell Ultra with MI355X, Trainium3 or a specialist inference service while measuring latency, utilization and cost per successful task. Real application data beats a generic hourly-price comparison.

Ownership becomes attractive when demand is predictable and utilization stays high. The calculation should include financing, electricity, cooling, maintenance, spare capacity, networking and engineering time. Obsolescence belongs in the model as well: a three-year-old cluster may still work perfectly while costing far more per token than newer hardware.

A hybrid setup often makes sense later. The company can own enough capacity for its stable baseline and use cloud systems for peaks, experiments or frontier training runs.

When startups should rent or own AI hardware

Q14What is the best hardware for running AI locally?

The RTX 5090 is the best-value high-end local AI card today, while larger models push buyers toward the RTX PRO 6000, DGX Spark or an M5 Max Mac.

NVIDIA sells the RTX 5090 with 32 GB of GDDR7 memory and mature CUDA support at a $1,999 official price. The card handles coding models, image generation, speech tools and quantized language models extremely well when they fit. Its limit is simple: 32 GB fills quickly once models or context windows grow.

For larger models, the RTX PRO 6000 raises capacity to 96 GB on a conventional desktop GPU. Professional users can avoid splitting a model across several consumer cards, although the price moves into workstation territory.

DGX Spark offers 128 GB of unified memory and supports models up to 200 billion parameters in a compact system. Its official price rose from $3,999 to $4,699 because of memory-supply constraints, a useful reminder that local AI value depends heavily on scarce high-capacity memory. Apple’s M5 Max also reaches 128 GB of unified memory with 614 GB/s of bandwidth, making it a strong portable option for models that fit poorly on 32 GB GPUs.

Buy the RTX 5090 when the target model fits. Move to the RTX PRO 6000 for more single-GPU memory and speed, or choose DGX Spark or M5 Max when unified capacity matters more than CUDA throughput.

Best local AI hardware by use case

Q15What is the best AI hardware for robots and edge devices?

Jetson AGX Thor is currently the strongest general-purpose platform for advanced robots that need large AI models on the machine.

Thor delivers up to 2,070 FP4 teraflops, 128 GB of memory and configurable power between 40 and 130 watts. NVIDIA says it provides up to 7.5 times the AI compute and 3.5 times the energy efficiency of Jetson AGX Orin. The developer kit and production modules are already generally available.

With that much local compute, a robot can combine vision, language, sensor processing and action planning without sending every decision to a datacenter. Lower network dependence improves response time and keeps the machine useful when connectivity is weak or unavailable.

The surrounding tools add weight to the choice. Jetson works with CUDA, TensorRT, Isaac robotics software and Holoscan sensor processing, and many robotics teams already build around that stack. Qualcomm and smaller edge accelerators may offer better efficiency for cameras, drones or consumer devices with narrower models. Thor targets a heavier class of machine.

For humanoids, autonomous industrial systems and robots running several demanding models together, Jetson Thor is the clearest current answer.

Q16Is NVIDIA’s AI hardware lead shrinking or getting stronger?

NVIDIA faces stronger chips than before, yet its control of the complete AI system is getting harder to challenge.

Competition is real now. AMD can approach or beat B200 on selected workloads. Google has split its eighth-generation TPU into purpose-built training and inference designs. AWS has almost one million Trainium2 chips working on Claude and is rolling that base into Trainium3. Cerebras and Groq have proved that specialist architectures can transform interactive inference.

NVIDIA has responded by expanding what counts as its platform. Blackwell Ultra arrives as complete racks with CPUs, networking and software. Rubin adds a newer NVLink fabric, storage processors, networking systems and Groq-derived LPX inference accelerators. The company can offer a general GPU path and a specialist low-latency path inside the same architecture.

A rival can produce a faster chip on one model and still struggle to match NVIDIA’s software, cloud availability, networking and developer familiarity. Customers increasingly compare the cost and reliability of entire clusters rather than isolated accelerators.

Chip competition is tightening. NVIDIA’s ecosystem advantage is not. An independent buyer can source NVIDIA systems from more vendors, hire more engineers who know the stack and run more models without porting work. Google and AWS are most dangerous where they control the model, cloud and custom silicon together.

Q17So what is the best AI hardware now?

Blackwell Ultra is the best AI hardware overall today. Rubin leads technically, and every other winner depends on a narrower workload or ecosystem.

Buyers can already verify leading training results, broad inference coverage, mature CUDA software and real cloud capacity. NVIDIA charges a premium, but Blackwell Ultra removes more deployment risk than any alternative.

Rubin has moved beyond a paper launch. CoreWeave has validated an operational NVL72 rack, and the architecture promises a large improvement in memory movement, power efficiency and low-latency inference. Broad access and independent evidence are still too limited for most buyers to make Rubin their first choice.

AMD MI355X is the strongest open alternative for workloads that can be tested in advance. Google’s TPU 8t and TPU 8i could become the best choices inside Google Cloud once customer access broadens. Trainium3 deserves attention from large AWS users. Cerebras and Groq win when instant token generation shapes the product. The RTX 5090 leads local value, while Jetson Thor leads advanced edge AI.

Pick Blackwell Ultra when the question is simply “what is best now?” Move elsewhere only when the workload gives you a specific reason.