> ## Content Index > Fetch the complete content index at: https://www.insidedeeptech.com/llms.txt > Use this file to discover other available public pages before exploring further. # The Best AI Inference Chips of 2026: A Detailed Comparison - URL: https://www.insidedeeptech.com/the-best-ai-inference-chips-of-2026-a-detailed-comparison/ - Published: 2026-06-26T12:35:08.000Z - Updated: 2026-09-08T08:33:18.000Z - Description: Ten inference accelerators compared on measured benchmark results where those exist, and on published specifications where they do not. Which is which is the finding. - Author: Irina Peskisheva - Tags: AI *Inside Deep Tech Editorial* *Last updated: 17 August 2026* Twenty-four organisations submitted results to MLPerf Inference v6.0, published by MLCommons on 1 April 2026\. Two of them were accelerator vendors, competing head to head, on the same workloads, under rules the other submitters could check. Everything else on this page is compared on numbers each company publishes about its own hardware. Those numbers are not wrong. They are also not comparable, because nobody measured them the same way, and a peak throughput figure from a datasheet sits in a different category from a benchmark result somebody else can reproduce. Knowing which chips have been independently measured, and which have not, is more useful for a purchasing decision than any single ranking of them. ## How this comparison is built The chips fall into two groups. The first group has results in MLPerf Inference v6.0, run on fixed models to a defined accuracy target. Two accelerator vendors are there, NVIDIA and AMD. Intel appears with server CPUs and workstation GPUs, which is a different claim and is covered below for completeness. The second group publishes specifications and internal benchmarks, and has not submitted to MLPerf. That includes every hyperscaler ASIC and every specialist inference chip on this page. Their absence is not evidence of weakness. It does mean that comparing them to the first group means comparing a measurement to a claim. ![Two accelerator vendors submitted to MLPerf Inference v6.0; eight chips on this page publish company figures only](https://storage.ghost.io/c/85/42/8542ca3e-ec07-47c4-9d72-d69d172c1781/content/images/2026/08/01-measured-versus-company-figures-mobile-760.png) Within each group, chips are ordered by what a buyer actually runs into first: memory capacity and memory bandwidth. Large language model inference is dominated by memory movement, and a chip that cannot hold the weights will lose to a slower one that can. ## Group one: measured in MLPerf Inference v6.0 ### NVIDIA B200 and B300 Blackwell swept the workloads NVIDIA entered. On GPT-OSS 120B the B200 delivered roughly two and a half times the per-GPU throughput of the H200\. Two things drive that: 8,0 TB/s of HBM3e memory bandwidth against the H200's 4,8, and FP4 support. Bandwidth alone accounts for a factor of 1,67, so the rest comes from the narrower number format. Fourteen partner organisations submitted results on NVIDIA hardware in this round, against nine on AMD’s. That matters more than the headline numbers. Partner submissions show the performance is reproducible outside the vendor’s own lab, and no other platform on this page has that property to the same degree. One caveat travels with the sweep. NVIDIA was the only vendor to enter every workload, so leading on all of them is partly a statement about who showed up. ### AMD Instinct MI355X The MI350 series carries 288 GB of HBM3E, the largest capacity of any accelerator with verified results, and that capacity is the argument. On Llama 2 70B in v6.0, AMD matched the B200 in the Offline scenario, came close in Server, and beat it in Interactive, which is the scenario with the tightest latency ceiling. Against the B300 the figures were 92% in Offline, 93% in Server and 104% in Interactive. AMD also reported cluster-scale throughput above one million tokens per second and had nine partners submitting on its platform, with partner results landing within about 4% of AMD’s own, and sometimes within 1%. The gap is coverage. AMD did not submit for DeepSeek-R1 or the multimodal Qwen3-VL, which leaves reasoning and vision workloads without a head-to-head number. ### Intel Xeon 6 and Arc Pro B-series Intel’s v6.0 submissions covered server CPUs and workstation GPUs, aimed at professional workstations and edge inference servers. A narrower claim than the headline accelerators make, and a measured one. Gaudi 3 does not appear anywhere in Intel’s v6.0 submissions. That absence turns out to be the beginning of a longer story, further down this page. ## Group two: specifications, no independent benchmark ### Google TPU v7, Ironwood Google’s seventh-generation tensor processing unit is positioned primarily for inference and has been generally available since late 2025\. Published per-chip figures: 4 614 TFLOPS at FP8, 192 GB of HBM3E at 7,37 TB/s, and 9,6 Tb/s of inter-chip interconnect. Pods run to 9 216 chips. Ironwood serves Gemini API traffic inside Google, which is as demanding a production reference as exists. The constraint is not the silicon. Ironwood is reachable only through Google Cloud, and the inference stack most teams already run does not transfer cleanly. TensorRT-LLM is CUDA-only. vLLM and SGLang reach TPUs through PyTorch/XLA, which in practice runs slower than native JAX. Choosing Ironwood means adopting a software stack as much as a chip. ### AWS Trainium3 Launched in December 2025 as AWS’s first 3 nm part: 2,52 petaflops of FP8 compute, 144 GB of HBM3e and 4,9 TB/s of memory bandwidth per chip, which AWS gives as 1,5 times the memory capacity and 1,7 times the bandwidth of Trainium2\. A Trn3 UltraServer holds 144 chips. Worth stating plainly, because published comparisons frequently get this wrong: there is no Inferentia3\. AWS has folded its inference-specific line into Trainium, and the current Inferentia part is Inferentia2, with two NeuronCore-v2 cores and 32 GB of memory. Trainium4 has been announced for late 2026 or 2027. ### Microsoft Maia 200 Microsoft published a full datasheet in January 2026: TSMC 3 nm, 216 GB of HBM3e at 7 TB/s, native FP8 and FP4 tensor cores, and a thermal design power of roughly 750 W. The chip handles Copilot and OpenAI API traffic inside Azure. It is not offered to Azure customers as a provisionable instance type. For anyone outside Microsoft, Maia is infrastructure that affects the price of a service, and not a chip that can be bought or rented. ### Intel Gaudi 3 Gaudi 3 carries 128 GB of memory and has been priced aggressively against NVIDIA parts. Intel has confirmed that the Gaudi line will be discontinued as its coming GPUs arrive in 2026 and 2027. That single fact outweighs every specification in this section. A discontinued accelerator can still be the correct purchase for a workload with a defined end date, and it is the wrong purchase for anything expected to run for five years. ### Groq LPU Groq’s architecture removes the layered memory hierarchy that GPUs rely on and executes deterministically, which is what produces the token generation speeds the company publishes. Determinism is a real operational property: once a model is tuned for the hardware, its timing does not drift with utilisation. Groq has not submitted to MLPerf. Its published throughput figures are company-stated, measured on models the company selected, and there is no third-party run to check them against. ### Cerebras Wafer-Scale Engine 3 The WSE-3 uses an entire silicon wafer as a single processor, which removes the chip-to-chip communication that dominates multi-accelerator systems. Cerebras states a peak of 125 petaflops. The usual description of this architecture overstates one thing. On-wafer SRAM is fast and finite, and frontier-scale model weights do not fit in it; they stream from an external memory system. Wafer-scale removes a bottleneck, and does not remove memory hierarchy altogether. The deployment cost is real: power and cooling for a wafer-scale system need data centre design that a standard rack does not provide. ### Qualcomm Cloud AI 100 Ultra Qualcomm’s entry draws on the company’s history in power-constrained mobile silicon and targets deployments where energy per token matters more than absolute throughput: regional edge servers and appliances. The company publishes performance-per-watt figures; there is no independent benchmark result to set them against. ### Tenstorrent Tenstorrent builds on RISC-V and sells both processor cores and AI accelerators, which are separate product lines and are frequently conflated in comparison articles. The accelerators are the relevant part here. The strategic argument is openness: an architecture without a proprietary instruction set, aimed at buyers who treat vendor lock-in as the primary risk. No MLPerf submissions, and the published figures are the company’s own. ## What the two groups mean in practice The most consequential number in v6.0 was not a hardware number at all. Software optimisation delivered up to 2,77 times more throughput on hardware that had not changed, and NVIDIA reported that per-GPU throughput on DeepSeek-V3 rose roughly 1,3 times in about six months from software alone. That has two consequences for anyone buying. A chip bought today will get faster without being replaced, and the size of that improvement depends on how many engineers are optimising for it. That is a function of installed base, which is where the platform with fourteen partner submissions has an advantage that no specification sheet shows. The second consequence is about how to read this page. Eight of the ten accelerators here publish no independently verifiable performance figure. For hyperscaler silicon that is structural, because the chips are not sold and the only customer is the parent company. For the specialists it is a choice, and it is a choice worth asking about during procurement. ## Choosing on the constraint ![Memory capacity and bandwidth across seven accelerators, with measured results marked separately](https://storage.ghost.io/c/85/42/8542ca3e-ec07-47c4-9d72-d69d172c1781/content/images/2026/08/02-memory-capacity-and-bandwidth-mobile-760.png) Memory capacity decides which models fit. At 288 GB the MI355X holds models that need sharding on 192 GB parts, and sharding costs interconnect traffic that the benchmark numbers already include. Memory bandwidth decides token generation speed. This is the number to compare when the workload is a chat interface or an agent loop, because generation is bound by how fast weights move, not by how much arithmetic the chip can do. Software maturity decides how much of the peak is reachable. Ironwood is the clearest case: a fast chip with an inference stack that shares nothing with the CUDA ecosystem most teams already run. Availability decides whether the comparison matters at all. Maia 200 cannot be rented by anyone outside Microsoft, Ironwood runs only on Google Cloud, and Trainium only on AWS. Three of the most capable parts on this page are only reachable by choosing a cloud first and a chip second. ## Questions readers ask ### Which AI inference chip is fastest? On the workloads measured in MLPerf Inference v6.0, NVIDIA’s Blackwell parts lead everything they entered, and NVIDIA was the only vendor to enter every workload. On Llama 2 70B, AMD’s MI355X matched the B200 in Offline and beat it in the Interactive scenario. Outside those two vendors there is no measured answer, because nobody else submitted accelerators. ### How much memory does an inference chip need? Enough to hold the weights, which is why capacity appears in every comparison. GPT-OSS 120B activates 5,1 billion parameters per token and still requires all 117 billion to be resident. Current capacities on this page run from 128 GB on Gaudi 3 to 288 GB on the MI350 series. ### Are custom cloud chips faster than NVIDIA GPUs? There is no measurement that answers this. Google, AWS and Microsoft publish per-chip specifications and internal results, and none of the three has submitted its accelerators to MLPerf. Google appeared in v6.0 as a system builder using NVIDIA hardware. ### Is Inferentia3 available? AWS has never announced such a chip. AWS’s inference-specific line stops at Inferentia2, and the company has moved inference workloads onto Trainium, currently Trainium3. ### Should anyone buy Gaudi 3 in 2026? Only with the end of the line accounted for. Intel has confirmed Gaudi will be discontinued as the GPUs replacing it arrive across 2026 and 2027\. The economics can still work for a workload with a known end date and no long support requirement. ### Why do vendors publish peak FLOPS if the real constraint is memory? Peak compute is the easiest number to state and the hardest to disprove. It describes what the arithmetic units could do if data arrived fast enough, which for large language model generation it does not. Memory bandwidth and capacity predict real throughput far better, which is why they appear in every serious comparison and rarely in a headline. ### What is the difference between the Offline, Server and Interactive scenarios? They are MLPerf’s three ways of applying load. Offline submits everything at once and measures maximum throughput. Server applies a request rate under a latency target. Interactive tightens that target hard, which is where chips with more memory bandwidth pull ahead and where AMD beat the B200 in v6.0. ## What to watch Watch whether any hyperscaler submits its own silicon to MLPerf. Google, AWS and Microsoft between them run a large share of the world’s inference on chips nobody outside those companies has measured, and the first one to submit will reset how this comparison is written. Watch the software line more closely than the hardware line. A 2,77 times gain on unchanged silicon is larger than most generational hardware jumps, and it accrues to whichever platform has the most people working on it. The same pattern shows up outside the data centre. [In autonomous racing, where every team runs identical compute](https://www.insidedeeptech.com/a2rl-autonomous-racing-compute/), software has closed most of the gap to a professional driver. For a purchase this year, the practical sequence is unchanged: establish which models must fit, check capacity, then bandwidth, then whether the stack you already run supports the part at all. The chip with the best headline number is rarely the one that clears all three. ## Sources MLPerf Inference v6.0 results, published by MLCommons on 1 April 2026, including the list of twenty-four submitting organisations. AMD’s submission summary and partner results, 1 April 2026\. NVIDIA’s submission summary for the same round. Google’s published specifications for TPU v7 Ironwood. AWS specifications for Trainium3 and Inferentia2\. Microsoft’s Maia 200 datasheet, January 2026\. Company statements for Groq, Cerebras, Qualcomm and Tenstorrent, marked as such in the text. Intel's position on the Gaudi line comes from the company's own statements about its coming GPUs. Where a figure is a company statement and not a benchmark result, the text says so. No figure on this page comes from an aggregator.