Google's TPUs now run AI inference in Meta's data centers
Native PyTorch support for TPUs lowers the barrier to running production AI workloads on hardware outside Nvidia's ecosystem.
Google's custom TPU accelerators are now serving production AI workloads at Meta, enabled by a PyTorch-native version of the SGLang inference engine developed with Meta and RadixArk. A matching open-source vLLM build for TPUs was also released, removing the need for previous JAX-based setups in most cases. Inferact reports a tuned vLLM setup on seventh-generation Ironwood TPUs reaching over 700 tokens per second for one user on Moonshot AI's 2.8-trillion-parameter Kimi K3, versus 452 tokens per second for vLLM on 16 Nvidia GB200s, though that is a vendor-run test. The moves come a day after Amazon open-sourced its own native PyTorch backend for Trainium chips, both seen as efforts to reduce dependence on Nvidia software. RadixArk plans further model support and Day 0 TPU support for new open models by Q1 2027, with more details due at the SGLang Summit in November.
Why it matters
For operators of large AI workloads, the change means Google's accelerators can now be used through PyTorch, the framework most of them already use, rather than mainly through JAX. Competitive pressure on Nvidia's software position is the wider effect, since Amazon's Trainium backend arrived a day earlier. The performance figure is a vendor-run comparison, so how these chips compare in practice remains to be shown by independent testing.
Matters or hype?
Does this story matter, or is it hype? Vote before you see what everyone else said.
Sources
- Tom's Hardware