Google open-sources ML Drift, a GPU engine for on-device AI across platforms
The engine behind LiteRT now runs on Windows and Linux desktops as well as macOS, and Google is retiring the legacy TFLite GPU delegate.
Google has open-sourced ML Drift under an Apache 2.0 license, a cross-platform GPU compute engine that powers its LiteRT on-device inference runtime and hides differences between GPU hardware and APIs such as OpenGL ES, OpenCL, Metal and WebGPU. Compared with the legacy TFLite GPU delegate, it adds unified shaders via tensor virtualization, extensible custom operations, five-dimensional tensor support, and tuned paths for Arm, Intel and Qualcomm GPUs. On desktop it runs on Windows and Linux through Dawn, Chromium's WebGPU implementation, while keeping native Metal support on macOS. Google reports 12% lower memory overhead than llama.cpp and MLX LM on Gemma models using developer workstations, and says the engine already runs in products such as Chrome and YouTube Shorts, where it cut average frame latency by up to 40% for segmentation effects. Google will stop updating the legacy TFLite GPU delegate and recommends migrating to the LiteRT ML Drift GPU accelerator.
Why it matters
For people who follow where AI runs, the shift is that inference is moving onto the devices themselves and across more kinds of GPU hardware and graphics APIs. Google's reported 12% memory saving on Gemma models and the 40% frame latency cut in YouTube Shorts suggest these gains are already showing up in shipping products, not only in benchmarks. Making ML Drift open source and steering users off the legacy TFLite GPU delegate signals that Google sees on-device GPU inference as the path it intends to standardise on.
Matters or hype?
Does this story matter, or is it hype? Vote before you see what everyone else said.