Google Launches LiteRT Tensor API for Building, Customizing, and Optimizing ML Models for On-Device

| Source: InfoQ AI/ML

Tags: Google, LiteRT, on-device inference, NPU, C++, safetensors

Google released the LiteRT Tensor API, a C++ API for building on-device ML model graphs op by op that run on CPU, GPU and NPU without rewriting code for each hardware backend.

Details

Google introduced the LiteRT Tensor API, a lightweight C++ API for building, customizing and optimizing models directly for on-device execution. Google says hand-written C++ computation is usually tied to backends such as Metal, OpenCL or Vulkan and must be rewritten per target, while zero-copy buffers and synchronization across CPU, GPU and NPU take heavy manual work. The new API lets developers build computation graphs operation by operation and run the same graph through LiteRT across CPU, GPU and NPU. It also supports loading weights directly from checkpoints including Hugging Face .safetensors files, zero-copy buffer interoperability across pre-processing, inference and post-processing, and use by AI coding agents that construct, validate and benchmark graphs.