Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction

Details

Google has deployed a new on-device inference acceleration technique for Pixel 9 and 10 series devices. The approach retrofits Multi-Token Prediction (MTP) onto existing frozen Gemini Nano v3 production models without requiring retraining, using what the team calls a 'late exit' strategy.\n\nTraditional speculative decoding requires a separate smaller drafter model to propose candidate tokens for a larger verifier to check. On mobile devices, this drafter competes for scarce RAM and must be separately maintained for each new task. Google's approach instead adds lightweight architectural components directly to the frozen production model, enabling it to generate multiple tokens per forward pass internally — eliminating the drafter entirely.\n\nThe technique builds on prior frameworks (EAGLE and CALM from speculative decoding literature) and is specifically tuned for mobile constraints: strict energy budgets, fixed RAM limits, and the need to avoid task-specific retraining. It is already deployed for AI Notification Summaries and Proofread features on Pixel 9 and 10.\n\nFor developers targeting on-device AI, the practical benefit is removing the need to fine-tune or maintain separate drafter models for new tasks — a significant operational friction point in edge AI deployment. Google recently made a similar MTP acceleration available for Gemma 4 through developer tools.