Beyond FLOPs: Energy-Aware Knowledge Distillation for Sustainable LLMs on Code-Related Task

Details

The AI industry uses FLOPs as a universal shorthand for computational cost, but this paper demonstrates it's a misleading proxy for actual hardware energy consumption. Using Morph, a many-objective optimization distillation methodology, the authors show that student models guided by direct CPU/GPU energy surrogates substantially outperform those optimized for FLOPs reduction.\n\nThe benchmark covers software engineering tasks: clone detection, vulnerability prediction, and code summarization (using CodeT5+). Distilled models reduce inference energy by up to 90% and memory usage by 86%, with only modest accuracy trade-offs. The key finding is that FLOPs-optimized models do not reliably achieve these savings because FLOPs don't map cleanly to energy consumption on modern hardware with complex memory hierarchies and cache effects.\n\nFor teams running LLM inference at scale — particularly on SE workloads like code review or vulnerability scanning — this methodology directly affects infrastructure cost calculations. Replacing a FLOPs budget with an energy budget can yield dramatically better cost/accuracy tradeoffs. The SE focus is specific, but the Morph methodology is general and applicable to other distillation scenarios.\n\nEnergy-surrogate models in the optimization loop require measuring actual hardware power during training, which adds engineering overhead but is increasingly tractable.