Guarded Compression of Adam Optimizer State with Shared-Gradient Bounds
| Source: arXiv AI
Tags: Adam optimizer, optimizer state compression, OPT-125M, ResNet-18, checkpointing
Guarded compression of Adam's moment state shrinks the optimizer payload to 28.3% of raw FP32 at τ=0.01 on OPT-125M while guaranteeing restored updates stay within tolerance, which matters because Adam moments take twice the memory of the weights.
Details
Adam stores two FP32 moments per parameter, twice the bytes of the weights, but compressing them is risky because the update divides by the square root of the second moment. The authors accept a coordinate's compressed moments only if they reproduce the native FP32 Adam update within a tolerance τ for every gradient in a clipped domain; other coordinates are stored exactly. A naive per-update certificate fails: on one million OPT-125M coordinates it rejects 98.9% and the payload grows to 126.6% of raw FP32. Bounding the difference between both updates directly, since they share the same gradient and parameter, cuts rejections to 0.53% and the payload to 28.3% at τ=0.01. Tests cover 120 full checkpoint restores of OPT-125M and ResNet-18 with guarded 8-bit log codes. This is relevant for anyone fighting checkpoint size and optimizer memory in training pipelines.