Sakana AI Researchers Introduce PC-ALM, a Layer-Local Alternative to Backpropagation That Trains 1000-Layer Networks

Details

Backpropagation requires a global backward pass coordinating updates across an entire network simultaneously — a mechanism with no known biological equivalent. Predictive coding (PC) offers a local alternative where each layer only sees its own prediction errors, but it struggles in deep, narrow networks because the credit signal fades before reaching early layers. PC-ALM (Augmented Lagrangian Predictive Coding) from Sakana AI attaches a Lagrange multiplier to each layer's constraint, effectively adding an integral term to the proportional prediction-error signal. This turns each layer into a PI controller that accumulates credit history rather than relying on single-shot diffusion. The result: residual MLPs up to 1000 layers match backpropagation within about 2 percentage points on MNIST. The math is principled — at a KKT optimum in the linear case, the Lagrange multipliers provably equal the backprop adjoints. The JAX reference implementation is MIT-licensed and runs on CPU, making reproduction accessible. The significant caveat: all experiments are on MNIST, a benchmark long considered solved. The method is a proof-of-concept training algorithm, not a system ready for production or standard benchmarks like ImageNet. For researchers studying biologically-plausible learning or hardware-efficient training without global communication, PC-ALM is a clear methodological step forward.