Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

Details

Cross-scale model fusion asks whether a small model can benefit from a larger one despite mismatched architectures — without the expense of training. APM (Activation-Prune-Merge) proposes a training-free approach: use task-conditioned activation maps on the large donor model to identify the most salient layers, attention heads, and neurons, prune the donor down to match the recipient architecture, then inject that compressed donor slice into the recipient using a micro interpolation coefficient.\n\nThe key insight is treating the donor as a source of concentrated functional components rather than requiring precise neuron-wise semantic alignment between models. This sidesteps a major challenge in heterogeneous model fusion.\n\nAcross 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves a 3B recipient from an overall average of 55.5% to 60.6% accuracy. Individual task gains are substantial: RTE from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, BoolQ from 70.8% to 79.2%. The approach requires no gradient computation and no labeled data for the fusion step itself, making it potentially useful for teams that have access to a strong large model but need to deploy a smaller one.