Low-Rank Hierarchical Merging for Efficient Long-to-Short Reasoning
| Source: arXiv AI
Tags: Model Merging, Long-to-Short Reasoning, Hessian Trace, Chain-of-Thought, NeurIPS 2026
Low-Rank Hierarchical Merging, a NeurIPS 2026 paper, merges long chain-of-thought and base models using second-order curvature, reporting over 4% accuracy gains and up to 93.1% shorter outputs without training.
Details
Long-to-Short model merging tries to keep the reasoning strength of long chain-of-thought models while adopting the concise style of base models. Methods based on linear interpolation and first-order activation statistics lose accuracy on hard tasks. LHM is training-free and swaps those heuristics for second-order curvature. It estimates per-layer Hessian traces with Hutchinson's estimator on random inputs, so no task data is needed. A Layer Second-Order Sensitivity index combines the Hessian trace with task-vector norm, and a Dual-Constraint Coefficient Mapping turns those scores into per-layer merge coefficients under a fusion-error bound and a global length-reduction target. On six math benchmarks with 1.5B and 7B models, LHM beat ACM-TIES and ORCA, with over 4% accuracy gain and up to 93.1% shorter outputs than the long-CoT model, which cuts inference cost for reasoning workloads.