Xiaomi’s MiLM Plus Releases PROVE: Perception-Aligned Object Removal Metrics RC-S and RC-T With a Real-World Video Benchmark

Details

Object removal from video has improved faster than the metrics used to judge it. Diffusion erasers now reconstruct shadows, reflections, and occluded structure convincingly, yet standard metrics like PSNR, SSIM, LPIPS, ReMOVE, and CFD frequently rank outputs incorrectly. The root cause: erasure is ill-posed and one-to-many — no unique ground truth exists. Xiaomi's MiLM Plus lab released PROVE (Perceptual RemOVal cohErence), accepted at ACM MM 2026. PROVE introduces two perception-aligned metrics: RC-S measures spatial coherence within edited frames using sliding-window Maximum Mean Discrepancy over DINOv2 features; RC-T measures temporal consistency across frames with the same approach. Neither requires a reference video. The paper documents three failure modes in existing metrics: PSNR and SSIM reward cutting diffusion inference steps even as visual quality collapses (regression to the mean); residual shadows occupy few pixels and incur almost no penalty; CFD's SAM-based hallucination detector misfires on legitimate restorations. PROVE-Bench provides the first two-tier real-world video benchmark for object removal. The code is Apache 2.0 with a single CLI entry point (run_prove_metrics.py), requiring Python 3.10+, PyTorch 2.6+, and DINOv2-giant weights. RC-S runs at 134.6ms per frame on a single RTX 4090 — practical for nightly CI gates in commercial video editing pipelines.