RayRoPE: Projective Ray Positional Encoding for Multi-View Attention

Details

Multi-view transformers process tokens from multiple posed camera images simultaneously and require positional encodings with three properties: patches must be encoded uniquely, attention must be SE(3)-invariant (robust to camera pose changes), and encodings should adapt to actual scene geometry. According to this Apple ML Research paper accepted to ECCV 2026, no prior absolute or relative encoding scheme satisfies all three. RayRoPE addresses this by representing each patch position using a predicted 3D point along the camera ray rather than the ray direction alone. This makes the encoding geometry-aware — sensitive to where in the scene the patch likely falls, not just the direction it came from. For SE(3) invariance, RayRoPE computes query-frame projective coordinates to build multi-frequency similarity, keeping attention consistent under camera rotation and translation. A practical wrinkle: the predicted 3D point may be imprecise, especially with noisy depth estimates. RayRoPE includes an analytical mechanism to compute the expected position encoding under uncertainty, making it robust to prediction error without requiring exact depth. The method also natively supports RGB-D input — positionally encoding depth information — which prior schemes cannot handle. Experiments show 15% relative improvement on LPIPS in the CO3D novel-view synthesis benchmark over alternate encoding schemes, plus consistent stereo depth estimation gains. The work is a collaboration between Apple ML Research and Carnegie Mellon University (Yu Wu, Minsik Jeon, and Shubham Tulsiani are listed as CMU-affiliated).