Luce: Relightable Gaussians for 3D Asset Generation

Details

Apple ML Research published Luce, a new system for image-to-3D generation that jointly captures geometry and physically-based rendering (PBR) materials — albedo, metallic-roughness, and surface normals — enabling assets that can be relit directly in standard rendering pipelines. The core innovation is a voxelized multimodal Gaussian cloud with dedicated Gaussian primitives for each PBR modality. A variational autoencoder compresses this into a material-aware latent space; a rectified-flow transformer then generates that latent from a single input image using multi-layer features from a pretrained encoder. The output is relightable PBR Gaussians plus an optional textured mesh with a tangent-space normal map — both compatible with conventional game engines and DCC tools. On the Toys4K benchmark, Luce improves FID by 28% over the strongest prior method. Apple also introduces a new benchmark of AI-generated images, where Luce achieves a CLIP alignment score of 0.8519 versus the best competing score of 0.8299. The system notably preserves fine details like text, logos, and inscriptions — a known failure mode for many 3D generation approaches. Apple has not announced a product or API tied to Luce. The work has practical implications for 3D content pipelines, AR/VR asset authoring, and game art workflows, but commercial availability remains unconfirmed.