NeoMME: an efficient Multimodal-native and Multilingual Encoder

Details

H company introduces NeoMME (260M and 800M variants), a multilingual multimodal encoder that breaks from the dominant VLM adapter pattern. Instead of combining a pretrained vision tower, projector, and causal decoder, NeoMME uses a single bidirectional Transformer trained from scratch to jointly process text tokens and raw image patches via a masked discrete-diffusion objective. The practical payoff is concrete: on an NVIDIA L40S at 2048×2048 image resolution, the 260M model encodes ~51 pages per second—roughly double ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization compress late-interaction indexes from ~1.5 MB to 6 kB per page (255× reduction) while keeping >95% of baseline nDCG@10, directly addressing the storage bottleneck of ColPali-style systems. Both model sizes sit on the ViDoRe v3 Pareto frontier for nDCG@10 vs. model size. NeoMME-Retriever returns dense and late-interaction embeddings in one forward pass, enabling flexible retrieval strategies without running the model twice. All checkpoints are available in Hugging Face Transformers today under Apache 2.0. For teams building visual document retrieval over contracts, PDFs, or invoices, the 255× index compression and 2× throughput gains over ColModernVBERT are meaningful reductions in both infrastructure cost and latency.