Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers

Details

Dense embedding models compress entire documents into one vector, which is inherently lossy for complex queries with multiple requirements. ColBERT-style 'late interaction' models instead keep one vector per token and score query-document pairs with a MaxSim operator — matching each query token to its best-matching document token. This preserves token-level matching information and typically yields stronger retrieval, especially for multi-faceted queries.\n\nSentence Transformers v6.0 adds MultiVectorEncoder as a fourth model type alongside dense, sparse, and reranker models. Any PyLate checkpoint or Stanford-NLP ColBERT checkpoint loads directly using the same familiar API, eliminating the need for separate libraries or integration work. Visual document retrieval — matching text queries against page images without OCR — is also supported through colpali-engine models.\n\nThe main trade-off is index size: multi-vector models store one vector per token rather than per document, significantly increasing storage requirements. The blog post covers token pooling and quantization strategies to manage this cost.\n\nFor practitioners building RAG pipelines, this update removes a significant integration barrier. Previously, using ColBERT-style retrieval required a separate library ecosystem (PyLate, PLAID, etc.); now it is a drop-in addition to any existing Sentence Transformers setup. Available immediately via pip install -U sentence-transformers.