DEPT: Document Embedding Preservation Tuning for Unified Query Expansion and Retrieval

August 19, 2026 | Source: arXiv AI

Tags: RAG, retrieval, query expansion, BEIR, Qwen3, LLaMA, information retrieval

DEPT trains a single decoder-only LLM end-to-end as both query expander and retriever, solving the moving-target problem by anchoring document embeddings while retrieval gradients improve query generation — outperforming staged and independent baselines on five BEIR datasets.

Details

Retrieval-augmented systems typically separate query expansion and document encoding into independently trained modules or optimize them in staged pipelines, leaving expansion quality only loosely tied to retrieval outcomes. DEPT (Document Embedding Preservation Tuning) closes this loop. A single Qwen3-4B or LLaMA-3.2-3B model both expands the query and encodes it alongside candidate documents. The core challenge: updating the model to improve query-side expansion also shifts document embeddings, creating a moving target. DEPT addresses this by keeping tuned document embeddings close to cached initial embeddings via a preservation loss, while routing retrieval gradients through straight-through decoding back into the generator. Experiments on five BEIR benchmark datasets show DEPT outperforms training-free expansions, independently trained modules, and staged unified baselines. Ablations confirm each component — preservation, whitening, end-to-end expansion training, and online hard-negative mining — contributes meaningfully. Code is publicly available. For teams building RAG pipelines, DEPT suggests that unified training with document-anchoring may outperform the common practice of separately tuning an expansion LLM and an embedding model.