PRISM2 model uses clinical dialogue to interpret pathology slides

Details

PRISM2 is a multimodal pathology AI developed jointly by Paige and Microsoft that takes a fundamentally different approach to slide analysis. Instead of classifying pixels, it generates natural-language answers to diagnostic questions by training on the intersection of visual data (2.3 million whole-slide images) and clinical text (685,507 pathology reports from Memorial Sloan Kettering Cancer Center, converted to question-answer pairs using GPT-4o). The architecture runs in two distinct phases. The first phase trains a perceiver-based slide encoder using Virchow2 tile embeddings, supervised by both a contrastive loss (pulling slide representations toward matching report language) and an autoregressive loss (forcing the encoder's output to support direct text generation). The dual-loss approach addresses a known limitation: contrastive-only training produces embeddings good at retrieval but weak at generation. The second phase freezes the trained encoder and fine-tunes a separate language model on clinical dialogue, teaching it pathology reporting conventions. PRISM2 exposes two distinct embeddings for different downstream uses: base embeddings for biomarker prediction tasks and diagnostic embeddings — extracted from the language model's hidden state — tuned specifically for cancer detection queries. The clinical dialogue supervision comes entirely from single-turn QA pairs, which limits the kind of interactive back-and-forth a deployed system could support. Real-world deployment and regulatory approval remain ahead. The scale of the training data — drawn from routine MSK care rather than curated research datasets — is notable for its clinical authenticity.