Shopify Introduces Gisting: Compressing LLM System Prompts into Learned Tokens

Details

Shopify's engineering team has published details on their production deployment of 'gisting,' a prompt compression technique that converts lengthy LLM system prompts into compact learned embedding vectors. The method builds on a 2022 research paper and works by training gist tokens to minimize KL divergence between model responses to the original prompt versus the compressed tokens — making the model behave as if it had processed the full original system prompt.\n\nApplied to their Sidekick GraphQL agent, Shopify reduced the system prompt from approximately 6,000 tokens to 1,500 gist tokens — a 4:1 compression ratio. At 350 requests per minute, median time to first token dropped from 438ms to 354ms, end-to-end latency fell from 6.8s to 4.2s, and throughput rose from 20.2 to 23.4 queries per second. These gains allowed Shopify to reduce GPU allocation for this workload.\n\nA notable engineering advantage: gist embeddings are written directly into the model's embedding matrix and registered as special tokens, requiring no custom attention masks, extra encoders, or modified serving paths. The model loads and runs identically to any standard deployment.\n\nFor teams running production LLM workloads with long, relatively static system prompts — common in agentic and RAG systems with fixed API schemas or instructions — gisting offers a validated path to latency and cost reduction without model retraining. The technique is most effective where system prompts are stable and can be trained offline.