Navigating Runtime Dynamism in Semantic Query Serving
| Source: arXiv AI
Tags: SeqoFlow, semantic query processing, LLM inference, query optimization, scheduling
SeqoFlow is a runtime for LLM-powered semantic queries that adapts query plans mid-execution and schedules concurrent queries together, cutting end-to-end latency compared with existing semantic query systems.
Details
Semantic queries mix relational processing with several LLM-based operators over unstructured data. Serving them is hard because arrivals and resource demands change across queries, and operator costs and selectivities change within a query, which makes static plans unreliable. SeqoFlow enumerates equivalence-preserving plan alternatives, splits queries into fine-grained units for safe plan switching and preemption at batch boundaries, and uses lightweight speculative estimation of runtime costs to guide both plan adaptation and inter-query scheduling. The authors report substantial end-to-end latency reductions over existing semantic query systems, adaptive query processing techniques and conventional scheduling, though the abstract gives no specific numbers.