Three Generations of Autoscaling — And Why Agentic Traffic Breaks All of Them

Details

Backend architect Shoumik Chakravarty identifies a structural mismatch between the traffic assumptions baked into a decade of autoscaling infrastructure and the actual behavior of agentic AI workloads. Human-driven traffic has predictable diurnal curves, gradual ramp-up times, statistically independent users, and self-limiting retries (people give up). Agent traffic has none of these properties. The article contrasts two scaling generations in detail. On-demand scaling (spin up instances as load grows) fails because a parallel agent fan-out or tight orchestration loop reaches full load in milliseconds—faster than reactive scaling can respond. Serverless scaling fails because agent workflows are stateful, and the cold-start and stateless assumptions baked into serverless architectures conflict with multi-step agent sessions. The author proposes three design shifts: budget-aware retry policies (explicit retry budgets so one fault doesn't become a storm), intent-based routing (route by expected token cost rather than request count), and speculative scaling (pre-warm capacity based on orchestration signals rather than observed load). Concrete, actionable advice for infrastructure engineers—though the article is from a practitioner rather than a research paper, and the proposals are not benchmarked.