Custom Incrementalization for Dynamic Tables | Snowflake
| Source: Snowflake Blog
Tags: Snowflake, Dynamic Tables, data engineering, incrementalization, Thrive Learning, data pipelines
Thrive Learning cut Snowflake Dynamic Table auto-clustering costs by 99% by applying custom incrementalization to a deduplication pipeline that was forcing expensive full-table refreshes — a concrete solution to a common scaling trap in modern data warehouses.
Details
Thrive Learning, a UK enterprise learning platform serving 500+ B2B tenants, has published a case study on using Snowflake's custom incrementalization feature to fix a cost spiral in their RecordStore pipeline. The work was led by senior data engineer Rob Howe and deployed to production in roughly two weeks. The problem arose when the pipeline needed to enrich records by joining up to five upstream sources to resolve missing IDs. Those joins produced duplicates, requiring deduplication logic complex enough to break Snowflake's incremental refresh path — forcing full table scans on every refresh cycle. Costs scaled in proportion. The fix used Snowflake's custom incrementalization to restore incremental updates despite the deduplication complexity, preserving the full lineage graph in the process. The team also introduced a split-lane approach — a 'fast lane' refreshing recent data every six hours and a slower lane for historical records — to reduce unnecessary reprocessing. Combined, these changes produced a 99% reduction in auto-clustering costs and a claimed 2x price-performance improvement. The pipeline underpins Thrive's 'Analyse with AI' paid add-on product, so the optimization had direct revenue implications beyond infrastructure savings. This is primarily a Snowflake vendor case study — the numbers are compelling but come from a single customer with no independent verification.