Databricks Adds On-Demand State Repartitioning for Spark Structured Streaming

Databricks announced on-demand state repartitioning for Apache Spark Structured Streaming, a Public Preview capability that lets engineers resize the number of partitions in a stateful streaming query without rebuilding the checkpoint or losing accumulated state. The feature is available in Databricks Runtime 18 and above.

What changed

According to Databricks, stateful streaming queries have historically had their partition count locked in at the moment the checkpoint was created. Changing the standard spark.sql.shuffle.partitions setting and restarting the query had no effect, because the partition layout was baked into the checkpoint. The only prior option, Databricks says, was to discard the checkpoint and start a new one, which meant losing all state the query had built up, such as fraud-tracking data across millions of accounts or multi-day session windows.

Databricks says on-demand state repartitioning removes that restriction. It applies to any stateful streaming query, including aggregations, stream-stream joins, deduplication, sessionization, and transformWithState workloads.

How it works

Databricks says users set a dedicated configuration, spark.sql.streaming.stateStore.partitions, and restart the query. This setting takes precedence over spark.sql.shuffle.partitions for stateful queries. Upon restart, the query first completes any pending microbatch, then performs a one-time repartition operation that physically redistributes state data across the new partition count, re-hashing keys into their correct locations. Once that completes, the query resumes processing under the new partition count.

Databricks says the requirements to use the feature are Databricks Runtime 18 or above and the RocksDB state store provider, which is the default for new queries created on DBR 17.3 and above.

Monitoring

Databricks says the repartition operation's duration appears in standard StreamingQueryProgress events, specifically in the durationMs metrics under a field called controlBatch.REPARTITION. The company says repartition time is proportional to the amount of state involved but expects it to take only a few seconds for most workloads.

Use cases cited

Databricks describes three scenarios for the feature: right-sizing a query after launch, when an initial partition count set for a small pilot no longer fits growth, such as a fraud-scoring stream expanding from one region to many; adjusting for changing workloads, such as an ad-bidding pipeline that could scale partitions up for daytime traffic and down overnight; and backfilling historical data, where a higher partition count could be used temporarily to speed up reprocessing of past data before scaling back down for steady-state traffic.

Customer example

Databricks quoted Alexis Chicoine, Senior Software Developer at Coveo, an early adopter of the feature, saying Coveo runs large-scale stateful streaming pipelines with fluctuating data volumes and has cut related Amazon S3 API costs by 40% using the capability. Chicoine said the company previously had to choose between overprovisioning or rebuilding from new checkpoints, which drove storage API costs close to compute costs, and can now scale without disrupting existing state or triggering checkpoint migrations.

Source: Databricks blog, "Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks," published September 14, 2026.