Databricks Details Lakebase Branching for CI/CD Testing
Databricks detailed how its Lakebase Postgres database's copy-on-write branching cut CI/CD database provisioning time for customer Snorkel AI from over 30 minutes to under 60 seconds.
Quick answer
What did Databricks announce about Lakebase database branching for CI/CD?
Databricks says its Lakebase serverless Postgres database uses copy-on-write branching to give each CI/CD pull request an isolated, writable copy of production data in under 60 seconds instead of over 30 minutes, as shown in a case study with customer Snorkel AI managing about 100 concurrent branches on a roughly 2.5 terabyte dataset.
Key takeaways
- Databricks says Lakebase's copy-on-write branching let customer Snorkel AI cut database provisioning time for CI/CD pull requests from more than 30 minutes to under 60 seconds.
- According to the blog post, Snorkel AI runs roughly 100 concurrent pull-request branches on a production dataset of about 2.5 terabytes, with each branch adding only 5 to 10 gigabytes of storage for its own changes.
- Databricks says branches keep production data secure through private networking, Okta single sign-on for developers, and short-lived OAuth tokens for CI pipelines running as service principals.
- Branch expiration settings automatically delete each pull-request branch once it is no longer needed, which Databricks says keeps storage costs close to the size of production plus active branches.
- Databricks says the same branching approach can give individual AI agent tasks their own isolated, disposable copy of a database.
The problem
Snorkel AI, a company that builds data development tools for AI teams, runs a CI/CD pipeline on Kubernetes where every open pull request gets its own ephemeral namespace for testing. According to Databricks, Snorkel wanted to test code against real data from its production database, an Amazon Aurora PostgreSQL cluster, but restoring a full copy of that database took about 20 minutes per environment. Databricks says this was too slow for a workflow where developers open pull requests throughout the day, so Snorkel had never been able to give every PR its own database copy.
The Lakebase branching approach
Databricks says Lakebase, its serverless Postgres database, let Snorkel create a new, isolated, writable branch of its database from any point in time, similar to a Git branch. Databricks describes these branches as copy-on-write, meaning a new branch does not copy any data at creation. Instead it shares storage pages with its parent branch, and only the pages a branch changes are written as new storage. According to the blog post, Snorkel used AWS Database Migration Service to replicate its Aurora production database into a Lakebase "main" branch, keeping it within about two minutes of production through continuous change data capture. Databricks says Snorkel's CI pipeline calls the Lakebase API to create a new branch from main whenever a pull request opens, sets an expiration time on the branch, and deletes it when the pull request closes. If that cleanup step does not run, Databricks says the branch expires automatically once its time-to-live setting runs out.
Security controls
Databricks says production data moves between Aurora, Lakebase, and Snorkel's CI cluster over private networking rather than the public internet. According to the company, Snorkel's developers authenticate through Okta single sign-on federated to the Databricks account, while CI pipelines run as Databricks service principals that obtain short-lived OAuth tokens through the Databricks SDK. Databricks says Postgres audit logs record the service principal identity behind every query, so each query can be traced back to the pipeline that ran it.
Results Databricks reported
Databricks says Snorkel's database provisioning time fell from more than 30 minutes to under 60 seconds after adopting Lakebase branching, and that the system is designed to support about 100 concurrent pull-request branches on a roughly 2.5 terabyte production dataset. According to the blog post, each pull-request branch writes about 5 to 10 gigabytes of changes on top of the shared base, and Snorkel rolled the system out to 60 to 70 developers across teams. Databricks quotes an unnamed Snorkel developer saying that previous data-copying jobs "took hours and frequently ran into network and other bottlenecks with various CI platforms," and that the team needed a solution meeting its security and compliance requirements for data leaving its virtual private cloud.
Use beyond CI/CD
Databricks says the same branching pattern applies to AI agents that write SQL, run migrations, or change data, with each agent task given its own branch of the main database so it can read and write data without affecting production or other agents' work. The company says agents can run as service principals with short-lived OAuth tokens, and that branches can be reviewed and deleted if an agent's changes are not wanted, or cleaned up automatically through expiration settings.
Source: Databricks blog, "Lakebase branching for CI/CD: lessons from 100 PR databases," published October 31, 2026.
Frequently asked questions
- What is Lakebase branching?
- Databricks says Lakebase branching lets users create an isolated, writable copy of a Postgres database from any point in time using copy-on-write storage, similar to creating a Git branch.
- How did Snorkel AI use Lakebase branching?
- According to Databricks, Snorkel AI gives every open pull request its own Lakebase branch of its production Aurora PostgreSQL database for CI/CD testing, replacing a process that previously took over 30 minutes per environment.
- How does Databricks say Lakebase branching keeps costs down?
- Databricks says copy-on-write storage means a branch only consumes additional storage for the data it changes, and compute for each branch scales to zero when idle.
- Can Lakebase branching be used outside of CI/CD?
- Databricks says the same branching pattern works for AI agents, giving each agent task its own disposable database branch to read, write, and test changes without affecting production.