Databricks details Zepto’s evaluation-first customer support agents
Databricks says Zepto uses Databricks and MLflow to evaluate and operate AI agents handling more than 100,000 support tickets daily.
ひとことで言うと
How does Zepto use Databricks and MLflow to evaluate its customer support AI agents?
Databricks says Zepto built a dual-loop evaluation framework for customer support agents using Databricks and MLflow. The system combines development testing, production monitoring, execution traces, golden datasets and deployment quality gates. According to Databricks, the agents fully manage more than 80% of support tickets with human oversight.
要点
- Databricks says Zepto’s multi-agent customer support system processes more than 100,000 tickets per day.
- The framework connects development and production loops through a quality gate, with production failures fed into subsequent development.
- Zepto uses MLflow traces, evaluation datasets, automated prompt optimization and scoring to test agent behavior.
- According to Databricks, AI agents fully manage more than 80% of tickets with human oversight and have reduced support costs by 65%.
- Databricks reports a payback period of less than one month, a 20% improvement in customer satisfaction and an 8% improvement in accuracy.
Databricks published details of an evaluation framework that Zepto uses to build and operate customer support AI agents on Databricks and MLflow. According to the company, Zepto’s multi-agent system processes more than 100,000 support tickets a day, with AI agents fully managing more than 80% of tickets under human oversight.
Databricks says the framework helped Zepto reduce support costs by 65% and achieve payback in less than one month. The company also reports a 20% improvement in customer satisfaction, an 8% improvement in accuracy, development cycles that are three times faster and a fourfold improvement in time to resolution.
Development and production loops
The framework uses separate development and production loops linked by a quality gate. Databricks describes the development loop as the environment where teams design, revise and evaluate agent versions before release. The production loop monitors live behavior and detects failures after deployment.
A feedback loop sends production failures back into development, while the quality gate determines whether an agent version can enter production or must return for further work. Databricks says failures are automatically captured and used in later iterations.
The approach begins with execution tracing. Each agent invocation records prompts, completions, retrieved documents, tool calls, latency and decision paths. Databricks says Zepto uses MLflow automatic tracing and custom trace annotations, with traces emitted as OpenTelemetry spans carrying unique identifiers. MLflow’s integration with Unity Catalog centralizes the logs in Delta tables.
Evaluation pillars and datasets
Zepto defines evaluation pillars based on stakeholder requirements, according to Databricks. The stated examples cover customer experience, operational efficiency, risk and compliance, and financial impact. Each pillar receives numeric thresholds that an agent must meet before deployment.
A golden dataset serves as the reference for evaluating behavior during development. It includes routine cases, edge cases and failures, along with stakeholder expectations and metadata such as scenario type, business line and risk level. Databricks says security teams can contribute adversarial examples involving prompt injection, identity attacks and attempted data exfiltration.
Over six months, Zepto expanded its MLflow evaluation dataset from 500 examples to 2,000 and then 5,247, according to the post. Databricks says the difference between development and production accuracy declined from eight points at 500 examples to two points at 2,000 examples and 0.4 points at 5,247 examples. Production failure traces are added to the dataset for evaluating future versions.
Prompt testing and scoring
Databricks says Zepto also uses MLflow prompt optimization instead of relying solely on manually written prompts. The process registers an initial prompt, produces and optimizes variants, evaluates them with the same scorers used for deployment decisions and deploys the highest-performing result.
The optimizer uses a stronger model for reflection and a less expensive model to score production candidates, according to Databricks. Candidate prompts are measured against the golden dataset before they reach production. The framework also scores production traces across the evaluation dimensions established for agent quality.
Source: Databricks’s “Evaluation-First AI Agents: How Zepto Scales Customer Support on Databricks and MLflow,” published September 9, 2026.
よくある質問
- What did Databricks announce about Zepto’s customer support system?
- Databricks described how Zepto built an evaluation-first framework for customer support AI agents on Databricks and MLflow.
- How many support tickets does the system process?
- Databricks says Zepto’s multi-agent system processes more than 100,000 support tickets each day.
- What is the dual-loop architecture?
- It consists of a development loop and a production loop connected by a quality gate. A feedback loop returns production failures to development for later iterations.
- What results did Databricks report?
- Databricks says AI agents fully manage more than 80% of tickets with human oversight. It also reports 65% lower support costs, payback in less than one month, a 20% improvement in customer satisfaction and an 8% improvement in accuracy.