Databricks Details How Its AI Research Team Uses AI Runtime
Databricks says its AI Research team uses AI Runtime to train and fine-tune models at scale, with agent-led experimentation growing training jobs 8x.
คำตอบโดยย่อ
How does Databricks' AI Research team use AI Runtime to train and fine-tune its models?
Databricks says its AI Research team relies on AI Runtime to train and fine-tune models from a few billion to nearly a trillion parameters, using built-in GPU reliability, fast job launches, and MLflow and Unity Catalog integration. The company says agent-led training has increased training jobs 8x.
ประเด็นสำคัญ
- Databricks says its AI Research team uses AI Runtime (AIR) to train and fine-tune models ranging from a few billion to nearly a trillion parameters, using NVIDIA H100 or B300 GPUs.
- The company says AIR performs preflight checks and monitors GPU node health during training, cordoning off unhealthy nodes to prevent future job failures.
- Databricks says AIR integrates with MLflow to automatically track experiments and with Unity Catalog, via FUSE mounting, to give governed access to research data as a local filesystem.
- The company says agent-led training, which it ties to the concept of autoresearch, has increased the number of training jobs for its AI Functions research by 8x.
- Databricks says researchers and agents use the AIR CLI and databricks-air-skills to launch workloads programmatically, with job launches starting in seconds.
Databricks published a blog post describing how its internal AI Research team uses Databricks AI Runtime (AIR), the company's GPU infrastructure product, to train and fine-tune the models behind its AI products.
According to the post, the research team builds models that power Databricks products including Genie, Smart Routing, and AI Functions, such as ai_parse_document and ai_extract. Databricks says this work spans tens of thousands of training and evaluation runs using supervised fine-tuning, offline reinforcement learning, and online reinforcement learning, on models ranging from a few billion parameters to nearly a trillion. The company says its largest training runs use NVIDIA H100 or B300 GPUs and can require hundreds of GPUs running for days or weeks.
GPU reliability and fast iteration
Databricks says AIR performs preflight checks on assigned GPU nodes before training starts and monitors node health during execution, watching for issues such as NCCL communication failures, thermal or power throttling, and GPU driver problems. If a node fails and causes a job to fail, the company says AIR cordons off the affected node so later jobs aren't scheduled on it. Databricks says this matters for projects such as KARL, where post-training models with more than 100 billion parameters can require thousands of GPU hours.
The company also says researchers use the AIR CLI to launch workloads programmatically and specify GPU requirements, with AIR handling provisioning and scheduling, including networking for distributed training across clouds. Databricks says it uses the same custom Docker containers for local development and training, and that investments in launch latency and container optimization let training jobs start in seconds.
Experiment tracking and data access
Databricks says AIR automatically creates and manages MLflow experiments for training jobs, letting researchers monitor job status, stream logs, and retrieve metrics. The company says this also gives its AI agents the information needed to evaluate results and decide what to try next.
The company says research data is stored in Unity Catalog, and that AIR's Unity Catalog FUSE mounting lets researchers access that data as a local filesystem, including across different Databricks workspaces. Databricks says this keeps data accessible, tracked, and governed without requiring changes to training code.
On GPU capacity, Databricks says AIR's workload prioritization and preemption features let it allocate resources across projects with different timelines, including using spare capacity for exploratory experiments, such as over a weekend, without disrupting higher-priority work.
Agent-led training
Databricks says it is increasingly using agents, rather than researchers, to propose changes, run experiments, evaluate results, and plan next steps, a practice the post refers to as autoresearch and attributes to being popularized by Andrej Karpathy. The company says this shift has grown the number of training jobs across its AI Functions research workloads by 8x.
Databricks says its agents use the Databricks CLI and databricks-air-skills to launch concurrent workloads on AIR, monitor job status, and retrieve MLflow metrics to iterate on results programmatically.
Source: Databricks blog, "How our AI Research team uses AI Runtime," published October 15, 2026.
คำถามที่พบบ่อย
- What is Databricks AI Runtime (AIR)?
- According to Databricks, AIR is the infrastructure its AI Research team uses to train and fine-tune AI models, providing GPU provisioning, scheduling, reliability monitoring, and integrations with tools like MLflow and Unity Catalog.
- What models does Databricks train using AIR?
- Databricks says its research team uses AIR to train models that power products such as AI Functions, including ai_parse_document and ai_extract, using supervised fine-tuning, offline reinforcement learning, and online reinforcement learning.
- How has agent-led research affected Databricks' training workloads?
- Databricks says agents now propose experiments, launch trials, evaluate results, and refine approaches, which has grown the number of training jobs for its AI Functions research by 8x.
- How does AIR handle GPU failures during training?
- Databricks says AIR runs preflight checks on GPU nodes and monitors for issues like NCCL communication failures and thermal throttling during execution, cordoning off unhealthy nodes so future jobs avoid them.