AWS Launches Model Caching to Cut Inference Cold Starts on SageMaker HyperPod
AWS announced model caching for Amazon SageMaker Inference on HyperPod, pre-loading model weights and container images to cut pod startup time from tens of minutes to seconds.
Quick answer
What did AWS announce for reducing inference cold starts on Amazon SageMaker HyperPod?
AWS launched model caching for Amazon SageMaker Inference on HyperPod. The feature pre-loads model weights and container images onto cluster nodes so pods can read from local NVMe storage at about 7 GB/s instead of downloading over the network, letting pods typically start serving traffic in seconds rather than tens of minutes, AWS says.
Key takeaways
- AWS says deploying an LLM on SageMaker HyperPod involves two sequential downloads before a pod can serve traffic: the inference server container image from Amazon ECR, then the model weights from a storage source such as Amazon S3, FSx for Lustre, or HuggingFace Hub.
- According to AWS, large models like DeepSeek-R1 at over 600 GB can take 30 minutes or more to become ready, and every scale-out event repeats the same download cycle.
- AWS states that model caching pre-loads model weights and container images onto cluster nodes before pods need them, letting pods read from local NVMe storage at approximately 7 GB/s instead of downloading over the network.
- AWS says the feature includes a weights cache, enabled by adding modelCacheConfig with weightsCache to an InferenceEndpointConfig or JumpStartModel resource, which triggers the HyperPod Inference Operator to create a ModelDataCacheConfig resource and begin downloading weights ahead of time.
- AWS says the two capabilities in model caching can be enabled together or separately.
Amazon Web Services announced model caching for Amazon SageMaker Inference on HyperPod, a feature designed to shorten the time it takes for inference pods to begin serving traffic.
The cold start problem
According to AWS, deploying a large language model for inference on SageMaker HyperPod involves a gap between requesting a pod and that pod being ready to handle requests. AWS attributes this gap to two sequential downloads: the inference server container image from Amazon Elastic Container Registry, and the model weights from a storage source such as Amazon S3, Amazon FSx for Lustre, or HuggingFace Hub.
AWS says container images for servers like vLLM or LMI are multi-gigabyte and take 5 to 7 minutes to pull because they bundle GPU drivers, CUDA libraries, and the serving framework. After that, the inference server downloads model weights. AWS states that a 145 GB model on Amazon S3 can take more than 20 additional minutes depending on network conditions, and a model over 600 GB, such as DeepSeek-R1, can take upwards of 30 minutes in total.
AWS also says this download cycle repeats for every new pod during a scale-out event. In its example, if traffic spikes and a HorizontalPodAutoscaler requests five new pods, all five go through the download sequence independently. AWS notes that while the autoscaling policy may react within seconds, the actual time before additional traffic can be served is 25 to 30-plus minutes, since each pod must complete its downloads first.
How model caching works
AWS says model caching pre-loads data onto nodes before pods are scheduled, introducing two independent capabilities that can be enabled together or separately.
One of these, the weights cache, downloads model weights to local NVMe storage on each node ahead of time, according to AWS. AWS describes the setup process as adding modelCacheConfig with weightsCache enabled to an InferenceEndpointConfig or JumpStartModel resource and applying it. Once applied, AWS says the HyperPod Inference Operator automatically creates a ModelDataCacheConfig resource and begins downloading model weights from the configured source, which can be Amazon S3 among others.
With model caching enabled, AWS says pods can read from local NVMe storage at approximately 7 GB/s instead of downloading over the network. As a result, AWS states that pods can typically start serving traffic in seconds rather than tens of minutes.
Source: AWS Machine Learning Blog, "Reduce inference cold starts on Amazon SageMaker HyperPod with model caching," published September 10, 2026.
Frequently asked questions
- What problem does model caching address, according to AWS?
- AWS says it addresses the gap between requesting a pod on SageMaker HyperPod and that pod being ready to serve traffic, a gap it attributes to sequential downloads of the container image and model weights.
- How much faster does AWS say pods start with model caching enabled?
- AWS says pods can typically start serving traffic in seconds rather than tens of minutes once model caching is enabled.
- What are the two capabilities within model caching?
- AWS describes a weights cache, which pre-loads model weights to local NVMe storage on each node, as one of two independent capabilities that can be enabled together or separately; the announcement's excerpt does not fully detail the second.
- How is the weights cache enabled?
- AWS says a user adds modelCacheConfig with weightsCache enabled to an InferenceEndpointConfig or JumpStartModel resource, which causes the HyperPod Inference Operator to create a ModelDataCacheConfig resource and start downloading weights from the configured source.
Sources
- Reduce inference cold starts on Amazon SageMaker HyperPod with model caching | Artificial Intelligence — Amazon Web Services (AWS)