Reduce GVisor Cold Starts with GPU Snapshotting

Yaseen Hamdulay Member of Technical Staff Reducing GPU Cold Starts with Memory Snapshots: Restoring CUDA Workloads in Seconds If you run AI models in production, you have a relationship with cold starts whether you want one or not. A three-minute startup time changes how you scale. You keep GPUs warm that could have been released. You over-provision to avoid making users wait. You stretch cooldown periods because scaling down too quickly creates pain on the next spike. The application starts accumulating complexity around one problem: getting a model ready to serve traffic fast enough. At Cerebrium, we have been obsessed with the cold start problem since day one. That obsession has pushed us to rethink almost every layer of our infrastructure: Custom VM images for faster node scale-ups A custom image runtime for sub-second container image cold starts A highly available, low-latency orchestrator for routing workloads across regions and clouds CPU and GPU memory snapshots for restoring fully warmed containers in seconds As more companies move large custom AI models into production, they hit the same wall. Our customers run large language models, real-time avatars, transcription models, diffusion models, and other GPU-heavy workloads where startup time can vary from a few seconds to more than five minutes. Most of that time is spent on work that gets a container ready to serve requests: importing libraries, loading model weights, initializing CUDA, compiling kernels, and warming up the runtime. That is the core problem checkpointing solves. Instead of rebuilding the same runtime from scratch each time a new container starts, we snapshot the fully initialized container - including CPU memory, GPU memory, process state, model weights, and compiled kernels - and restore it directly into a new container in a fraction of the time. For some workloads, this reduces cold start time by more than 80%! This post explains how we built CPU and GPU memory checkpointing at Cerebrium, how it works inside our highly customised gVisor-based runtime, and what it took to make real CUDA workloads like vLLM restore reliably and quickly. It s tempting to think of cold starts as simply “pulling the image”: downloading the application image onto the machine that will run the container. But for AI workloads, that is only the first part of getting a model ready to serve traffic and it is no longer the bottleneck. We have solved the container download problem already. The real cost in a CPU or GPU container is everything that happens after the image is on the machine and the application starts initializing. That initialization path includes importing Python modules, loading PyTorch, assembling model weights, copying them onto the GPU, and running the framework’s warmup path - torch.compile, CUDA graph capture, KV cache initialization, and whatever else the serving stack needs before it can take traffic. Every one of these stages is deterministic. Importing PyTorch produces the same loaded modules every time. Building the model and copying weights onto the GPU produces the same bytes in GPU memory every time. torch.compile and CUDA graph capture produce the same kernels every time. Yet on every scale-up, we pay to recompute a result that is known. The idea is simple: do the expensive startup work once, freeze the result, and restore it on demand. Pause execution: pause all application processes, threads, and, crucially, GPU work. Dump memory: serialize the in-memory state from both CPU and GPU to files. Upload: push those files to fast, durable storage. Restoring runs the same process in reverse. We pull the checkpoint files down, rehydrate CPU and GPU memory, repair the pieces of state that cannot survive a move, and unpause the workload. The restored application process is the same warmed-up runtime we froze earlier: PyTorch has already been imported, model weights are already resident on the GPU, kernels are already compiled, and the application is ready to serve traffic. The mental model is straightforward. Making it work reliably for real GPU workloads is not. At a high level, checkpointing needs to sit in the one place where it can control the full lifecycle of a container: between the container runtime and the sandbox running the workload. Cerebrium runs user workloads inside gVisor sandboxes for isolation. To support checkpointing, we extended that runtime path so that when a container starts, we can make a decision before the normal boot sequence completes: Should this container start from scratch, or should it be restored from a checkpoint? If no checkpoint exists, the container follows the normal path. The image starts, the application boots, models load, GPU memory is populated, and the workload becomes ready. Once the container is fully warmed, the user can trigger a checkpoint. At that point, we pause the workload, capture its CPU and GPU state, write the checkpoint to disk, and upload it to fast

Reduce GVisor Cold Starts with GPU Snapshotting

Key takeaways

More in computer-science