WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups

Postgres backups are one of those pieces of infrastructure that should be boring. They sit in the background, continuously archiving WAL files, uploading backups, and making sure that when something goes wrong, recovery is possible. At ClickHouse Cloud, this path is critical. WAL archival is what allows us to preserve durability and recoverability for our Postgres services. WAL-G has been a strong and reliable tool for this job. It is mature, battle-tested, and has served the Postgres community well. But as we pushed Postgres into tighter and more resource-constrained environments, we started hitting a specific problem: memory predictability. That led us to build WAL-RUS, an open-source Rust-based implementation of Postgres backup and WAL archival tooling, designed for predictable memory-efficiency and WAL-G compatibility. WAL-G is written in Go, a garbage-collected language. While Go makes it easy to build reliable infrastructure software, garbage-collected runtimes make memory usage harder to predict, especially for long-running services like WAL archival. The challenge isn't just resident memory (memory actively being used), but also virtual memory (memory reserved from the operating system). Go's runtime manages its own memory pools and can reserve significantly more virtual memory than the application is actively using. As workloads change, this footprint can fluctuate in ways that are difficult to reason about and tune. The Go GC guide describes this as a characteristic "sawtooth" pattern, where memory usage grows between garbage collection cycles and then falls after collection, making it difficult to predict peak memory consumption and provision resources efficiently. For operators, that creates a simple but important problem: how much memory should be reserved for backup infrastructure? The answer is usually "more than necessary" to avoid unexpected memory pressure. Memory budgeted for WAL archival is memory that cannot be confidently allocated to Postgres itself for queries, shared buffers, and page cache. Postgres runs most reliably with overcommit disabled, making virtual memory a valuable resource modern software often leaves as an afterthought. WAL-G remains a proven and reliable tool, but as we scaled Postgres into increasingly resource-constrained environments, we wanted a backup system with a more predictable memory profile, delivering the same functionality while consuming fewer resources and making capacity planning simpler. We weren't looking for new functionality. WAL-G is a mature and reliable backup system we’re happy to contribute to. Our goal was to preserve core functionality and compatibility while providing a more predictable resource profile. WAL-RUS is a Rust implementation of Postgres backup and WAL archival tooling built to address the operational challenges we encountered with memory predictability and resource usage. 1. Predictable Resource Usage: Unlike garbage-collected runtimes, Rust gives us direct control over memory allocation and concurrency. WAL-RUS uses bounded worker pools and carefully controlled concurrency, making memory consumption easier to reason about and reducing the need to over-provision resources for backup infrastructure. 2. Built for Continuous WAL Archival: WAL-RUS prioritizes WAL-G’s daemon architecture. Instead of spawning a new process and establishing new connections for every WAL file, it maintains persistent object storage connections that continuously process archival requests in the background. 3. Optimized for Streaming Workloads: WAL archival is fundamentally a streaming problem: read WAL files, compress them, and upload to object storage. WAL-RUS minimizes unnecessary buffering and data copies throughout this pipeline, allowing it to perform the same archival work with a smaller and more predictable memory footprint. 4. WAL-G Compatibility: WAL-RUS uses the same WALG_ configuration variables as WAL-G and is continuously tested for interoperability. WAL-G can read archives generated by WAL-RUS, and WAL-RUS can read archives generated by WAL-G, making migration straightforward for existing deployments. To evaluate WAL-RUS, we built a reproducible benchmark that compares WAL-RUS, WAL-G, and pgBackRest under a sustained, WAL-heavy PostgreSQL workload. The benchmark continuously generates WAL, archives it to S3, and measures how efficiently each archiver uses memory while keeping up with WAL generation. To ensure a fair comparison, all three tools were configured with four concurrent archival workers. Memory efficiency was the primary motivation behind WAL-RUS, making memory consumption the first metric we examined. WAL-G reached nearly 2.8 GB of peak virtual memory during the benchmark, while WAL-RUS remained below 1 GB, a reduction of more than 70%. WAL-RUS also maintained a stable memory profile throughout the run, making its resource requirements easier to reason about in production environments. pgBackRest deserves credit

WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups

Key takeaways

More in computer-science