Show HN: ZeroFS – A log-structured filesystem for S3

ZeroFS serves S3-compatible buckets as POSIX filesystems over NFS and 9P, or as raw block devices over NBD. The engine is log-structured: writes go out as immutable objects and compaction reclaims what you delete. Data is compressed and encrypted before upload, and warm reads come from local cache in microseconds. or via Docker · GitHub Action · other platforms CI runs pjdfstest, xfstests, kernel builds, stress-ng, ZFS scrubs, Jepsen's local-fs suite, and a Jepsen failover suite on every change. The first three run separately over NFS, 9P, and the FUSE client. Each card links to the CI pipeline. The pjdfstest suite runs on every change, once per protocol: permissions, ownership, links, rename behavior. The exclude lists, a few cases per protocol, are published in the repository. The kernel filesystem test suite runs over NFS, 9P, and FUSE in separate workflows. These are the tests ext4 and XFS themselves are validated against. CI builds a ZFS pool on ZeroFS block devices, extracts the Linux kernel source tree onto it, and runs a full scrub. The scrub reports no checksum errors. CI compiles the Linux kernel on NFS, 9P, and FUSE mounts with make -j$(nproc). Parallel compilation is the stress test: many processes writing the same tree at once. Filesystem stressors run against live mounts in CI: access, chdir, chmod, chown, and the rest of the file-handling set, all at once. Jepsen's local-fs suite drives a 9P mount with random operation histories and checks each against a reference filesystem model. A crash mode kills the server mid-run and verifies the recovered state matches the last fsync. A second Jepsen suite runs a leader and standby over MinIO, then kills, restarts, and pauses the leader mid-workload. It confirms every acknowledged write survives the failover and the recovered filesystem stays consistent. The Rust toolchain builds ZeroFS on a filesystem that ZeroFS itself is serving. This one is a recorded session rather than a CI job. All three servers run in one userspace process against the same bucket. Clients mount it with the NFS and 9P support already in Linux, the NFS clients other systems ship, or nbd-client for block devices. NFS 9P NBD S3 ENCRYPTED OBJECTS NFS 9P NBD S3 ENCRYPTED OBJECTS File · everywhere NFS macOS, Linux, Windows, and the BSDs mount it over their own NFS support, with nothing extra installed on the client. The server stays in userspace. # mounts from any major OS mount -t nfs 127.0.0.1:/ /mnt/zerofs File · precise 9P 9P follows POSIX more closely than NFS, and fsync returns only after data reaches stable storage. The bundled FUSE client mounts without root and reconnects on its own. # bundled FUSE client, no root zerofs mount 127.0.0.1:5564 /mnt/zerofs Block · raw NBD Raw block devices stored in the bucket hold ext4 filesystems, ZFS pools, or VM boot disks. New devices are picked up at runtime, with no server restart. # attach a block device nbd-client 127.0.0.1 10809 /dev/nbd0 -N vol1 Runs onAmazon S3Google Cloud StorageAzure BlobAny S3-compatible storelocal disk 03Geo-distribution A ZFS mirror across three S3 regions. Each ZeroFS instance exposes one S3 region as a block device. To ZFS they are plain disks, so a mirror that spans continents is set up like any other pool. If a region becomes unreachable, the pool degrades and the data stays available from the other two. Every block is encrypted with XChaCha20-Poly1305 before upload. The data key is wrapped with a key derived from your password using Argon2id. There is no unencrypted mode. Data is compressed with zstd or lz4 before encryption. The codec can change at any time without migration, since the codec of existing data is detected on read. Configurable memory and disk caches hold recently used blocks. Warm reads return in microseconds. A raw S3 round trip takes 50 to 300 milliseconds. Named checkpoints capture the filesystem at a point in time. Any of them can be opened read-only with a single flag at startup. One instance writes while read-only instances serve the same bucket. Replicas pick up the writer's changes automatically and return EROFS on writes. Discard from any filesystem or zpool frees the corresponding extents. Compaction repacks the live data and deletes the emptied segments from S3, so freed blocks come off the bill. File data is packed into 32 KiB extents inside immutable segment objects, addressed by a separate metadata index. Nothing is rewritten in place, so checkpoints and read replicas see a consistent view of the bucket. A successful fsync means every acknowledged write is durable in S3. If a failover may have lost unflushed writes, the next fsync returns an error instead of a false success. An optional standby tracks the leader over the same bucket and takes over automatically when the leader fails. While the two are connected, the standby also holds writes the leader has acknowledged but not yet flushed, so a failover preserves them. Enable [servers.webui] and the sa

Show HN: ZeroFS – A log-structured filesystem for S3

Key takeaways

More in computer-science