Speeding up the Oxen.ai File System

We started Oxen.ai not just to make the latest AI models easy to use, but to help people customize and train their own. A core piece of that is data storage.

Oxen has two sides: the models, and the file system where all your generations and training data live. Today we're talking about the file system.

We have teams generating and sharing hundreds of thousands of images and videos inside one project to collaborate on films. We have users storing terabytes of data in a single repository, versioning it, and training AI models from scratch. At this scale, three things have to be true: file access has to be fast, every version has to be tracked, and you need provenance for all of it. Where the data came from, who made it, and how it was created.

This post is about the "fast" part. TLDR: by moving our Merkle tree into LMDB, browsing your repo got up to 5.7x faster, commits up to 3x faster, and the first command after a reboot up to 41x faster, all while shrinking the node store 4x.

Speeding Up Our Merkle Tree with LMDB 🏎️

In Merkle Tree 101 and Merkle Tree VNodes we walked through how Oxen models a repository as a tree, and how VNodes keep a single directory with a million files from turning every add into a million writes. This post is about something one layer down, where those nodes actually live on disk.

Until now, every Merkle node was a pair of files in a directory on your filesystem. We moved them into LMDB, a memory-mapped key-value store, and measured what happened on macOS, Linux, and Windows. The short version is that everything which reads or writes nodes got faster, the store got about 4x smaller, and we saw an amazing improvement on a cold cache.

Why the node store was the bottleneck

A Merkle node in Oxen is small. It holds a hash, a little metadata, and pointers to its children. The old backend gave each one its own files, two per node, laid out in a directory structure derived from the hash.

This design is nice and simple. It is trivial to debug, since you can cat any node, it needs no dependencies and it works the same everywhere.

It also means that a repository’s tree is spread across tens of thousands of tiny files. Our benchmark repo has 5,220 directories and 52,200 files, which comes to 10,443 Merkle nodes, which comes to 20,886 files in the node store. Walking that tree means the operating system opens, reads, and closes 20,886 files, most of them a few hundred bytes.

Filesystems are not good at reading lots of small dispersed files, each of those files take an inode lookup, a read syscall, and a full block of disk regardless of how little is in it. On Windows, with Defender’s real-time protection inspecting each open, the cost per file is even higher.

Writing is worse than reading. Each node opens a node file and a children file, writes both, then flushes and fsyncs each one before moving on. For 10,443 nodes that is 20,886 fsync calls, and fsync is one operation a faster CPU cannot help with, because it blocks until the storage device confirms the data is durable.

What LMDB does differently

LMDB is a single B+tree living in a memory-mapped file. We use it through the heed crate, with one LMDB database holding all the merkle tree nodes. Keys are fixed 16-byte integers rather than paths, so finding a node is a handful of integer comparisons down a tree rather than resolving a path through several directory levels.

Our LMDB implementation pays off in three significant ways:

Reads are page faults, not syscalls. The database file is mapped into our process’s address space, which means the operating system presents the file as if it were memory. Reading a node becomes touching an address, and if the kernel already has that page resident, nothing happens at all beyond a pointer dereference. There is no open and no read per node. It also means the page cache is the database’s cache.

Writes batch into one transaction. Against the file backend’s two fsyncs per node, our LMDB writer opens a single write transaction, puts every node and every link into it, and commits once. That is one durability barrier for the whole batch, whether it holds ten nodes or ten thousand.

Locality. Twenty thousand small files land wherever the filesystem chooses to put them, so walking them is close to random I/O, and the kernel’s readahead cannot predict what comes next. A B+tree in one file is laid out contiguously, so reading one part of a subtree tends to pull in the neighbours you are about to ask for. This is the mechanism behind the cold-cache numbers below, and it is why reads speedup while writes do not.

What got faster

All numbers come from Oxen v0.52.9, measured on 2026-08-06. All write numbers are the median of 3 runs. All read numbers are the median of 5 runs.

Here is the warm-cache speedup, which is the common case once you have run a command or two.

Browsing a repository’s tree is between 2.0x and 5.7x faster depending on the platform. Committing is between 1.4x and 3.0x faster. oxen status is consistently around 1.5x faster across all three.

The `ls` command we used only traversed a single-digit number of nodes, which ended up not being enough to make a measurable difference. `add`, `log` and `init` were included as control items, as they did touch the merkle tree at all.

macOS ran on an M5 Pro, Linux on a Graviton m7g.2xlarge, Windows on an m7i.2xlarge with NTFS. The architectures differ, so absolute times are not comparable between columns. Every ratio is computed within a single platform, between two arms that ran on the same machine minutes apart.

The first command of the day

Where this really gets interesting is when your disk cache is cold. The first Oxen command you run after booting up your laptop, when nothing is in the page cache, shows some dramatic results.

Measuring that fairly is fiddly. We drop the page cache immediately before every timed operation and treat both arms identically, alternating which arm goes first. On Linux that is sync followed by drop_caches, on macOS it is purge. Windows has no equivalent, so each cold sample required a reboot.

A cold `oxen tree` on Windows was over 41 times faster!

The last row, commit, shows 1.68x cold against 1.64x warm, which is essentially no amplification at all. Commit is dominated by writing, so the cache state of the store barely matters.

Migrating your oxen repository

Migrating an existing repository is a single command. On this repository’s 10,443 nodes it took 2.01 s on Linux, 7.97 s on macOS, and 28.97 s on Windows.

Migrating your repository is definitely worth it. The node store is about 4x smaller and lives in 1 file instead of tens of thousands. On our test repository we saw that browsing the tree was 2x to 5.7x faster with a warm cache, and 41.5x faster with a cold cache after a reboot. Committing a large tree ran 1.4x to 3x faster, oxen status was 1.5x faster, and pulls were between 1.5x-2.2x faster.

How much of this you feel depends on the shape of your repository, since a Merkle tree is sized more by directory count than file count. A repository that has only a few directories will only see modest gains.

The LMDB backend is in Oxen v0.52.9 or later. Upgrade your existing repository with:

oxen migrate up merkle_nodes_to_lmdb

Try it

If you are looking for a version control system that handles huge files and extensive directory structures, that’s where Oxen shines. Come give Oxen a try!

Come find us in Discord, or check out the project on GitHub. We are also hiring.