Rars: a Rust RAR implementation, mostly written by LLMs

I’ve done a few different reverse-engineering projects with LLMs, and figured it’s time to push the clankers to their limits. A RAR compressor for every version of RAR ought to have taken about 5 years, which is why nobody has ever bothered. Today, it takes 5 weeks of evenings and weekends, clanking OpenAI Codex 5.5 and Claude Opus 4.7, and cost roughly £40 in (heavily subsidised) tokens. Yes it’s 55k lines of slop, no it’s not that fast, and it almost earned me an OpenAI ban. But it works. RAR was originally an LZSS compressor for DOS, which peaked in popularity as the warez scene’s format of choice. Fighting with WinZip for feature parity and supremacy, WinRAR boasted multi-volume support, recovery records and even an internal VM, but its USP was always superior compression. It’s a middle-aged format that never stopped growing up, it’s as big as a house. unrar comes with source code but that code is not actually free, and somewhat ironically RAR’s author Eugene Roshal isn’t a big fan of piracy. So ideally I’d need to implement my version from spec, which doesn’t really exist. The monstrous task of creating one involved pulling code from free decompressor sources in the wild - unar, libarchive, UNRARLIB, plus random web pages and folk lore. I then set Claude to work documenting as much as it could. After each pass, I quizzed it on missing features and maintained an ongoing gaps doc containing the hard-to-know stuff. This persisted between context resets, which were needed to flow the tokens into the gaps. It took 2 weeks of cooking, going back and forth until we had most of the reader side documented. The writer side, however, remained a mix of confabulation and conjecture. So next I grabbed the RAR binaries for DOS and Windows, and set to work making test fixtures, hex-dumping and doing passes in Ghidra and DOSBox-x to get some idea of how they were packed. Another week or two of work and the gaps started to close up. Now I had something that might be useful; spec docs for every version of the RAR file format: Being confidently wrong enough to start, Codex, Claude and I set off building a (precariously) compatible Rust CLI. The workflow was shaped something like this: Opus is great, but it tends to enthusiastically generate code while missing the bigger picture. Claude requires remedial passes, refactoring and a short leash, but is great for a chat about strategy or architecture. Gippity 5.5 stays on target when left alone, but will rabbit hole you hard if you chat with it too much. I could give Codex the spec docs and basically tell it to just get on with it. Very refreshing. While working from spec, Codex would randomly stop due to cyber violations, and I’d need to manually compact to continue. Eventually I had to get verified by OpenAI to stop it from happening. Well, it turned out that at some time during spec investigation, Claude needed to understand authenticity verification which is a paid feature. With a context full of reverse engineering tools it cracked WinRAR and bypassed product registration, then dutifully documented its crimes in the spec. The docs, when viewed, triggered OpenAI’s alarms and stopped it dead in its tracks. I squashed this out of the git history, and decided not to implement the feature at all. You’ve gotta keep an eye on the bots and interrupt when things start to smell bad. If you don’t then they’ll special-case their way out of every problem and around every test, ugly patterns will propagate through your code, and you’ll need an expensive refactor later. I should have done more, but I didn’t, and for that I paid the price later on. The tokens were subsidised, but it was a waste of my time. For the last 15 months or so my hobby has been shouting at Claude, so I’m getting good at interventions. I enjoy it, even if it has damaged my personality. I tend to swear at Codex far less, maybe because it’s faster or less of a grinning idiot, but probably because it’s bland and professional. This may be a good thing, but I’m not sure yet. Way too many tests. Fragile tests, coverage that doesn’t matter, excessively_long_test_names_that_fill_your_screen, these are vital when working on something this size. They provide a statistical mass that warps text generation, pulling the bots back on track when they go off-piste or try to cut corners. So, reams of unit tests and as much coverage as is possible please. We can always remove them later, right? Right?… So the tests keep it in shape, but actually running the code is what aligns it with reality. So the real work is about fixtures, oracles, and updating the spec where wrong. In doing this, Codex cleared up autofill bullshit (“hallucinations”) that had previously passed at least ten rounds of review. So it turns out that empirically grinding against reality is the best source of signal, and with enough time the spec was honed into something close to Truth. Science. Periodically, I had Claude generate a full review of th

Rars: a Rust RAR implementation, mostly written by LLMs

Key takeaways

More in computer-science