Why removing 'um' from a recording is harder than it sounds

Linguists have a word for the ums, uhs, ers, and elongated versions (ummmm, uhhhhh) that pad spoken English: disfluencies. I don t record a lot of voice audio, but a few friends do, and they tell me editing those out by hand is miserable. So I built erm to do it. uvx erm input.wav That s the whole interface for the common case. It writes a cleaned .wav and a JSON cut list next to the input. This post walks through how it works, because the obvious approach doesn t sound very good and most of the code is the stuff that fixes that. You d expect the job to be: transcribe with word-level timestamps, find tokens like um and uh, cut those ranges with ffmpeg. That gets you maybe 60% of the way, and the result sounds worse than the original. Three reasons: Most of erm is the work of fixing those three things. Whisper is OpenAI s open-source speech-to-text model. You hand it audio, it hands you back a transcript, and with the right flag it ll also tell you the start and end timestamp of every word. It runs locally, which is what makes a tool like this possible without sending your recordings anywhere. erm uses faster-whisper , a reimplementation that s several times faster than the reference one and uses less memory. Same model weights, same output, just a better runtime. The default is the medium.en model, which is a good speed/accuracy balance. You can override with --model if you want small.en (faster), but I d actually reach for large-v3. It s noticeably better at picking up fillers and worth the extra compute. First, run Whisper. erm asks for word-level timestamps and gives it a small instruction up front telling it not to clean up the transcript. Whisper, left alone, will edit out fillers because most of its training transcripts are clean prose. Any word that comes back as a known filler (um, uh, er, etc.) is flagged for cutting. Elongated versions like ummmm get matched against the um stem on the fly. Whisper still misses things, so three more passes look at the audio directly: Gap fillers. If there s an unusually long pause between two transcribed words (more than 350ms by default), erm checks whether somebody is actually making a sound during that pause. If a chunk of voice is sitting inside what Whisper marked as silence, that s a filler Whisper deleted entirely. It really does just drop them. No token at all, just a hole in the transcript where an um used to be. Fillers hiding inside a word. Whisper sometimes glues a filler onto an adjacent word, so "in, uhhhhh" comes back as a single in token. erm looks at long single-token words, splits them at brief dips in the audio, figures out which chunk is the actual word (based on how long that word should reasonably take to say), and treats the rest as filler. Words that are much too long. If a word lasts way longer than its text could plausibly take to pronounce, the tail end is suspicious. erm scans the tail for voiced sound, and optionally double-checks with a pitch test: does the suspicious chunk sound like someone holding a vowel (uhhhhh), or like someone just speaking slowly? A held vowel has a steady, simple acoustic shape; real speech is constantly changing as you move between sounds. The pitch test keeps the tool from trimming slow talkers. All four passes (the Whisper one and the three audio ones) produce candidate cuts independently, and the lists get merged before the next step. A cut at exactly t = 1.234s lands wherever the waveform happens to be at that instant, almost never at zero. Stitching two arbitrary points together leaves a step in the waveform, and that step is the click you hear. Two small fixes, in order. First, each cut endpoint is allowed to slide a tiny bit (up to 60ms) to land in the quietest spot nearby. If there s a momentary lull in the audio just before or after the original cut point, slide there. The slide is bounded so it can t cross into a neighboring word, otherwise you d chew off real speech. Second, from that quiet spot, the endpoint snaps to the nearest moment when the waveform is exactly crossing zero. Two zero points stitched together produce a continuous waveform with no step, and no click. After all that, very short surviving fragments get cleaned up: if two adjacent cuts would leave a sliver of audio shorter than about 120ms between them, the sliver gets merged into one bigger cut. A fragment that small can t survive the smoothing on either side anyway and just sounds like a blip. ffmpeg does the actual stitching using a crossfade. Instead of butting the two pieces of audio together, it briefly overlaps them and fades one out as the other fades in. That smooths over any remaining mismatch. The trick is picking how long to overlap. A fixed length (most tutorials say 80ms or so) sounds wrong both ways: short cuts get smeared together, long cuts still pop. erm scales the length to the size of the cut: a tiny clip of uh gets a short crossfade, a long ummmmm gets a longer one. There s a floor and ceiling

Why removing 'um' from a recording is harder than it sounds

Key takeaways

More in computer-science