WebCodecs vs ffmpeg.wasm for in-browser video

The two options, briefly

ffmpeg.wasm is ffmpeg compiled to WebAssembly. It runs the entire, mature ffmpeg pipeline inside the browser tab, in software. It handles an enormous range of formats and codecs, and its encoders are the product of decades of tuning.

WebCodecs is a browser API that exposes the platform's own media encoders and decoders, including the hardware encoder on the device. You feed it frames and it hands back encoded packets, using the same silicon your phone uses to record video in the first place.

Why we chose WebCodecs

Two reasons, and they compound.

Speed. ffmpeg.wasm runs in software, and in the browser it is effectively single-threaded for most workloads. On large files it crawls; we watched several ffmpeg.wasm-based tools stall partway through big videos. WebCodecs hands the work to the hardware encoder, which is dramatically faster because it is the purpose-built chip for exactly this job.

No download, no server. A full ffmpeg.wasm build is large, so every user pays a substantial download before anything happens. WebCodecs is already in the browser, so there is nothing to ship. And because the encoding runs on the user's own device, there is no server doing the heavy lifting, which is what lets the file never leave the device and lets the tool stay free with no per-video cost to us.

The tradeoff we made, honestly

WebCodecs can only touch codecs the browser already supports. That is the whole catch. So MP4, MOV, MKV and WebM work well, but AVI, WMV and FLV do not, because their codecs are not part of the browser's built-in set. The irony is not lost on us: those are exactly the formats ffmpeg.wasm would have handled without complaint.

HEVC (H.265) is a similar story. Whether it decodes depends on the browser and operating system, so an HEVC file can work in Safari on a Mac and fail in Firefox on the same machine. When something is not supported, we made a point of saying so clearly rather than failing silently, because a silent dead button is the worst possible outcome.

So we traded format breadth for speed, no download, and no server. For a tool whose entire reason to exist is your file never leaves your device, that was the right trade. If your priority were "compress literally any file anyone can produce," ffmpeg.wasm would be the better call. Different goals, different engine.

The part nobody warns you about: Safari's audio encoder

The WebCodecs path is not free of sharp edges, and the sharpest one cost us about a week. Videos kept coming out silent, but only on iPhone, while the same file kept its audio on desktop Chrome. It is very easy to conclude at that point that iOS simply cannot do this and move on.

It was not an iOS limitation. It was two separate problems stacked on top of each other. First, our own bitrate budgeting was letting the audio bitrate drop low enough that the AAC encoder quietly switched to HE-AAC (mp4a.40.5) instead of plain AAC-LC (mp4a.40.2). HE-AAC uses a lower sample rate and different signaling, and it is exactly the kind of thing that works everywhere except the one place you are testing.

Second, Apple's browser AAC encoder silently fails re-encoding certain streams at 22050 Hz but works fine at 44100 Hz, even though every capability check reports the config as supported. So it advertises support, accepts the job, and then produces no audio. The fix for half of it was, after a week of chasing codec, bitrate, and muxer theories, forcing a single sample rate.

The lesson we would pass on to anyone going down the WebCodecs road: capability detection (isConfigSupported) can return true and the actual encode can still fail at runtime. Build for that. Inspect the real output, not just the API's promises, and have a graceful fallback when the delivered file is not what you asked for.

Would we make the same choice again?

For this product, yes. The speed and the no-server, no-upload guarantee are the whole point, and only WebCodecs delivers them. But it is genuinely a tradeoff, not a free win, and we are still a little unsure the format-support sacrifice is right for every user who shows up with a random file. A future version may add an ffmpeg.wasm fallback for the unsupported formats, so the common case stays fast and private and the rare case at least works. That is the nice thing about the tradeoff being explicit: you can revisit it.