Koe: the browser subtitle extension that didn’t work out

↗

Koe is a browser subtitle extension I made, and a spin-off from mimi. mimi captions audio as it plays on the computer. With Koe, I wanted to try a different route: how far could general-purpose offline video captions go, and how good could the experience become?

For recorded video, I could accept waiting a few seconds if recognition and translation got more context and, hopefully, produced a better result. A livestream can’t wait that way, but recorded video has upcoming audio available to process. That was the idea I wanted to try; offline processing isn’t inherently more accurate.

I also tried bringing mimi-style live captions into the browser and ran into plenty of problems. I no longer remember the specific old issues. Koe didn’t become an extension I could simply hand to someone else either, and installation remained complicated.

Should an hour-long video be processed before it starts?

Long videos were the first problem. Sending the entire recording to speech recognition and waiting for all the captions before playback wouldn’t work. A few minutes of waiting is already too much for people. What about an hour-long video? They might never watch the second half. Does all of it need to be transcribed and translated first?

So I started processing it in chunks: work near the playback position first, then prepare the upcoming parts, rather than computing the whole video immediately. Local processing uses the computer’s own resources and time. Calling a cloud service can add service charges too.

Processing ahead requires access to the upcoming audio. Koe’s direct-reading path currently handles only certain HLS videos on demand, where a playlist points to media segments. The source must be public and unencrypted, with supported segment formats. Websites don’t all deliver video this way.

Ordinary MP4 and DASH sources aren’t handled by this direct-reading path. Not using HLS doesn’t make their audio impossible to obtain; this implementation simply doesn’t handle them in a general way. When the browser allows it, Koe can capture the tab’s playing audio and transcribe it locally. That returns to processing sound as it plays, without access to dialogue ahead of playback.

For sites whose audio was accessible, how should it be split so the wait wouldn’t feel too long? Every five seconds, or ten? Shorter chunks might produce captions sooner, but would they lose context? Longer ones meant waiting more. I kept weighing these tradeoffs at the time.

Chunking also needs to handle sentences crossing a boundary. The current code overlaps neighboring chunks to retain context, then merges the results to avoid duplicate captions.

↗

The numbers represent pieces of audio; both chunks include 3. This illustrates the chunking approach, not a real transcription.

People also seek through videos. The chunks prepared for the old position are no longer useful; processing has to restart near the new one. The current implementation cancels the old task, clears the old caption state, and starts reading ahead from the new playback position. Late results also need checking against the current session. The whole process had become very complicated.

Timestamps must refer to the whole video too. A line three seconds into a clip starting at 10:00 belongs at 10:03. The repository contains implementations for isolating old session results (opens in a new tab) and handling translation backlogs (opens in a new tab).

A page can contain more than one video

A webpage can have the main video, an advertisement, and previews. A general-purpose extension also needs to work out which one the viewer is watching. Finding a <video> element alone isn’t enough.

Koe already selects a video using playback state, whether it has sound, and picture size. It also lowers the priority of elements with advertisement-like class names. These are guesses about which one is the main video, and they can be wrong. Letting the viewer correct the selection is something to consider next; that control isn’t implemented yet.

Browsers already have a standard for displaying captions. A <track> (opens in a new tab) can attach timed subtitles to a video and display them at its playback position. The harder questions come before that: generating captions when none exist, obtaining the corresponding audio, and working out which video they belong to.

Chrome has Live Caption and Live Translate too (opens in a new tab). Live Caption processes audio and captions locally; Live Translate sends captions to Google for translation. Calling both together offline translation would be wrong. These are also separate from translating text already on a webpage.

Installing it was another problem

Koe’s companion Helper runs Whisper on the Mac, with an initial model download of about 626 MB. Local Chinese translation also needs macOS 26 or later, Apple’s On-Device translation enabled in system settings, and the relevant supported language packs downloaded. The compatibility build for macOS 15–25 can only transcribe locally.

The model and language packs need downloading first, while video audio still comes from the original website. Offline here means keeping recognition and translation on the Mac, rather than opening every online video without a connection. Koe also has a DashScope cloud mode that sends audio to the provider.

I mainly tested with ego-lite at the time and hard-coded quite a few things. For example, the installer fixes ego-lite’s browser identity, and automatic loading is built around it. Chrome still needs manual extension loading. The installation route also requires an Apple silicon Mac.

After downloading it, I was stopped at the door by Apple’s security checks too 😭. The existing Helper binaries are only ad-hoc signed, without Developer ID signing or Apple notarization.

The Koe website (opens in a new tab) does have a cute girl, though.

In the end, Koe achieved nothing except a cute girl… 😭 Download it, and it’s still a pile of shit.

The code is in the Koe repository (opens in a new tab). If it interests you, you could try making one too, haha.