Protoke

Twenty-eight seconds

Everything moving here was written into a subtitle file

The voice is synthetic, from a locally trained Piper model. The background is one black PNG that never changes. Every eye, every mouth shape and every word of text is a single libass pass over that still — no frames were animated, and nothing was composited.

27.5 seconds, 1920×1080, 1.7 MB. The mouth follows the words from their spelling; the eyes blink on their own schedule, roughly once every seven seconds. There is no beat in this recording, so the head breathes rather than nods — the tool measured that and said so in its log rather than inventing a tempo.

Why it works this way

A face is cheaper as geometry than as pictures

The obvious way to put an animated face on a video is to render frames and composite them. That means an image sequence, an alpha channel, a second encoder pass, and a pile of intermediate files that all have to agree about size and frame rate.

ASS subtitles can draw filled vector shapes. libass is already in the ffmpeg command drawing the lyrics. So the face is emitted as drawing commands into a second subtitle track and burned in by the pass that was going to run anyway. One ffmpeg invocation, no intermediates, and the "animation" is a text file you can open and read.

The cost is that everything has to be expressible as outlines and timings, which turned out to be a useful constraint rather than a limit.

The face

Eleven expressions, no sprite sheet

Early versions swapped between fixed shapes and looked exactly like that: rigid, held together with tape. The fix was to stop picking expressions and start blending dial values, the way a protogen firmware blends morph weights.

Six numbers describe an eye — openness, bow, slant, hook, roundness, weight. Six more describe the mouth. Two describe the whole face. Every named expression is just a point in that space, and anything between two of them is a valid face, so a blink is a continuous move rather than a cut.

These are the renderer's own frames. The geometry was exported straight out of the Python that writes the subtitle track, so the page cannot drift from the videos — it has no drawing code of its own, only a replay loop. What you are watching is the idle state: breathing, because there is no beat.

Each of these is a set of dial values, not a drawing. sharp is the clearest case: anger reads as straight edges and stillness, so it raises the sharpness dial — which bends the curves toward their control polygons — and damps the idle motion, rather than swapping in an angry picture.

The app

A local page, not a cloud service

The interface is a token-protected page on 127.0.0.1. With pywebview installed it opens in the operating system's own webview — a couple of megabytes, rather than the couple of hundred an embedded browser costs — and in your default browser otherwise.

The preview is a real layout at output resolution, with the title-safe area drawn as a guide. Changing a colour, a typeface or the line count redraws it immediately; nothing here needs a render to check.

All fourteen dials, with the preview above them rendered by the same code that renders the video — so the shape on screen is the shape you will get, not an approximation drawn a second time in JavaScript. Saved faces become presets you can pick per track.

Transcription and separation live behind a probe: the app asks each engine whether it can actually run here and reports what it finds, rather than assuming. Vocal separation stays off by default because it pulls PyTorch — about a gigabyte — for an accuracy gain most tracks do not need.

Get it

Python, ffmpeg, and nothing else that is mandatory

The rendering, alignment, face and subtitle code imports only the standard library. Every dependency belongs to an optional transcription or separation engine, so with a timed JSON file beside your track the tool runs with nothing installed at all.

git clone https://github.com/snepssen/protoke.git
cd protoke
./setup.sh          # creates .venv, installs Parakeet
./start.sh

git clone https://github.com/snepssen/protoke.git
cd protoke
setup.bat
start.bat

ffmpeg must be built with the subtitles (libass) filter — brew install ffmpeg, winget install Gyan.FFmpeg, or apt install ffmpeg all provide one.

Replacing MacWhisper was the point of the rewrite: it is macOS-only and paid, and the tool used to need it. Parakeet TDT through onnx-asr does the same job free on every desktop platform.

If something looks wrong

Say so

Bug reports and shape critiques are equally welcome — the face got most of its improvements from someone describing exactly what looked off about it.