Interpreter

The problem

Phrase by phrase is how a walkie-talkie works

Translation apps wait for you to stop talking, or translate every half-sentence as it comes. The first is a conversation in turns; the second is a stream of fragments with the verb still to come, which in German or Hungarian is often the part that mattered.

A human interpreter does neither. They let the speaker run ahead, hold on until a thought is complete, then say it — a few seconds behind, all the way through. That is the behaviour this copies: listen without stopping, cut at the end of a thought, translate it with what came before as context, and speak it while the next one is already being heard.

It is built for one table in particular: Polish, Hungarian and Romanian around an English speaker, with Dutch words dropped in. English is understood already, so it is shown, kept as context and never spoken back.

How it works

Deciding when a thought is complete

mic ─▶ voice activity ─▶ segmenter ─▶ translate ─▶ voice ─▶ headphones
                          ▲      │
                          └ ASR ◀┘  re-reads what is not yet cut, every 0.5 s

Every half second the uncommitted tail of speech is transcribed again, and words count as settled once two passes agree on them. A piece is cut at the end of a sentence that speech has carried on past, at a pause long enough for this speaker, or at a comma once it has run four seconds. At five it is cut wherever it can be, and at six wherever it stands; the next piece picks up the sentence.

A general model such as Qwen is also asked, in a single forward pass, whether the words so far can be translated yet. That is why it speaks sooner in quick conversation: it can cut at “Nie wiem, czy zdążymy na pociąg.” and hold at “Nie wiem, czy.” TranslateGemma cannot be asked, so it relies on the rules alone, and translates better.

When it falls behind

Speech does not wait for the interpreter, and people speak unevenly: they pause for effect, think aloud, build up slowly. That is meant to queue, not be cut; the pauses are context for the next piece. Pieces that pile up while it is busy go out together as one translation, and once a second of speech is queued the voice speeds up, to at most 1.3 times. Everything is translated in full.

Some listeners would rather keep up than hear all of a long story. Summarise to keep up, off unless chosen, says only the gist once the interpreter is behind: “complaining about back pain and their boss.” It needs a translator that takes instructions, Qwen or Gemma 4.

What it does

A halting German announcement, as it is heard

[ 0.00– 3.52] clause   │ Also, ich wollte eigentlich nur kurz sagen,
  #0  lag 2.7 s → Also, I just wanted to say briefly…
[ 3.90– 6.70] clause   │ dass wir morgen nicht um neun,
  #1  lag 4.6 s → that we won’t be meeting at nine o’clock.
[ 6.92– 9.80] sentence │ sondern erst um halb elf anfangen.
  #2  lag 4.2 s → but to start at 10:30.
[10.10–13.14] force    │ Der Grund ist, dass der Zug aus München
[13.52–15.92] pause    │ wegen Bauarbeiten ausfällt.
  #3  lag 5.2 s → The reason is that the train from Munich…
  #4  lag 5.0 s → …is cancelled due to construction work.
[17.98–21.74] clause   │ Wer kann, sollte vielleicht mit dem Auto kommen,
  #5  lag 4.6 s → Those who can should perhaps come by car.

lag  median 5.0 s, max 6.6 s over 8 pieces

“halb elf” is 10:30, not half past eleven. A 4B model reads European clock times the English way and got 3 of 10 right, so German and Dutch half hours and quarters are rewritten to digits before the model sees them. On held-out times in seven languages: 14 of 14.

[ 2.30– 5.90] clause   │ No, scusate. Il treno parte alle 7 e un quarto,
  #1  → No, sorry. The train leaves at 7:15.
[ 8.26–12.26] flush    │ Quindi ci vediamo in stazione verso le 7-5.
  #3  unclear, not spoken
      The clock time was unclear in speech recognition. Please repeat it.

Real conversation

Tuned on seventy minutes of a real kitchen table

Samples read clearly are the easy case. The segmenter was tuned on a 70-minute recording of English, Polish and Hungarian, with Dutch words in it, from an iPhone's bottom microphone. What that changed:

• The median delay of spoken pieces went from 7.8 to 3.9 seconds, the worst from 24.9 to 12.8.
• Pieces cut by force fell from 75 to 11.
• 916 of 1,022 pieces were English: shown, used as context, and left quiet.
• Accented English the recogniser mangles, which a model then “translates” into itself, is caught and not spoken: 40 of 101 would-be translations in the recording were that.
• A pause is measured from the last word heard, not only from the voice detector, which went on hearing “speech” in background noise for half a minute.

The voice

One voice, eight languages, 64 MB

The interpreter speaks with snepssen: one small Piper voice that speaks English, Polish, Hungarian, German, Dutch, French, Spanish and Italian as the same character, whispery, tired and steady, rendering a sentence in milliseconds on the device.

It began as an English voice. Its other languages were taught from speech generated with the same vocal profile, read from scripts written for each language's hard sounds, cut, checked line by line and trained together. Sounds the phonemizer gets wrong, Polish hard dz, Hungarian silent h, or shares across languages that say them differently, German, Dutch and Italian r, are corrected before the voice sees them, by voice-polyglot's Polyphon; the apps carry a port of it held to the original symbol for symbol. Each language plays at its own pace, picked by ear, since the new languages were read faster than the English.

It is still training. Native listeners understood its Polish and Hungarian; German and Dutch sound native enough to be rushed. A better-trained voice is a rebuild away.

Snepssen interpreting snepssen

The voice also makes the test speech. Each language is spoken by snepssen, heard by the app's speech recognition and interpreted into another, which tests the whole chain and how legible the voice is at once:

TranslateGemma, in real time on an M1 Mac, a halting announcement or a fluent one per language. The misses that mattered were clock times: fél tizenegykor heard wrong became the wrong time in German. That is the next thing to make safe.

Models

Choose the translator; it downloads once

Models are picked in the app on first launch, and can be added, switched or deleted in Settings. One translator is in memory at a time; on an 8 GB iPhone 17 a 4B model fits beside recognition with the increased-memory entitlement. Measured on the same 26 lines in ten languages and a 17-case test of complete thoughts:

Into English, TranslateGemma's is the most natural; into Polish and Hungarian the comparison is still to be made.

On the iPhone and the Mac

One engine, two apps

Getting it

Built from source, for now

There is no App Store build. The apps need an Apple silicon Mac with macOS 14, or an iPhone with 8 GB of memory, and Xcode to build them. espeak-ng's data comes from voice-forge, cloned beside this repository, and the voice from voice-polyglot. Everything to build them is here: the app's source and build scripts, and in voice-polyglot the whole pipeline the voice was made with, from the scripts it read to training and export. The voice itself is fetched by the build from voice-polyglot's releases, the newest one each time: still in training, marked with an asterisk and the epoch it was exported at.

# beside each other
git clone https://github.com/snepssen/voice-forge
git clone https://github.com/snepssen/translate-core
cd translate-core

apple/build.sh             # the Mac app: apple/Interpreter.app
apple/iOS/build-ios.sh     # the iPhone app, onto a connected iPhone
uv run translate-core live # the Python rig the apps were measured against

The iPhone build signs with your own development team, set in apple/iOS/project.yml. The first launch downloads speech recognition and the translator you choose — about 3 GB, best on Wi-Fi.

Where it is going

Roadmap

The goal does not change: be included in a conversation you would otherwise sit out, in your own language, with nothing leaving your device.

Honesty

What it is not

It is not a certified interpreter, and should not be relied on where a mistranslation costs more than a repeated question: a doctor, a contract, a police station. A 4B model gets most things right and some things confidently wrong, and short or garbled phrases are where it goes wrong most.

Recognition can invent words from faint speech or noise, and it covers 25 European languages; Norwegian and the rest need Whisper, which is about ten times slower. It speaks eight languages well; others only through a setting, with an accent. Into languages other than English the translators are less tested, and Qwen's Polish is rough. On the iPhone it works only while the app is open.

It is still being tested in the field. The numbers above are from recordings and one kitchen table, not from many people in many rooms.

Get in touch

If something looks wrong, say so

There is no analytics here and no crash reporter, so a failure has no way of reaching me on its own.