companion-forge

On an iPhone or iPad, Apple's voice processing is on, which cancels the voice's echo. It can only subtract what it plays itself, so nothing is played any other way. On a Mac it is not used. On the M1, on 2026-10-20, its microphone side delivered nothing at all, whatever devices it was pointed at, so the microphone is taken as it is. The checks below keep the voice out, and the first words of a turn are also checked against the end of the last reply.

Some echo always gets through, so talking over a reply has to pass two checks. It has to sound like confident speech for 0.35 seconds. Then it is compared two ways: by its words, because the reply's own voice coming back says the words being said, and by its level. The companion knows what it played, learns how loud that comes back in this room, and a person talking over it is louder than twice that. Either check can call it echo. Only something that passes both stops the reply, and then only the part actually said is kept.

Where the voice sits

With the spatial voice on, which it is in Settings by default, the voice is placed half a metre in front of you and a little below, where a phone or tablet held at the chest is. On AirPods that track the head, on iOS 18 and macOS 15 and later, it stays there when you turn your head. On speakers it sounds as before. If that path will not start on a device, it falls back to the plain voice and says so. It has not yet been heard on AirPods.

The face looks straight at them while it listens, mostly at them while it answers, and only glances while it thinks. Its head turns a little after its eyes. The front camera does this if you turn on “Look at whoever is talking” in Settings, which is off by default. Vision finds the faces and their lips about ten times a second. The one whose mouth keeps opening and shutting is talking, and with nobody talking it looks at the nearest face. Each picture is looked at and let go. On a Mac it uses the built-in camera, never a virtual one. This works with the echo canceller on.

On an iPhone or iPad with the echo canceller off, and headphones, the level difference between the front microphones' two sides says left or right. The canceller gives one channel, so out loud this is the camera's job. A Mac's microphone is one channel. ccrun --from-right speaks from one side of a stereo microphone and then the other: the face turned to 0.84 across, then to −0.84. Neither the camera nor the phone's stereo microphones have been tried on a device yet. If the face turns the wrong way, the picture or the microphone is mirrored.

The face hangs behind the glass on a spring. Move the phone and it lags and swings back; turn it and the face tilts after it. On a Mac, which has no motion sensors, dragging the window does the same. Shake it hard for a second and its eyes go wide and it says so (“Woah there! This ain't a rodeo. Slow down, please.”, and four others in turn), at most once every 25 seconds, and never over somebody talking. Walking measures about 0.05 on the shake meter, a shake 2 and up, and it speaks at 1.2. With nothing else going on, its eyes look around: a quick dart, a hold of one to five seconds, often back to the middle. It looks at you while it listens and up to one side while it thinks.

Conversations are kept in full, one JSON line per turn, and nothing is decided there about what matters. Notes are short sentences about the person, written by the language model once a conversation has been quiet for 45 seconds, or when Talk is stopped. Each note names the conversation and the turns it came from. Small talk, greetings and general questions are left out. A note that says the same thing as an older one replaces it.

Notes about who you are, your name, your work, the people and pets in your life, go into every reply. A 4B model asked to mark these seldom does, so the companion recognises them too. Before each reply, what you just said is compared with the notes by meaning, using bge-small-en-v1.5 on MLX, 134 MB, downloaded on first launch. Apple's built-in sentence embedding ranked too poorly to use. A note must score 0.45 and be within 0.12 of the best match to be brought up. What comes to mind goes in just before your words, so the rest of the conversation stays in the model's cache. When you ask about the past, the original turns behind the best note are read back word for word.

You can also ask out loud. “Remember that my sister's birthday is on the third of March” is noted at once, in your own words. “Forget what I said about my sister's birthday” finds the closest note and deletes it, and it says which it did. Both happen only once your turn is over, so a half-heard sentence changes nothing, and “remember when” is a question, not a command. Writing notes gives way to talking: it stops the moment you speak and picks up where it left off. Settings shows everything it remembers, with a button to forget any of it, or all of it and the logs.

ccrun --memory   Gemma 4 E4B
  two earlier conversations, 16 turns
  7 notes in 9.6 s, three of them lasting
  recalling a note: 7 to 25 ms

  “Do you remember what I told you about my sister?”
  brought up the note and the turns behind it
  “You mentioned your sister Ana is visiting.”

  “What's the weather like?”
  brought up nothing

A diary, a calendar, and catching up

Under the notes are two layers the person never sets up. Once a conversation is written up, the companion writes a few lines about it. Once a week is over, those entries are summed up in a few more. Anything said with a date on it goes on a calendar, each entry pointing at the note and the diary line it came from. Dates are read as people say them: tonight, on Friday, this weekend, the 3rd, 12 March, in two weeks, counted from the day they were said. Birthdays and anniversaries come round every year. Something on today, asked before evening, or tomorrow, is brought up in passing, at most once every five days.

A note about something with a time on it becomes worth asking about once it has happened or had a few days to go somewhere: an evening thing the next morning, a weekend on the Monday, something under way after a few days. From the fourth turn, at a calm moment, it asks how it went. What they say is written up like anything else. Something they told it after the fact is not asked about again. Opening the app after two hours or more, it says hello: a few words that fit the time of day and how long it has been, with anything to catch up on in the same breath. The first time, it says it is glad to meet them and asks their name. Back within two hours, it says nothing.

Sealed on this device

What it remembers, the logs and the notes, the diary and the calendar, is sealed with AES-GCM under a key made on first use and kept in the Keychain on this device only. The key is never synced, and never put in a backup that could be restored elsewhere. On an iPhone or iPad the folder is also unreadable while the device is locked. A Mac relies on the sealing, and on FileVault. Files written before sealing still open, and are sealed the next time they are written. If the Keychain will not give the key, it says so and runs unsealed. It never makes a new key over an old one, which would lock away everything sealed with it.

It keeps that voice as the average of CAM++ voiceprints, through FluidAudio, of what they have said, steadier the more it hears. Anyone else is a guest, met kindly, told nothing it knows about its person, and kept out of its memory. Each sees only their own side of the conversation. Told everything, it once told a guest about its person's nerves over a visit mentioned minutes before.

ccrun --speakers   thirteen system voices, eight clips each
  against an averaged print at 0.80
  every clip over 1.5 s went to the right voice
  none went to a wrong one
  short turns (“yeah”, “hm”) decide nothing

Asked how to say breakfast in Romanian, what “köszönöm” means, or “and in Polish?”, it looks the word up before the model answers, in an offline dictionary built from English Wiktionary, about 130 MB: translations between English and fifteen languages, and what English words mean. None of the three models knew “micul dejun”. Told nothing is there, it says it does not know rather than guess.

A sentence can mix languages. A quoted word in the language they named, or one Apple's detector is sure of, and a whole sentence in another language, are each spoken in their own language by the polyglot voice and joined. English is that same voice. The eight languages the voice was trained on sound like it, and the rest go through plain espeak-ng, accent and all.

Who it is

Three personalities, chosen in Settings. Playful is the default: jokes, wordplay, plays along. Gentle is warm and calm. Deadpan is dry wit, delivered straight. Playful and deadpan drop the act when someone is having a hard time. Small models follow examples far better than descriptions, so each comes with a few lines in its voice. Told only the rules, it was correct and flat.

Footprint is what iOS holds against an app, and on Apple silicon it includes what MLX keeps on the GPU. The run is ccrun on 2026-10-20, with recognition, the embedder, the voice and memory all loaded. The same day, with Gemma 4 E4B, the reply started 0.9 to 1.8 seconds after you stop talking.

Most of that time is the learned wait itself. A reply is drafted halfway into the pause, transcribed in 0.14 seconds, its first words written 0.3 seconds later and its first sentence voiced by about 0.8 seconds, so it is ready when the wait ends. The rest is a long first sentence.

With the reply's echo leaking back at 0.05, roughly what voice processing leaves, echo measured 1.0 times its learned level and an interruption 11 to 18 times, so none was stopped falsely and the interruption was caught on its first check. At 0.3, barely cancelled, 17 echo fragments were let go and the interruption was caught on its third check.

The model is shown the last 16 to 23 turns. The window moves 8 turns at a time rather than one, and after each reply the model reads the next prompt in while you talk. Sliding a turn at a time made every reply past the sixteenth re-read about 230 tokens, and its first words came 1.86 seconds after you finished instead of 0.31. With both changes, every turn of a 22-turn conversation started in 0.31 to 0.37 seconds.

It has run on an M1 Mac, an iPhone 17 and an iPad with M4. The numbers above are the Mac's. Memory and lag on the phone and the tablet are still to be measured the same way.

The spatial voice has not been heard on AirPods. The camera, and the phone's stereo microphones, have not been tried on a device. Desktop beyond the Mac, and Android, are not this app: the README's next step is llama.cpp and sherpa-onnx, as interpreter-forge's Android port already runs Silero, Parakeet and Piper.

The sad side of the visor is unused on purpose. If the Keychain will not give the key, it says so and runs unsealed rather than inventing a new key. Without the voiceprint model, everyone is treated as its person.