The voice menu bar

AxstreamBar is a native Swift menu bar app: hold ⌃⌥ (control-option), say a command, release. Roughly 200–700ms later the action is running. Everything on the hot path is local — whisper.cpp for ears, an embedding shortlist, the fine-tuned matcher, then the verified replay engine.

hold ⌃⌥ ──▶ mic opens (AVAudioEngine, rebinds to the current input device)
release ──▶ whisper.cpp (large-v3-turbo, Metal)            ~450ms
        ──▶ embedding shortlist (top-10 of the library)     ~20ms
        ──▶ matcher (fine-tuned 350M, schema-constrained)  ~120ms
        ──▶ verified macro replay                          1–3s
             └─ no match? → the axstream engine constructs it
                            with an LLM, executes, and LEARNS it

A bottom-of-screen HUD narrates every state: 🎤 Listening…, the transcript it heard, ▶ workflow, ✓ done · 2.1s — or an honest ✗ with the reason.

Build and run

The app lives in the repo at swift/AxstreamBar (SwiftPM). It links whisper.cpp statically, so build that first:

# 1. whisper.cpp as static libs (Metal shaders embedded)
git clone https://github.com/ggml-org/whisper.cpp ../whisper.cpp
cmake -S ../whisper.cpp -B ../whisper.cpp/build \
  -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=OFF \
  -DWHISPER_BUILD_EXAMPLES=OFF -DGGML_METAL_EMBED_LIBRARY=ON
cmake --build ../whisper.cpp/build -j

# 2. the app
cd swift/AxstreamBar && swift build -c release && cd ../..

# 3. a speech model (best-present wins: turbo > small > base)
mkdir -p ~/.axstream/models
curl -L -o ~/.axstream/models/ggml-large-v3-turbo-q5_0.bin \
  "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo-q5_0.bin"

# 4. launch (also how you relaunch after reboot)
axstream menu

Grant Microphone and Accessibility to AxstreamBar when macOS asks — the hotkey is dead until Accessibility lands (the app polls and enables it the moment you grant, and the HUD tells you what's missing). The matcher (llama-server on :8791) must be running; axstream up or --doctor handles that.

Optional: the embedding shortlist

With a growing macro library, serve a small embedding model and the app shortlists the 10 most relevant workflows before matching — the matcher's prompt stays small forever, at any library size:

curl -L -o ~/models/Qwen3-Embedding-0.6B-Q8_0.gguf \
  "https://huggingface.co/Qwen/Qwen3-Embedding-0.6B-GGUF/resolve/main/Qwen3-Embedding-0.6B-Q8_0.gguf"
llama-server -m ~/models/Qwen3-Embedding-0.6B-Q8_0.gguf --port 8793 \
  --embedding --pooling last -ngl 99 -c 2048 --no-webui

Without it the app simply uses the full ranked library — nothing breaks.

What you can say

Anything in your macro library, by any phrasing its examples cover — plus:

• Slots fill from speech: "add a torus in blender" runs the mesh macro with shape=torus, including values nobody ever recorded. Slot values must appear verbatim in what you said — the matcher is forbidden from inventing them.
• Compound commands chain: "open blender and select the shape and delete the shape" splits on and/then, and runs as a sequence — but only when every clause matches a macro. A false split ("…note that says milk and eggs") falls back to normal single matching, so chaining never makes anything worse. The HUD shows ▶ 2/3 progress and stops honestly at the first failed step.
• Unknown commands still work: no match hands the utterance to the axstream engine — its learned-template store answers instantly, and with an LLM key set (~/.axstream/env), novel tasks are planned against the live screen, executed, and saved as macros with their variables identified. Slow at most once.

The flywheel

Two background mirrors keep the library growing without you curating it:

1. Learned constructions — when the LLM tier completes a novel task, a second model pass parameterizes it (identifies the slots), generates example phrasings, and writes a .axstream file into your library.
2. Store migrations — when the engine's internal template store answers an utterance and the replay succeeds, that template is copied into the file library too. Neither mirror ever overwrites or displaces a hand-authored macro.

New files are voice-matchable immediately: the matcher was trained to match description-only templates, the compiler seeds a spoken example into every capture, and the shortlist embeds new macros on first sight.

Test modes (no mic needed)

AxstreamBar --status                    # permissions, models, matcher health
AxstreamBar --match "open notes"        # match text → macro + slots + latency
AxstreamBar --plan "open blender and add a sphere then delete the shape"
                                        # resolve a compound chain, execute nothing
AxstreamBar --say "add a cone in blender"   # full path INCLUDING execution
AxstreamBar --transcribe clip.wav       # whisper only
AxstreamBar --pipeline clip.wav         # whisper → match

(AxstreamBar = swift/AxstreamBar/.build/release/AxstreamBar.)

Speech models

* per utterance, warm, Apple Silicon. Override with AXSTREAM_WHISPER_MODEL=/path/to/model.bin.

Troubleshooting

• Hotkey does nothing, no HUD → Accessibility isn't granted (or a rebuild changed the binary's signature — re-toggle AxstreamBar in System Settings → Privacy & Security → Accessibility; it revives within seconds).
• HUD shows "Mic delivered silence" → Microphone permission. Plugging in a different mic mid-session is fine — the recorder rebinds to the current default input on every hold.
• "Matcher offline" → start llama-server on :8791 (or axstream up).
• Transcripts land but nothing matches → AxstreamBar --match "<text>" shows exactly what the matcher decided; add the phrasing to the macro's header examples (header edits don't invalidate verification, and take effect immediately — no restart).

Run history records macro names, outcomes and durations — never slot values. The app dies with your login session; axstream menu brings it back.