Introduction

Voice to action, as fast as dictation

You say "reply to Lawrence, I'll see him tomorrow" — and it happens at the speed of speech. That's the goal. Today's voice control lags because of three things, and each has a fix:

1. Vision models. Agents screenshot the screen and ask a vision model where to click. Your computer already exposes every button and field as text — the accessibility tree. Reading text beats looking at pixels.
2. Waiting for the full response. Models stream token by token, yet agents wait for the whole plan before acting. Stream the actions instead, and execute each one the moment it arrives.
3. Figuring out what you meant. Which app? Who's Lawrence? This is the real latency killer — and most of what you do, you've done before. Learn it once; next time, skip the model entirely.

The idea

axstream is a small structured language — one action per line — that a model can stream, a runtime can replay, and a person can read:

{"op":"act","do":"open","target":"Notes"}
{"op":"act","do":"wait","ms":500}
{"op":"act","do":"type","text":"remember to buy milk"}
{"op":"done","status":"success"}

Each line executes the instant its newline arrives — the open fires while the model is still generating the type. A line either arrives whole and valid or it doesn't execute, so streaming is safe by construction.

One language, three speeds:

• Instant — commands you've used before replay directly, ranked by frequency and recency (zoxide's trick, applied to actions). A local 350M model matches your words to a learned command in ~100ms. No LLM in the loop.
• Fast — novel commands are planned by a streaming LLM over the accessibility tree, executing as lines arrive. Every success is captured into the instant tier — the system gets faster the more you use it.
• Fallback — apps with no clean accessibility surface get driven by computer use.

Hold ⌃⌥ and talk

The reference surface is AxstreamBar, a native menu bar app: hold control-option, speak, release. Local whisper hears you (~450ms), an embedding shortlist plus the fine-tuned matcher pick the workflow (~140ms), and the verified replay runs — with a HUD narrating every step. Compound commands chain ("open blender and add a sphere then delete the shape"), slots fill from your words, and anything unknown is constructed once by an LLM and saved as a workflow with its variables identified. → The voice menu bar

Not just voice: your coding agent's fast hands

The same replay layer plugs into Claude Code and Codex — one axstream install gives the agent a skill plus MCP tools: read the screen as text instead of screenshots (~200ms), execute whole batches of verified actions in one call, and compile any task it has done once into a macro that replays in seconds with zero model calls. Clicks resolve through a verified ladder (accessibility element → OCR text anchor → visual patch → window-relative pixels) and a macro is never trusted until a live replay passes its outcome assert. → For coding agents

Measured, not promised

From the reference implementation, on a live macOS machine:

The format is model-agnostic and transport-agnostic: any LLM that can stream text can produce it; any runtime that can parse a line can perform it.