How it works

You say a command. axstream does one of two things:

• It recognizes it — a tiny local model matches your words to a command it already knows, and replays it. ~100ms decision, no LLM, no network.
• It doesn't — a full LLM looks at the screen, does the task once, and the successful run is saved. Next time, it's recognized.

Every command is slow at most once.

The stream

When the LLM does plan, axstream doesn't wait for it to finish. Actions come out one per line, and each line executes the moment it ends:

A line either arrives whole or it doesn't run — the newline is the commit signal, so a half-written action can never fire. This alone saves about a third of the wall-clock time versus wait-then-act.

The matcher — the model we trained

The recognizer is a 350M-parameter model (open on Hugging Face) that answers one multiple-choice question with fill-in-the-blanks:

in:   your command library            "which of these is it,
      + what you just said            and what are the variable words?"

out:  {"template": "open_app", "slots": {"app": "apple music"}}

That's its entire job. It never plans, never sees your screen, never invents actions. Three properties make it trustworthy:

• It can only answer from the list. Output is grammar-constrained — an answer outside your command library is impossible to generate.
• Slots are copied, never composed. A slot value that you didn't actually say is rejected. The matcher cannot put words in your mouth.
• When unsure, it says "none". A wrong "none" costs one slow LLM call. A wrong match could do the wrong thing — so it's trained to refuse.

Fine-tuning took it from 47% to 94% end-to-end — trained in minutes on one rented GPU for well under a dollar. The current v2 round trained on a harder curriculum — variable library sizes (10–40 templates), near-identical sibling commands, deliberate look-alike traps ("open apple maps" when no maps command exists), and description-only templates — taking wrong-action on unknown commands from 23% to under 5% and holding accuracy flat as the library grows.

The shortlist

Before the matcher sees anything, a small local embedding model ranks the whole library against your words and passes only the top 10 candidates. Measured on the live library, the right command is in that shortlist 100% of the time — so the matcher's prompt stays small and in-distribution no matter how many workflows you accumulate. Similarity decides membership only; the candidates keep their stable library order (ordering by similarity measurably flips borderline picks). If the embedding server is down, the full ranked library is used instead — strictly graceful.

Chains

Compound commands — "open blender and add a sphere then delete the shape" — split on and/then and run as a sequence, but only when every clause matches a known command; otherwise the utterance is matched whole, so false splits are harmless. Each step is verified before the next fires, and the chain stops honestly at the first failure.

The library

Learned commands live in one JSON file: ~/.axstream/macros.json. Each entry is a template with holes:

{
  "id": "new_note_titled",
  "examples": [{"utterance": "make a note titled standup"}],
  "actions": [
    {"do": "open", "target": "Notes"},
    {"do": "key",  "keys": ["cmd", "n"]},
    {"do": "type", "text": "{title}"}
  ]
}

The LLM writes the program once; the matcher becomes its parser forever after. Commands you use most float to the top (the same frequency-plus-recency ranking zoxide uses for directories), replays are guarded against the live screen, and anything hard to undo is gated behind a risk flag.

File macros — the human-readable .axstream files in ~/.axstream/macros/ that agents author and the menu bar replays — are the durable form of the same idea, with slots, spoken examples, and a verification stamp earned by one live gated replay. Two background mirrors keep the two forms converging: every novel task the LLM tier completes is parameterized and saved as a file, and every store template that executes successfully migrates to a file on its first run. Nothing ever overwrites a hand-authored macro.

That's the whole system: recognize or learn, stream instead of wait, and get faster the more you use it.