How to Build a Voice-Driven Agentic OS on Claude Code
How to Build a Voice-Driven Agentic OS on Claude Code
You can turn Claude Code into a voice-driven personal assistant that reads your day back to you, runs your skills with a single click, and does it mostly on local models. I call mine Jarvis — a Claude Fable 5 OS that layers a visual, voice-first interface on top of Claude Code's skill architecture. The interesting part isn't the sci-fi UI. It's the routing logic underneath that decides when to use cheap code, when to use a cheap model, and when to spawn a full Claude Code session — and how little of it actually depends on any one model.
Here's exactly how it works and how you can build something like it yourself.
What Is an Agentic OS Built on Claude Code?
An Agentic OS is a layer on top of Claude Code that adds the things you can't get inside the terminal: visual metrics, a voice interface, and skills turned into clickable buttons. The backbone is still a completely customizable Claude Code skill architecture — the same one that takes your manual workflows and daily tasks and turns them into skills and automations.
Jarvis is the next evolution of that model. It was built using Fable 5, but here's the key detail most people miss: it does not require Fable 5 to run. A lot of what's happening is completely local, and it can run on essentially any model you want.
This isn't productivity theater. If you operate across a bunch of different domains — content, an agency, sales, community — a system like this gives you a real boost. And because it's a web app, you can bundle it and hand it to a non-technical client or team member who will never open a terminal in their life.
The value add is that anyone can sit in the chair and get 80-90% of the power of Claude Code with a click of a button.
What Does the Jarvis Interface Actually Do?
When you ask Jarvis for a rundown of your day, it doesn't read a hard-coded script. It looks at the reports automatically generated inside an Obsidian vault, decides what's actually important, and tells you about it out loud.
As it talks, contextual pop-ups appear — each one a report or a link relevant to what it's saying. Mention the latest video's numbers? A deploy pop-up links straight to that video. Mention an AI news story? The source article comes up. Everything is linked back to Obsidian, so you can read it in the app or click "open in Obsidian" to get the original.
The rest of the interface is built the same way:
- Skills panel — every Claude Code skill or automation is a button. Click "inbox brief" and it queues instantly, shows a progress bar, and gives you both a written report and a verbal rundown when it's done.
- Schedule — linked to Google Calendar; click it to open your calendar.
- Vitals — subscriber counts, latest video stats, a tracker for your Claude token usage over the last 5 hours.
- Directives — the top three things you should work on today, generated dynamically by Claude Code from your schedule.
- Documents trail — everything Jarvis creates or references lands here, one click from opening.
Every piece of this is customizable. You don't want a calendar? Rip it out. You want different metrics? Swap them in. The whole point of an Agentic OS is that it isn't one-size-fits-all — it's a set of tools you bend to your own workflows.
How Does Jarvis Route a Voice Command?
This is the part that actually matters. When you talk to Jarvis, your request can go down one of several paths, and the system is deliberately built to use the cheapest, fastest option that will do the job.
Here's the full flow:
- You speak. Your microphone audio goes to Faster Whisper, a free, locally-run transcription program. It turns your voice into text. There are a million local transcribers — this is just one that works fine, and you can swap it for any other.
- The system decides how to route the transcript. There are three options, in order of cost.
Option 1 — Regular Expressions (Free)
First, a deterministic script checks whether the transcript matches a pre-coded trigger phrase. No AI involved. When I said "give me the rundown for today," the word rundown is a trigger word that regex catches and routes to a specific action automatically.
This is the ideal path because it's just code — it costs nothing, involves no AI, and is extremely fast. In a perfect world you'd handle everything this way. But most real requests are too murky for a fixed keyword.
Option 2 — Haiku (Fractions of a Penny)
When regex can't handle it, Jarvis brings in Haiku — Anthropic's cheapest, fastest model. Why the "dumbest" model? Because at this stage you're not doing the work. You're just routing. You need to figure out where the request goes and who needs to handle it, and Haiku is very good at that for a fraction of a penny per request.
Option 3 — A Local Model (Free)
If you'd rather not touch Haiku at all, you can run the routing on a local model on your own machine. Same job — read the transcript, figure out where it goes — with zero API cost and full local control.
What Happens After the Routing Decision?
Say you asked for the morning report. The router — Haiku, in this case — looks at Obsidian first. Does this report already exist on disk? If yes, that's cheap and easy: read it and respond.
If it doesn't exist, the router hands the actual work to a stronger model. The router never executes the task itself. By default that's Opus, but you can drop it to Sonnet or, if you're feeling ambitious, Fable 5.
To generate the report, Jarvis spawns a headless version of Claude Code — it's like opening Claude Code, but invisible, using the -p flag. It creates the report, uploads it to Obsidian, reads it back, and generates a summary.
That summary goes to Kokoro, another local open-source model that turns the text into speech — think of it as a mini ElevenLabs running on your own computer. The voice you hear from Jarvis is Kokoro, and it can be swapped for any voice you want.
So the full loop is: you talk → Faster Whisper transcribes → regex, Haiku, or a local model routes → Obsidian is checked or headless Claude Code generates the report → Kokoro speaks the answer back to you.
Does a Voice Assistant Like This Burn Through API Credits?
This is a fair worry, and the honest answer is: probably not, if you're sensible about it. Note that headless Claude Code run with -p pulls from the additional API credits included with your subscription — not your regular subscription usage.
Could that be a problem at huge scale? Yes. That's why you'd lean on Sonnet for a lot of the heavy lifting instead of Opus, and why the local-model option exists at all.
But be realistic about what Jarvis is for. This is a personal-assistant and task-manager tool, not a codebase builder. You're not building Facebook with Jarvis — if you were, you'd be hardcore in the terminal. For personal-assistant tasks and non-technical team members, running through $200 of credits in a month is unlikely.
And if it ever is a problem, don't use Claude Code for it. It's called a Fable 5 OS, but you could swap in Codex or replace the whole thing with local models. The infrastructure is customizable and can be swapped out with whatever you want.
Why Is the Skill Architecture the Real Backbone?
Strip away the UI and the voice, and what's left is the thing that actually gives Jarvis its power: the skill architecture.
A "morning report" isn't a single command. It's a skill — one made up of smaller skills that say "go check these sources, go check these social pages, pull this information." And that skill architecture is only as strong as you make it.
Here's how you build one. Sit down with Claude Code and give it a stream of consciousness about what you actually do every day — the tasks you repeat over and over in your business or personal life. Then ask it to break those into individual tasks, turn the tasks into skills, and, where it makes sense, turn the skills into automations.
That's how you get real skills. A few of mine:
- YT pipeline — searches YouTube for information, sends videos to NotebookLM, and returns summaries.
- Deep research — a customized skill that goes beyond YouTube into Twitter and the wider internet.
- Graph RAG query — an entire LightRAG system holding what I've done in the past, turned into a queryable skill.
Then you repeat that across every domain — content, community, agency, sales — until your whole working life is codified.
The payoff is determinism. When you tell Jarvis to run a skill, you get output that's correct and consistent, because you've already mapped out the process. Nothing is left to chance. Every time you can make an AI system more deterministic, you're less exposed to the "maybe it works, maybe it doesn't" problem that plagues AI in general.
How Would You Build This for a Client or Team?
This is where an Agentic OS earns its keep. Because Jarvis is a web app, you can bundle it and share it with clients or team members. They tell you what they want on the customization side — their metrics, their one-click skills — and you build it around their workflows.
The real work you do for them is walking them through codifying their own tasks into skills. Once that's done, anyone can sit in the chair and get 80-90% of the power of Claude Code the way a technical user gets it — every skill and automation one click away, no terminal required. Add the voice layer on top and it feels effortless, but the substance is the skill architecture underneath.
Frequently Asked Questions
Do you need Fable 5 to run an Agentic OS like Jarvis?
No. Jarvis was built using Fable 5, but it doesn't require Fable 5 to run. Most of what runs under the hood uses Opus, Sonnet, or even local models, and much of the system — transcription and voice — is fully local. You can tell it to use Fable when it's available, but it's not required.
What local tools does Jarvis use for voice?
Two open-source, locally-run models. Faster Whisper transcribes your spoken commands into text for free, and Kokoro turns Claude Code's text responses back into speech — a lightweight local alternative to a cloud service like ElevenLabs. Both can be swapped for other tools or voices.
Why route commands through Haiku instead of a bigger model?
Because routing isn't the work — it's just deciding where the request should go. Haiku is cheap and fast, costs a fraction of a penny per request, and is more than capable of figuring out which path a command should take. The heavier execution is then handed to Opus or Sonnet, or a local model if you prefer.
What does headless Claude Code mean here?
Headless Claude Code is Claude Code run invisibly with the -p flag — no interactive terminal window. Jarvis spawns it in the background to generate reports and run skills, then feeds the output back into Obsidian and the voice layer. On current subscriptions, these runs pull from included API credits rather than your normal subscription usage.
Is an Agentic OS worth it if I'm not a content creator?
Yes — the specific skills just change. The content-research skills shown here are examples; the pattern is what transfers. Any repeated task in your business or personal life can be codified into a skill and put one click away, which is exactly what makes the system valuable for non-technical teams and clients.
If you want to go deeper into building your own Agentic OS on Claude Code, join the free Chase AI community for templates, prompts, and live breakdowns. And if you're serious about building with AI, check out the paid community, Chase AI+, for hands-on guidance on how to make money with AI.
