Case study / 01 / Local-first macOS agent
Jarvis is a local-first macOS agent I built for everyday desktop work. It combines voice input, authorized screen context, persistent memory, and bounded actions while preserving visible confirmation for anything consequential.
Have a Mac workflow you would like to simplify? Tell me what you do today and where it gets in the way.
Private build. Public access and pricing are not available yet.
01 / Problem
A voice assistant becomes unreliable when audio is noisy, permissions change, context shifts, or a workflow completes only halfway. Jarvis separates perception, interpretation, policy, execution, and verification so each failure can be handled deliberately.
02 / My role
I defined the product behavior and safety contract, directed the architecture, implemented and reviewed the native and Python layers, designed acceptance tests, and own the release process.
03 / Approach and decisions
Jarvis is structured so perception, interpretation, policy, execution, and verification remain distinct. The model can help understand a request; deterministic policy and visible confirmation decide what may happen next.
Voice recognition personalizes access; Touch ID and visible confirmation authorize credentials and consequential actions.
Local chart-vision models were measured and rejected when speed and accuracy could not meet the bar. The working system keeps deterministic paths fast and specialists on demand.
A bounded watchdog reports failures without repeating commands, forwarding an unverified voice, or bypassing confirmation.
Request lifecycle
Each layer has one bounded responsibility. That keeps permissions, confirmation, cleanup, and success reporting outside model confidence.
04 / Authentic interface
The interface grows from a compact 112-point presence to a 248-point active state, then settles back into standby. This native preview shows the current plasma layers, iris, sweeping light, and response aura across seven states. Use the controls to explore the sequence.
Native interface preview · Simulated state and audio input. Motion is off until you press play.
Rendered from the current macOS animation code, including the full expansion and collapse. Subtle GPU particle effects are absent from this render. Download the 22-second preview ↗
05 / Delivered system
Apple SpeechAnalyzer handles everyday dictation; local Whisper is an on-demand recovery path for low-confidence audio. Silence detection, background media, device changes, and natural phrasing are treated as normal operating conditions.
ScreenCaptureKit supplies explicit visual context while SQLite FTS5 preserves useful preferences, project evidence, and workflow state with provenance and a locally enforced private mode.
Actions are separated from model suggestions. Consequential typing, clicking, sending, settings changes, and external effects stay behind visible confirmation and verification steps.
Deterministic commands stay instant. Apple Foundation Models, local Qwen, transcription recovery, and visual reasoning are selected by task and unloaded when idle.
06 / What the build demonstrates
Installed-build evidence and hardware-dependent behavior are kept separate. These design outcomes are part of the current working system.
The seven-state Core Animation presence makes system state visible without a display loop or resident inference model. An August 2026 installed-build check measured 42.7 MB at settled idle on the development Mac.
Voice acceptance is tuned against labeled samples, and model routes are kept only when measured accuracy, latency, and resource use clear the product bar.
Permission changes, interrupted audio, partial actions, Siri handoff, device changes, and workflow cleanup each have bounded recovery and a verifiable outcome.