A self-hosted AI agent platform

Problem

Most agent projects stop at a single call to a hosted model. I wanted to own the whole stack, from hardware and speech recognition to model serving, tool calling, and tracing, and to find out where a small local model is good enough and where it isn’t.

What I built

Everything runs on a GMKtec M6 Ultra mini PC running Omarchy Linux, reachable remotely over Tailscale.

• Jarvis, a local-first voice assistant. Home Assistant Voice PE devices capture speech, Whisper turns it into text, and Gemma 4B, served locally through Ollama, handles requests with tool calling (web search and other agent tools).
• A progressive web app that exposes agent tools (scheduling, task management) and Home Assistant device control through one interface.
• Self-hosted Langfuse, tracing the agent’s LLM calls.

1. Voice PE device

   Home Assistant

2. ↓

   Whisper

   speech to text

3. ↓

   Intent matcher

   Home Assistant

   matched → device action, instant, no LLM

4. ↓ unparsed

   Gemma 4B

   Ollama · tool calling

   traces → self-hosted Langfuse

5. ↓

   Tools

   web search + more

   Jev router will gate these calls (in progress)

Key decisions & trade-offs

Route before you reason

Home Assistant’s intent matcher sits in front of the model. Simple commands it can parse run immediately as device actions; only requests it cannot parse escalate to the LLM.

That trade-off is the core of the design. The fast path is near-instant, costs nothing, and never leaves the house. The LLM path is slower but can handle open-ended requests and call tools. Anything the matcher can parse never waits on the model, so the model only spends time where its capability is actually needed.

Local model over cloud API

A 4B model served locally is less capable than a frontier cloud model. In exchange, speech recognition, intent matching, and the model all run on hardware I control: voice data stays at home, there is no per-request cost, and only tools that need the internet, like web search, reach outside.

Trace it like production

Running Langfuse against a home system is deliberate. When the assistant picks the wrong tool, I want the trace, not a guess.

In progress

I’m integrating TypeSafe’s Jev, a “System One” decision model, as a router that gates tool calls before they execute. It is not running yet.

Result

The platform runs today on hardware I own. Everyday device commands are handled instantly by the intent matcher without touching the model, open-ended requests go to a local LLM with tool calling, and every LLM call is traced in Langfuse.