About

This is an implementation of a real-time voice assistant. Your speech is transcribed live, reasoned with a language model and responded to with a natural voice. It can also answer questions about PDF or TXT files you upload, and reach live tools for weather, currency conversion, and more.

Pipeline

0 0.5s 1s 1.5s 2s

Capture

hold Space, or touch and hold, and speak

STT

Moonshine STT

LLM

Groq LLM Call

TTS

Piper TTS

Playback

Browser · Web Audio · gapless · Space or touch-hold = barge-in

Speech is streamed to the server as raw audio over a single WebSocket; transcription, generation, and synthesis are chained so stages overlap: the transcript streams while you speak, and playback starts while the voice is still being synthesised.

Agent

• A · Turn input: The transcript, plus the last 20 messages, the active system prompt (custom, preset, or default), and an “a PDF is uploaded” note when a document is loaded.
• B · LLM: Inference call with tool_choice = auto using Groq gpt-oss-20b.
• C, D, E · Tool execution loop: Evaluates if the model emitted tool calls (C). If yes, it runs the chosen tool and appends the result (D), looping back to the LLM. This is capped at 5 iterations per turn (otherwise triggering a fallback reply). When no more tool calls are needed, the flow proceeds to the final answer text (E).
• F · Speak reply: Cleans the final text via strip_markdown, then synthesizes speech using Piper TTS.
• G, H · Tools & Document search (RAG): The agent’s registry of 7 free, keyless tools (G). When search_document is chosen, it processes PDFs (≤ 10 pages / 20 MB) into 1000-char chunks with 150-char overlap, generates bge-small-en-v1.5 embeddings, and retrieves the top-4 results by cosine similarity (H).

Delivery

A GitHub Actions workflow is set up to identify affected components and rebuild only what has changed. Both the backend and frontend are containerized (compatible with Docker Compose) and built for linux/amd64, leveraging the GitHub Actions layer cache to keep rebuilds fast and efficient.

The application is hosted on a Hetzner CX33 server, running a K3s cluster orchestrated by the voice-agent Helm chart. This chart provisions the namespace, Deployments, and Services, alongside an ingress-nginx Ingress configured with cert-manager for automated Let’s Encrypt TLS at qharo-voice-agent.duckdns.org. The Ingress is explicitly tuned with WebSocket-friendly timeouts and a 20 MB upload cap. Sensitive credentials like the GROQ_API_KEY are safely committed to the repository as a Bitnami SealedSecret, remaining encrypted until decrypted in-cluster. During deployment, the workflow applies changes efficiently: it runs helm upgrade if the chart itself changed, or falls back to a targeted kubectl rollout restart for the specific Deployment, waiting for the rollout to stabilize.

For observability, the backend exposes Prometheus metrics at /metrics. A ServiceMonitor scrapes these endpoints every 15 seconds, feeding a comprehensive Grafana dashboard that is shipped directly as a Kubernetes ConfigMap. This provides real-time visibility into active sessions, queue depth, per-stage latencies (STT, LLM, TTS), tool call rates, and error counts.

Notes

Design tradeoffs

• Push-to-talk instead of voice-activity endpointing — you decide when a turn ends, which keeps the interaction predictable and the server simple.
• The LLM is hosted (Groq) for speed and quality, but STT and TTS run locally in the backend process — allows for no per-request cost, no extra network hop, and offline inference.
• Concurrency is capped at four turns so a small box stays responsive; extra callers wait in a fair queue rather than degrading everyone.
• RAG holds one document at a time, and the prompt / preset / document state is process-global rather than per-session — deliberately minimal for a single-user demo.

Known limitations

• The LLM response isn’t streamed into TTS yet — the full reply is generated first, then synthesised. STT already emits partial transcripts, but the live turn still waits for the final text.
• The RAG index lives in memory, so an uploaded document is lost on a backend restart or redeploy.
• Voices are English-only.

Credits & links

The models and tools are all free and keyless: Moonshine (STT), Piper (TTS), gpt-oss-20b served by Groq, and open APIs for weather (Open-Meteo), currency (Frankfurter), Wikipedia, dictionary, and world time.

Source on GitHub  ·  Live demo