Skip to content
All projects

Case study 03AI · Automation · Interface

Personal AI Assistant

An assistant that does things, not just answers.

A custom AI assistant with a reactive, systems-style interface. It routes intent, calls tools, remembers context and automates the small workflows that eat a day.

Role
Design and engineering — architecture, orchestration, interface
Platform
Desktop & web
Stack
Python, LLM tool calling, RAG, Speech I/O, WebSocket, Web UI
Links
Private project — demo on request
local / hud

01Overview

A personal assistant inspired by the intelligent-assistant systems of film, built to be genuinely useful rather than theatrical. The interface communicates what the system is doing at every moment; the backend turns language into actions through a controlled set of tools.

02Problem

General chat assistants answer well but act poorly: they don't know your context, can't touch your tools, and give no sense of what they're doing while they work.

03Goals

  1. G1Turn natural language into real actions safely
  2. G2Remember relevant context across sessions
  3. G3Make latency feel like progress, not waiting
  4. G4Keep the architecture modular so new tools are cheap to add

04My role

Design and engineering — architecture, orchestration, interface.

  • Assistant architecture and orchestration loop
  • Tool design, permissioning and error handling
  • Memory and retrieval pipeline
  • Interface design and realtime state visualisation

05Technology

Core
Python / Async orchestration
Intelligence
LLM APIs / Tool calling / RAG / Embeddings
Interface
Web UI / WebSocket streaming / Speech-to-text / Text-to-speech

06Architecture

A small orchestrator sits between the interface and the model. The model proposes; the orchestrator decides what actually runs.

  1. 01

    Interface

    Voice or text in; streamed tokens and state changes out over WebSocket

  2. 02

    Orchestrator

    Conversation state, intent routing, retries and timeouts

  3. 03

    LLM + tools

    Model selects from an allowlisted tool registry with typed arguments

  4. 04

    Memory

    Rolling summary for short-term context, retrieval for long-term knowledge

  5. 05

    Automations

    Scheduled and on-demand workflows reuse the same tool layer

07Challenges & solutions

C1Latency is the user experience

Multi-step tool use can take seconds. A silent screen makes a capable system feel broken.

Multi-step tool use can take seconds. A silent screen makes a capable system feel broken.

Stream everything: tokens, tool calls and state transitions. The interface maps each state to a distinct visual so waiting reads as work happening.

Stream everything: tokens, tool calls and state transitions. The interface maps each state to a distinct visual so waiting reads as work happening.

C2Letting a model act safely

An LLM with unrestricted tools is one misread instruction away from doing damage.

An LLM with unrestricted tools is one misread instruction away from doing damage.

Typed, allowlisted tools with validation, and an explicit confirmation step for anything destructive or outward-facing.

Typed, allowlisted tools with validation, and an explicit confirmation step for anything destructive or outward-facing.

C3Context that scales

Stuffing full history into every request is slow, expensive and eventually impossible.

Stuffing full history into every request is slow, expensive and eventually impossible.

A rolling conversation summary plus retrieval over personal notes, so each request carries only what's relevant.

A rolling conversation summary plus retrieval over personal notes, so each request carries only what's relevant.

08Screens

  • local / hud
    a.System view — state, tools and memory in one frame
  • local / chat
    b.Conversation with visible tool calls

09Results

  • A working assistant used for real daily workflows
  • A modular tool layer where adding a capability is a small, isolated change
  • An interface that makes the system's state legible at a glance

10Lessons learned

“The orchestrator matters more than the prompt.”

“Visible state builds more trust than faster answers.”

“Design tools for the model like you design APIs for people: small, typed, forgiving.”