Custom engines & architecture

flowchart LR
  A(["audio"]) --> STT["SttEngine<br/>(whisper.cpp)"]
  STT --> T["text"]
  T --> LLM["LlmEngine<br/>(Ollama)"]
  LLM --> R["reply"]
  R --> TTS["TtsEngine<br/>(Piper)"]
  TTS --> O(["audio out"])
  P["VoicePipeline<br/>(owns history)"] -. wires .-> STT & LLM & TTS

Design

whispa is deliberately thin. The core is VoicePipeline, which depends only on three interfaces:

interface SttEngine { transcribe(audio: Uint8Array | string): Promise<string>; }
interface LlmEngine { chat(messages: Message[]): Promise<string>; }
interface TtsEngine { synthesize(text: string): Promise<Uint8Array>; }

Everything else — whisper.cpp, Ollama, Piper — is an adapter implementing one of these. This keeps the loop pure and fully unit-testable with mocks, and lets you mix engines freely.

Write your own engine

Any provider works — a cloud STT, a different local LLM, a hosted TTS — as long as it satisfies the interface:

import type { LlmEngine, Message } from 'whispa';

class MyLlm implements LlmEngine {
  async chat(messages: Message[]): Promise<string> {
    // call your model, return the assistant text
    return '...';
  }
}

Then drop it in:

new VoicePipeline({ stt, llm: new MyLlm(), tts });

Conversation state

• history holds the running Message[] (system prompt first, if provided).
• respond() appends the user message, calls the LLM, appends the reply.
• maxHistory bounds retained turns (the system prompt is always kept).
• reset() clears everything except the system prompt.

Extending the loop

Common additions you can layer on top without touching the core:

• Wake word / VAD before calling turn().
• Streaming TTS by implementing a TtsEngine that yields chunks.
• Tool calling inside your LlmEngine.chat implementation.
• Barge-in by cancelling playback when new audio arrives.