Voice AI Companion
A production-shaped voice pipeline — speech in, reasoned reply out — with memory, safety, and observability as first-class services, not afterthoughts.
Business Context
The situation
An internal product build (codename "Haiya") that answers a question most voice-AI demos dodge: what does it take to run a conversational voice system responsibly? Not a single model call, but a pipeline where speech is transcribed, reasoned over with memory, spoken back, and gated for safety — with the auth and observability you'd put behind anything real.
The hidden cost
- Most voice demos are one model call with no memory, no safety, and no session state
- Latency compounds across transcription, reasoning, and speech — it has to be engineered, not hoped for
- A companion that remembers is only trustworthy if it also forgets, filters, and stays in policy
The Problem
What actually had to be solved
Turn a raw voice interaction into a dependable loop: transcribe speech, assemble the right context (including memory), reason with an LLM, speak the answer back, and do it all behind hardened auth — while keeping safety and memory as independently controllable concerns.
Why the existing approach failed
A prompt-and-a-model prototype has no session, no memory boundaries, and no safety gate — so it can't be trusted with continuity or edge cases. Bolting those on afterward tangles them into the prompt. They needed to be services from the start.
Constraints
- Perceived latency across STT → reasoning → TTS must stay conversational
- Memory has to help continuity without leaking or over-retaining
- Safety must be enforceable independently of the model's own output
- Treat a personal-feeling product with production-grade security anyway
Before → After
How the workflow changed
Before
- User speaks
- Single model call
no memory, no safety
- Text reply
- No session continuity
After
- User speaks
- Streaming STT
- Context + memory assembled
- Safety gate
deterministic policy check
- DeepSeek reasoning
- Polly speaks reply
- Session persisted
Architecture
How the system is built
Decompose the voice loop into dedicated services — conversation, memory, safety, session, persistence, and observability — so each concern evolves on its own. Speech is transcribed by an STT service, context (including retrieved memory) is assembled and passed through a safety gate, DeepSeek generates the reply, AWS Polly voices it, and the session is persisted. Auth is hardened with TOTP-2FA, CSRF protection, and rate limiting.
Architecture — click any node
Client
API
Orchestration
AI
Integration
Data
Observability
Client
Web App
Next.js 16 / React 19
- Purpose
- Captures mic input and plays synthesized speech.
- Responsibility
- Voice UX, streaming, session state (Zustand).
- Failure mode
- Falls back to text if audio capture fails.
AI vs. Engineering
Where AI helps — and where it doesn't
AI Layer
The LLM (DeepSeek) reasons over an assembled context; the memory service decides what past context is relevant to include. AI is used precisely where reasoning and relevance create leverage — the dialogue itself.
Engineering Layer
Everything around the model is deterministic software: the STT/TTS pipeline, the safety gate (which fails closed), session and persistence services, and the hardened auth surface. This is what turns a model call into a system you can operate.
Engineering Decisions
The hard calls, with their tradeoffs
Decision 01
Safety and memory as separate services — not prompt text
Choice — Enforce safety in a dedicated gate and manage memory in its own service, outside the prompt.
Why — Policy you can't enforce independently of the model isn't policy. Memory that lives in the prompt can't be bounded, audited, or evolved.
Tradeoff — More service boundaries to maintain, in exchange for concerns you can control and change independently.
Decision 02
STT + Polly TTS pipeline vs. a single voice model
Choice — Compose transcription, reasoning, and synthesis as separate stages.
Why — Discrete stages give control over latency, caching (Polly output to S3), and swapping any one component without rebuilding the loop.
Tradeoff — More moving parts and orchestration, in exchange for control over cost and quality at each stage.
Decision 03
Production security on a companion app
Choice — TOTP-2FA, CSRF protection, rate limiting, Helmet, and reCAPTCHA — from the start.
Why — A friendly-feeling product still holds personal conversation data; it deserves real auth, not a login form.
Tradeoff — More onboarding friction, in exchange for a defensible security posture.
Cost & Reliability
Built to run, not just to work
Cost strategy
DeepSeek chosen for a strong cost-to-quality ratio on conversational reasoning; TTS caching and Redis reduce repeat work. The pipeline shape makes it cheap to route simpler turns to cheaper components later.
Reliability
Each pipeline stage degrades gracefully — a failed STT re-prompts, a model timeout falls back to a safe response, and the safety gate fails closed. Session state persists so a dropped connection resumes rather than resets.
Security
JWT auth with TOTP-2FA (otplib + QR enrolment), CSRF protection, rate limiting, Helmet headers, Firebase-admin, and reCAPTCHA v3 on the frontend.
Performance
Redis caches hot session and context reads to trim per-turn latency; Polly output is cached to S3 so repeated phrases aren't re-synthesized.
By the Numbers
What we can point to
6 dedicated
VerifiedPipeline services
Conversation, memory, safety, session, persistence, observability.
TOTP-2FA + CSRF
VerifiedAuth hardening
Plus Helmet, rate limiting, reCAPTCHA v3.
Fails closed
VerifiedSafety enforcement
Policy gate independent of model output.
Conversational target
IllustrativePer-turn latency
Engineered via streaming STT + caching; not a benchmarked figure.
Business Impact
System → workflow → outcome
System
A decomposed voice pipeline with memory, safety, and observability services.
Workflow
Speech → transcription → context + memory → safety gate → reasoning → spoken reply → persisted session.
Operational change
A conversational loop that stays in policy, remembers within bounds, and can be observed and operated.
Business impact
Demonstrates that a voice AI product can be built to a production standard — the difference between a demo and something you can put in front of users.
Reflection
Lessons & what we'd do next
Lessons
- Making safety a fail-closed gate — not prompt text — is what makes it trustworthy.
- Separating the voice loop into services made each latency and quality tradeoff visible and tunable.
Risks
- Latency is cumulative across stages and needs continuous attention as components change.
- Memory scope is a product and privacy decision, not only an engineering one.
Improve next
- Add turn-level evaluation (transcription accuracy, safety catch-rate, latency budget per stage).
- Introduce model routing so simple turns use a cheaper path.
Technology
The verified stack
Frontend
Backend
AI
Voice
Data
Security
Ops
Could this be relevant to your business?
If you're taking a voice or conversational AI prototype toward production — and need memory, safety, and latency handled as real engineering — this is the kind of build we do.
Related Work
More systems we've built
AI Agent Operations Platform
An AI sales/support agent built like infrastructure — services over gRPC and a message queue — to explore what agent operations look like when they're engineered, not scripted.
Autonomous Browser Agent + Harness
An autonomous browser agent with a real harness: durable, resumable state and a human approval gate before it's ever allowed to act.