Ada
Real-Time Turkish AI Voice Sales Assistant
The Operational Problem
Context & Domain
Most production voice AI is English-first and cloud-only. For Turkish businesses two things matter: natural Turkish speech, and the option to keep customer conversations off third-party APIs. Ada was built in staged increments (v0.1 → v0.2.2) to test both, as a browser-only technical preview before any telephony work.
Overview
A sales conversation is more than chat: the assistant has to move through stages, remember what the prospect said and score the lead, consistently, whichever speech and language engine is underneath. Small local models are far less reliable at strict instruction-following and tool calling than hosted realtime models, and a turn-based local pipeline adds latency at every stage.
System Architecture
Ada separates the sales logic from the voice engine. A provider-independent Sales Brain (stage machine, lead context, lead scoring) sits behind an IVoiceAgent abstraction with two implementations: an OpenAI Realtime agent that reports sales data through a native tool call, and a local agent that chains in-process Silero VAD, faster-whisper STT, Qwen3 8B on Ollama and Piper TTS, reporting sales data through a validated JSON envelope. The browser speaks one event protocol and needed zero changes when the local provider was added.
┌───────────────────────────────────────────────────────────────────────────┐
│ BROWSER CLIENT (static HTML/JS, no build step) │
│ AudioWorklet mic capture · PCM16 mono 24 kHz both directions │
│ Live pipeline stages: Mic → WS → VAD → STT → LLM → TTS │
└─────────────────────────────────────┬─────────────────────────────────────┘
│ same-origin WebSocket /ws/voice
▼
┌─────────────────────────────────────┴─────────────────────────────────────┐
│ ADA.API (ASP.NET Core 8) │
│ ├── IVoiceAgent: provider picked by config, Local → OpenAI fallback │
│ └── SALES BRAIN (provider-independent) │
│ stage machine · lead context · lead scoring │
│ one four-part prompt: identity · sales · behaviour · compliance │
└─────────────────┬───────────────────────────────────────┬─────────────────┘
│ │
OpenAI │ │ Local (no cloud)
▼ ▼
┌─────────────────┴─────────────────┐ ┌─────────────────┴─────────────────┐
│ OPENAI REALTIME API │ │ Silero VAD in-process (ONNX) │
│ streaming speech ⇄ speech │ │ faster-whisper STT (FastAPI) │
│ sales data via tool call │ │ Qwen3 8B on Ollama → JSON reply │
│ update_sales_context │ │ Piper TTS (persistent process) │
└───────────────────────────────────┘ └───────────────────────────────────┘Captures microphone audio with AudioWorklet, plays PCM16 replies, and shows live pipeline stages and per-turn latency
Same-origin WebSocket relay, health checks and provider selection with fail-fast configuration validation; API keys never reach the browser
Stage machine, lead context and lead scoring shared by both providers, driven by one four-part system prompt
Silero VAD in-process via ONNX Runtime; a FastAPI sidecar for faster-whisper and a persistent Piper process; Qwen3 8B served by Ollama
Engineering Decisions
One Sales Brain, Two Voice Providers
Sales behaviour must not depend on which speech engine is active; otherwise every provider change means re-testing the whole conversation.
Each provider maps its own protocol onto Ada's internal event model, but the sales logic is tested once and the browser client stays provider-agnostic.
JSON Envelope Instead of Local Tool Calling
Tool-calling reliability of a small local model is not guaranteed.
The local LLM always replies with {response, context_updates}. Every field is trimmed, length-capped and validated before it reaches the tracker, and a malformed reply degrades to plain speech instead of ending the session.
Model Chosen by Live Conversations, Not Benchmarks
qwen3:1.7b was about 2x faster than qwen3:8b in a scripted benchmark.
In a live five-turn conversation it repeated identical replies to different inputs, so qwen3:8b stayed the default: about 2 s slower per turn, but coherent. qwen3:4b leaked chain-of-thought and phi3.5 produced broken Turkish.
Technical Challenges & Solutions
Silent Voice Activity Detection Failure
Silero VAD returned near-zero speech probability for real speech, without raising any error.
Current Silero ONNX builds need each 512-sample call prefixed with a 64-sample rolling context window from the previous call. Ada's .NET implementation adds it and was cross-checked against the reference Python wrapper.
Per-Turn Latency of a Local Pipeline
The first local version took about 7.3 s per turn in steady state.
Kept one Piper process alive instead of spawning one per utterance (TTS −32.8%), moved STT from Whisper small to base after benchmarking (−58.8%), and added per-stage timing logs and a live latency panel. Steady state dropped to about 5.9 s per turn (−19%).
Technology Stack
- • ASP.NET Core 8
- • C#
- • WebSockets
- • Serilog
- • OpenAI Realtime API
- • Silero VAD
- • faster-whisper
- • Qwen3 8B (Ollama)
- • Piper TTS
- • Python
- • FastAPI
- • uvicorn
- • xUnit
Key Implementation Highlights
- One PCM16 24 kHz binary and JSON event protocol for both providers; the local agent resamples internally
- Automatic Local → OpenAI fallback when the local stack is down, always logged, never silent
- /health/local-ai endpoint reporting LLM, STT and TTS availability separately
- No audio persisted anywhere; local STT audio stays in memory for a single utterance
System Diagram & Measurements
System topology with real values measured in the development environment.
Ada Voice Pipeline: Two Providers, One Sales Brain
Measured local turn on a 6-core CPU with an RX 580: STT 824 ms, LLM 4,324 ms, TTS 739 ms, about 5.9 s per turn in steady state.
Results & Current State
End-to-end Turkish voice sales conversations work in the browser on both providers. Telephony (SIP/Asterisk), CRM and calendar integration are deliberately out of scope for this stage.
Key Architectural Takeaways
- 01.Keep business logic independent of the AI vendor; swapping engines should not mean rewriting the product.
- 02.Small local models need defensive parsing, not trust: validate every field the model returns.
- 03.Measure in real conversations; the fastest model in a benchmark was not the best one in a live call.