How to Build a Real-Time AI Voice Agent: Speech Recognition, LLM and Speech Output

A real-time voice agent connects audio capture, speech recognition, a language model and speech output, with explicit turn-taking and cancellation. Build a simple push-to-talk pipeline first, then reduce delay and add interruption support. A working transcription script and a speaking chatbot are only components; the conversation needs coordinated state and permission rules.

Build one complete turn before streaming everything

Start with microphone permission and a visible recording control. In a browser, getUserMedia documentation describes requesting audio access and its secure-context and permission requirements.

A minimal instructional capture step is:

async function openMicrophone() {
  try {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    return stream;
  } catch (error) {
    console.error("Microphone unavailable:", error.name);
    return null;
  }
}

This requests a stream; it does not encode, transmit or transcribe it. Add an explicit stop control that calls stop() on the stream's tracks. No microphone or voice test was performed for this guide.

For an initial prototype, record one consented utterance, transcribe it, send the checked text to a model and synthesise a short answer. Use documented interfaces for each chosen component. faster-whisper supplies one local transcription route, while Piper is one local speech-output candidate with its own model and licence requirements.

Keep each adapter separate: audio input, transcription, model request and playback. This makes it possible to test a bad transcript without invoking the model or to inspect generated text before speaking it.

Add turn state and cancellation deliberately

Define states such as idle, listening, processing and speaking. A user action should trigger a clear transition, and an error should return to a recoverable state. Avoid leaving the microphone active after the interface claims recording stopped.

Voice activity detection and endpoint detection help determine when speech begins and ends. Their thresholds influence whether the agent interrupts too early or waits too long. Evaluate them with background noise and natural pauses instead of copying settings from a quiet demonstration.

For streaming responses, avoid speaking each incomplete token immediately. Accumulate text into suitable units and preserve punctuation or sentence boundaries where practical. Too much buffering increases delay; too little can produce unnatural or misleading partial speech.

Assign an identifier to each conversational turn. When the user interrupts, cancel ongoing generation where supported, stop playback and ignore late results from the previous turn. Otherwise old speech can resume after a new question, making the assistant appear confused.

Handle echo: the microphone may capture the agent's own speech. Audio processing and device conditions affect the solution, so test with speakers and headphones. A voice detector alone may not distinguish the user's interruption from playback.

Measure experience and protect actions

Record timestamps for end of user speech, final transcript, first model output and first audible response. Break down the delay before optimising. A fast language model cannot compensate for excessive endpoint waiting or slow speech synthesis.

Use realistic recordings with accents, code-switching, silence and network failures. For Indian-language workflows, evaluate transcription and speech output separately. A model supporting a language does not prove natural pronunciation or reliable recognition for your vocabulary.

Keep model responses concise enough for listening. A long answer that reads well can be difficult to follow aloud. Offer clarification or a visible transcript for detailed information, and make recording state and cancellation accessible.

Before external actions, confirm important details through a trusted interaction. Misrecognising a name or amount should not silently create a payment or send a message. Tool permissions and approval belong in the application, not in a promise that speech recognition is usually accurate.

Review recordings, transcripts, logs and provider endpoints as one data path. Minimise retention and obtain appropriate recording permission. A local transcription component does not establish that the model or speech service stays local.

Your next step is a push-to-talk prototype with a complete checked turn, then interruption tests using synthetic tasks. Add continuous listening or streaming only when the basic state machine and cancellation behaviour are dependable.

Quick questions, clear answers

Is batch transcription enough for real-time conversation?

It can support a first push-to-talk prototype, but interactive experience also depends on partial-result delay and detecting the end of speech. Measure the whole turn rather than only file-transcription speed. A fast offline benchmark may not capture the delays users hear.

What is barge-in?

It is the user's ability to interrupt while the assistant speaks. Implementing it requires detecting the interruption, stopping playback and cancelling or ignoring stale work. Without turn identifiers and cancellation handling, the previous answer can continue after the user has moved on.

Should a voice agent act on every recognised request?

No. Recognition can be wrong and background speech may be unintended. Validate intent, permissions and important parameters before actions. Use confirmation appropriate to the consequence and show an accessible review path for details that are difficult to verify accurately by listening alone.

About this guide

Aniket Thakur prepared this guide from secondary research checked on 22 October 2026. No first-hand software testing or performance benchmarking was conducted. Examples are instructional and require validation in your environment.