Skip to content
Blog

What It Takes to Ship AI Voice Agents to Production

Mukhammad Ali Toshboev5 min read

Everyone has seen the demos. An AI voice agent having a smooth conversation, booking appointments, answering questions. It looks easy.

Then you try to ship one that handles real customer calls with interruptions, background noise, confused callers and awkward silences. That's where things get interesting.

I built an AI voice agent for real-time customer service calls. Here's what the journey from demo to production actually looks like.

How the Pipeline Works

A real-time voice AI system is basically a chain of services that need to work together very fast:

Caller → Telephony → Speech-to-Text → LLM → Text-to-Speech → Caller
           ↕                                      ↕
      Call Management                        Tool Execution
      (hold, transfer)                    (CRM, scheduling, payments)

Every hop in that chain adds latency. Humans notice conversational delays around 300 to 500 milliseconds. Go past that and the conversation starts feeling weird.

To keep things fast I used streaming everywhere. The text-to-speech engine starts generating audio before the LLM finishes its full response. The speech-to-text runs in real time with endpoint detection so it knows when someone stops talking. The LLM streams tokens as they're generated.

Interruptions Are the Hardest Problem

In normal conversation people interrupt each other all the time. "Yeah I know but" or "Wait actually" are totally normal. Most voice AI demos just ignore this.

Handling it properly means tracking two audio streams at once:

class InterruptionDetector {
  private isAgentSpeaking = false;
  private userSpeechBuffer: Float32Array[] = [];
  private silenceThreshold = 0.02;
  private interruptionThreshold = 150; // ms of sustained user speech

  detectInterruption(audioFrame: Float32Array): boolean {
    const energy = this.calculateEnergy(audioFrame);

    if (this.isAgentSpeaking && energy > this.silenceThreshold) {
      this.userSpeechBuffer.push(audioFrame);

      const speechDuration = this.userSpeechBuffer.length * this.frameDuration;
      if (speechDuration > this.interruptionThreshold) {
        return true;
      }
    }

    return false;
  }
}

When an interruption is detected you need to stop the current audio stream, flush the buffer, feed the caller's words back to the LLM as context and generate a new response. All within 300 milliseconds.

The tricky part is distinguishing a real interruption from a cough or a quick "uh huh." That's why there's a threshold for sustained speech. Without it the agent would stop talking every time someone breathed.

The Agent Needs to Do Things Not Just Talk

During a live call the agent looks up addresses, checks appointment availability, books meetings and updates the CRM. Every one of those API calls adds latency. So the agent needs to fill the gap naturally:

const tools = [
  {
    name: "check_availability",
    handler: async (params) => {
      await streamFiller("checking_availability");
      const slots = await calendar.getAvailableSlots(params.date);
      return formatSlots(slots);
    }
  },
  {
    name: "schedule_appointment",
    handler: async (params) => {
      await streamFiller("scheduling");
      const booking = await calendar.createBooking(params);
      await crm.updateContact(params.contactId, {
        nextAppointment: booking.datetime
      });
      return booking;
    }
  }
];

That streamFiller function is doing more work than it looks like. While the API call runs in the background the agent plays a natural sounding response like "Let me check on that for you" or "One moment please." Without it there's a one to two second silence that makes people think the call dropped.

Edge Cases Nobody Warns You About

The hello loop. If the agent takes too long to respond at the start of a call, callers say "hello" over and over. Each "hello" triggers a new speech-to-text result which interrupts the agent's greeting which makes the caller say "hello" again. Fun times.

Background noise. Someone calling from a car or a restaurant generates constant audio that the speech-to-text engine interprets as words. I added energy-based voice activity detection before the STT layer to filter out ambient noise.

Accents. Speech-to-text accuracy drops noticeably for non-standard accents. I tuned the system prompt to handle common misrecognitions and added phonetic hints for industry-specific terminology.

Silence. Sometimes people go quiet because they're thinking or looking something up. The agent needs to tell the difference between "I'm thinking" and "I hung up." After five seconds of silence the agent asks if they're still there. After fifteen it ends the call gracefully.

What to Measure

Voice agent metrics are different from chatbot metrics. Here's what I tracked:

  • First response time. Target was under 800ms from the end of caller speech to the start of agent speech.
  • Interruption recovery. How fast the agent adapts when someone cuts in. Target was under 300ms.
  • Task completion rate. What percentage of calls ended with the intended action completed.
  • Escalation rate. What percentage got transferred to a human. You want this to be low but not zero. Zero means you're handling calls you shouldn't be.

After three months in production the agent handled roughly 60% of routine calls without any human involvement. That took a real load off the support team.

What I Took Away From This

Latency matters more than accuracy. People will forgive a slightly imperfect answer way faster than they'll forgive a slow one.

Natural filler responses make a huge difference. A simple "hmm let me check" makes an AI agent feel surprisingly human.

Build for failure from the start. Every external API call will fail eventually. You need graceful fallbacks at every step.

And test with real calls not scripts. Lab conditions don't prepare you for someone calling from a speakerphone in a moving car while their kids are yelling in the background.

The voice AI space is moving incredibly fast. What took months to build from scratch a year ago can now be prototyped in days. But understanding the underlying architecture still matters when you need something that works reliably at scale.

Working on voice AI? I'd love to hear about your experience. Connect with me on LinkedIn.

Discuss on TwitterSuggest a change