Speech for AI Agents: Speech to Text, Text to Speech, Voice Live and Translation (AI-103)

Voice is becoming a normal way to talk to agents: call centres, in-car assistants, kiosks and accessibility features. AI-103 tests speech as an agent modality: converting speech and text, custom speech models, reasoning over audio, and speech translation.

Speech to text and text to speech

Azure Speech (a Foundry Tool) provides:

• Speech to text (STT): real-time recognition from a microphone or stream, fast transcription for files, and batch transcription for large volumes, with diarization (who spoke when) and timestamps
• Text to speech (TTS): hundreds of neural voices in many languages, controlled with SSML
• Speech translation: recognise speech in one language and output text or speech in others

import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(endpoint=os.environ["SPEECH_ENDPOINT"], token_credential=DefaultAzureCredential())
speech_config.speech_recognition_language = "en-US"
speech_config.speech_synthesis_voice_name = "en-US-AvaMultilingualNeural"

recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config)
heard = recognizer.recognize_once_async().get().text            # STT

reply = agent_reply(heard)                                       # your agent call
speechsdk.SpeechSynthesizer(speech_config=speech_config).speak_text_async(reply).get()   # TTS

SSML controls how speech sounds:

<speak version="1.0" xml:lang="en-US" xmlns:mstts="http://www.w3.org/2001/mstts">
  <voice name="en-US-AvaMultilingualNeural">
    <mstts:express-as style="cheerful">Your order has shipped!</mstts:express-as>
    <break time="300ms"/> It should arrive on <say-as interpret-as="date" format="md">11/15</say-as>.
  </voice>
</speak>

The voice agent pipeline

The classic cascaded pipeline:

microphone → STT → agent (LLM + tools) → TTS → speaker

It's flexible (any agent, any voice) but adds latency at each hop. Improve it with streaming at every stage: partial recognition results, streamed model tokens, and TTS that starts speaking the first sentence early.

Real-time voice: Voice Live and audio models

For natural conversations, use speech-to-speech options:

• Voice Live API (Azure Speech): a single real-time WebSocket API that combines recognition, a generative model and synthesis, with turn detection, interruption handling (barge-in), noise suppression and echo cancellation. You can connect it to a Foundry agent.
• Realtime audio models (such as the gpt-realtime family): models that take audio in and produce audio out directly over WebSocket or WebRTC, with function calling for tools

Choose real-time APIs when users expect phone-call-like responsiveness; choose the cascaded pipeline when you need a specific custom STT model, voice or agent behaviour at each step.

Custom speech models

When recognition struggles with domain vocabulary (drug names, product codes) or acoustic conditions (noisy factories, accents, phone audio):

1. Phrase list: add terms at runtime. It's quick, no training needed.
2. Custom speech: train a model with your text (domain sentences) and/or audio + human transcripts, then evaluate it by word error rate (WER) against the base model and deploy a custom endpoint
3. Custom neural voice: a brand voice for TTS, which requires approval and the speaker's consent (a responsible AI control)

Reasoning over audio input

Audio-capable multimodal models can take audio directly, not only transcripts. That means they can use how something was said (hesitation, tone, background noise) as well as what was said. Typical uses:

• Summarise a call and flag frustration from the voice
• Answer questions about a recording ("did the agent confirm the address?")
• Combine audio with documents or images in one request

Content Understanding analyses audio files into transcripts with speakers and timestamps, plus custom fields (call reason, resolution, compliance phrases spoken), which is ideal for call-centre analytics at scale.

Translate speech

Two approaches:

• Speech translation (Foundry Tool): real-time recognition plus translation into one or more target languages, with optional synthesised speech output. Good for live captions and multilingual meetings.
• Language model flow: STT → LLM translation (with tone, glossary and context) → TTS in the target language. Good when phrasing and cultural adaptation matter more than raw speed.

Live interpretation features can preserve aspects of the speaker's voice in the translated speech.

Responsible voice

• Tell users they're talking to an AI voice (transparency)
• Get consent before recording, and set retention for audio and transcripts
• Custom voices need the voice talent's consent and Microsoft approval
• Apply the same content filters and tool approvals as text agents

Key takeaways

• Azure Speech: STT (real-time, fast, batch, diarization), TTS with SSML, and speech translation.
• Voice agents: cascaded STT → agent → TTS with streaming, or real-time Voice Live / realtime audio models with turn detection and barge-in.
• Improve recognition with phrase lists and custom speech (measured by WER).
• Audio models and Content Understanding reason over audio; translate with speech translation or LLM flows.

Next lesson: retrieval pipelines in Azure AI Search.