Speech AI on Azure: Speech to Text, Text to Speech and Spoken Prompts (AI-901)

Speech is how many people prefer to talk to AI: voice assistants, live captions, call centres and accessible apps. AI-901 covers speech concepts (recognition and synthesis) and two implementation skills: responding to spoken prompts with a multimodal model, and building an app with Azure Speech in Foundry Tools.

Speech recognition (speech to text)

Turns spoken audio into text.

• Real-time transcription: text appears as someone speaks, used for live captions and voice commands.
• Batch transcription: transcribe large numbers of recorded files asynchronously.
• Fast transcription: quick, synchronous transcription of a single audio file.
• Diarization: labels different speakers ("Speaker 1", "Speaker 2").
• Custom speech: adapt recognition to your vocabulary (product names, medical terms) or audio conditions.
• Language identification: detects which language is being spoken.
• Pronunciation assessment: scores how accurately someone speaks, used in language-learning apps.

Recognition is hard because of accents, background noise, overlapping speakers and homophones (their / there). Models combine an acoustic model (sounds → phonemes) with a language model (phonemes → likely words).

Speech synthesis (text to speech)

Turns text into natural-sounding audio.

• Neural voices: hundreds of lifelike voices in many languages, some with speaking styles (cheerful, empathetic, newscast).
• SSML (Speech Synthesis Markup Language): XML that controls voice, speed, pitch, pauses, emphasis and pronunciation.
• Custom voice: a voice built for a brand. Because of deepfake risks, it's a limited-access feature with consent requirements.
• Visemes: mouth positions that sync avatars or animations with speech.

<speak version="1.0" xml:lang="en-US">
  <voice name="en-US-AvaMultilingualNeural">
    Your order has shipped. <break time="500ms"/>
    <prosody rate="-10%">It will arrive on Friday.</prosody>
  </voice>
</speak>

Speech translation

Recognises speech in one language and returns text, or synthesized speech, in another, in near real time. Useful for multilingual meetings and travel apps.

Build an app with Azure Speech in Foundry Tools

Install the Speech SDK:

pip install azure-cognitiveservices-speech

You need your resource's key and region (or endpoint). Test your microphone first with our mic test.

Speech to text from the microphone

import os
import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(subscription=os.environ["SPEECH_KEY"], region=os.environ["SPEECH_REGION"])
speech_config.speech_recognition_language = "en-US"

audio_config = speechsdk.audio.AudioConfig(use_default_microphone=True)
recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config)

print("Say something...")
result = recognizer.recognize_once_async().get()

if result.reason == speechsdk.ResultReason.RecognizedSpeech:
    print("You said:", result.text)
elif result.reason == speechsdk.ResultReason.NoMatch:
    print("Speech couldn't be recognised.")

To transcribe a file instead, use AudioConfig(filename="call.wav"). For long audio, use continuous recognition (start_continuous_recognition) with event handlers rather than recognize_once.

Text to speech to the speaker

speech_config.speech_synthesis_voice_name = "en-US-AvaMultilingualNeural"
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config)

result = synthesizer.speak_text_async("Hello! Your appointment is at 3 PM tomorrow.").get()
if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print("Done speaking.")

Use speak_ssml_async(ssml) to send SSML, and AudioOutputConfig(filename="reply.wav") to save audio to a file.

Remember the classes: SpeechConfig (credentials and settings), AudioConfig (input or output device or file), SpeechRecognizer (speech → text), SpeechSynthesizer (text → speech), TranslationRecognizer (speech translation).

Respond to spoken prompts with a multimodal model

Newer audio-capable multimodal models can take audio directly in a prompt, so you don't need a separate transcription step. The model "hears" the question, including tone, and replies in text (and some models can reply in audio too).

import base64

with open("question.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

completion = openai.chat.completions.create(
    model="gpt-audio",  # the name of your audio-capable deployment
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Answer briefly."},
        {"role": "user", "content": [
            {"type": "text", "text": "Answer the question in this recording."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
        ]},
    ],
)
print(completion.choices[0].message.content)

Here openai is the client from project.get_openai_client() (see the chat app lesson). The key idea is the content part with type input_audio, carrying base64-encoded audio. Which audio models are available depends on your region and the current catalog.

For natural back-and-forth voice conversations with very low latency, realtime audio models stream audio both ways, and Azure Speech offers a Voice Live API for building voice agents.

Two ways to build a voice assistant

Key takeaways

• Recognition = speech → text; synthesis = text → speech. Know diarization, custom speech, SSML, neural and custom voices, translation.
• Speech SDK: SpeechConfig, AudioConfig, SpeechRecognizer.recognize_once_async(), SpeechSynthesizer.speak_text_async().
• Spoken prompts to a multimodal model use an input_audio content part with base64 audio.

Next lesson: computer vision and image generation.