Azure Content Understanding: Extract Data From Documents, Images, Audio and Video (AI-901)

Businesses are full of content that people read and retype: invoices, contracts, receipts, ID cards, call recordings, training videos. Azure Content Understanding in Foundry Tools turns that content into structured data your apps can use. It has its own skill group on AI-901, covering documents and forms, images, audio and video, and a lightweight extraction app.

What Content Understanding does

Content Understanding is a multimodal service. You give it a file and an analyzer, and it returns:

• The content itself, such as document text as Markdown, or a video transcript
• Fields you asked for, such as InvoiceTotal or CustomerName, with confidence scores and grounding (where in the source each value came from)

Behind the scenes it combines OCR, layout analysis, speech transcription and generative models deployed in your Foundry resource. Because generative models are involved, it can do more than copy text: it can classify content and generate values like a summary or sentiment.

Key concepts

Analyzer: the configuration that says how to process content and what to return. Use a prebuilt analyzer as-is, or create a custom one.

Field schema: the list of named, typed fields an analyzer returns. Types include string, number, date, boolean, object (nested fields) and array (repeated items such as invoice line items).

Field methods: how each value is produced:

• Extract: the value appears in the content (an invoice number)
• Generate: the value is created from the content (a one-line summary of a call)
• Classify: the value is picked from a list of categories (document type: invoice, receipt or contract)

Confidence and grounding: each field can include a confidence score and its source location (page and bounding box for documents, time range for audio and video). Use these to send low-confidence values to a person for review, a human-in-the-loop pattern that supports reliability and safety.

Model deployments: analyzers call generative and embedding models in your Foundry resource, so you deploy supported models and set them as the resource's defaults before analysing.

By modality

Prebuilt analyzers include document-type analyzers such as prebuilt-invoice, and general analyzers for each modality, such as prebuilt-imageSearch, prebuilt-audioSearch and prebuilt-videoSearch, which produce descriptions, transcripts and segments that are ready for search and RAG. The full list changes over time, so check the documentation for the current set.

A custom analyzer, conceptually

Suppose you process car insurance claims. A custom analyzer for photos could define:

{
  "fieldSchema": {
    "fields": {
      "VehicleType": { "type": "string", "method": "classify", "enum": ["car", "van", "motorbike"] },
      "DamagedParts": { "type": "array", "method": "generate", "items": { "type": "string" } },
      "Severity": { "type": "string", "method": "classify", "enum": ["minor", "moderate", "severe"] },
      "LicensePlate": { "type": "string", "method": "extract" }
    }
  }
}

You can design and test schemas visually in the Foundry portal (or Content Understanding Studio) using sample files, then save the analyzer and call it from code.

Build a lightweight extraction app

Install the SDK and set your endpoint and key as environment variables:

pip install --pre azure-ai-contentunderstanding

import os
from azure.ai.contentunderstanding import ContentUnderstandingClient
from azure.ai.contentunderstanding.models import AnalysisInput
from azure.core.credentials import AzureKeyCredential

client = ContentUnderstandingClient(
    endpoint=os.environ["CONTENTUNDERSTANDING_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["CONTENTUNDERSTANDING_KEY"]),
)

poller = client.begin_analyze(
    analyzer_id="prebuilt-invoice",
    inputs=[AnalysisInput(url="https://example.com/invoice.pdf")],
)
result = poller.result()

for content in result.contents:
    for name, field in (content.fields or {}).items():
        print(name, "→", field.value, f"(confidence {field.confidence})")

The pattern to remember:

1. Create a client with the endpoint and a credential.
2. Call begin_analyze with an analyzer ID and the input (a URL or file data).
3. Analysis is asynchronous, so the SDK returns a poller; result() waits until it finishes.
4. Read the fields (and confidence scores) from the result.

With the REST API you'd send POST .../contentunderstanding/analyzers/prebuilt-invoice:analyze, receive 202 Accepted with an Operation-Location header, and poll that URL until the status is Succeeded.

The SDK is evolving (note the --pre flag), so check the current samples if a name has changed. The concepts are what the exam tests.

Want sample invoices to test with? Make some with our invoice generator.

Content Understanding vs other options

Key takeaways

• Content Understanding turns documents, images, audio and video into structured fields.
• An analyzer (prebuilt or custom) defines processing and a field schema; fields use extract, generate or classify.
• Results include confidence and grounding, so you can route uncertain values to a human.
• Code: client → begin_analyze(analyzer_id, inputs) → poll → read fields. REST: 202 + Operation-Location, then poll.

Next lesson: the AI-901 cheat sheet.