Multimodal Understanding: Captions, Visual Q&A, Alt Text and Video Analysis (AI-103)

On AI-103, "computer vision" mostly means multimodal models and Content Understanding, not hand-trained classifiers. This lesson covers analysing images and video, captions and alt text, visual question answering, object and region detection, and keeping visual workflows safe.

Analyse images with a multimodal model

Send images as input_image content parts next to your instructions:

def describe(image_url: str, detail: str = "concise") -> str:
    r = openai.responses.create(
        model="vision-main",
        instructions=(
            "Describe images for an online store. "
            + ("One sentence, max 20 words." if detail == "concise" else "A detailed paragraph covering product, colour, material and setting.")
        ),
        input=[{"role": "user", "content": [
            {"type": "input_text", "text": "Describe this image."},
            {"type": "input_image", "image_url": image_url},
        ]}],
    )
    return r.output_text

• Concise vs detailed captions come from the instructions (and max_output_tokens).
• Multiple images: add several input_image parts and ask for a caption per image, a comparison ("which photo shows damage?") or a combined summary.
• Structured analysis: use structured outputs to return fields such as {"objects": [...], "dominant_colour": "...", "contains_text": true}.
• Some models accept an image detail level: low detail is cheaper, high detail reads small text better.

Visual question answering grounded in evidence

For questions about an image ("Is the safety helmet being worn?", "What's the total on this receipt?"), instruct the model to:

• Answer only from what is visible
• Point to the evidence ("the receipt's bottom line shows…")
• Say "can't determine from the image" when the answer isn't visible, instead of guessing

That's grounding, the same idea as RAG, with pixels as the source.

Alt text and extended descriptions

Accessibility is part of inclusiveness, and the exam names it explicitly:

• Alt text: short (often under 125 characters), describes the image's purpose in context, and doesn't start with "image of". Decorative images get empty alt text.
• Extended description: for complex images such as charts, diagrams and maps, a longer description that conveys the data or structure, linked or placed near the image
• Give the model the surrounding page context, because the same photo needs different alt text on a product page and in a news story

Have a person review alt text for important content, since models can misidentify people or objects.

Content Understanding for images and video

Azure Content Understanding in Foundry Tools turns images, documents, audio and video into structured fields using analyzers (see the document extraction lesson for the full model).

For visual content:

• Image analyzers extract visual characteristics into a schema: product type, colour, defects, damage severity, brand present. Fields use extract, generate or classify.
• Video analysis splits video into segments (shots or scenes), with key frames, a transcript, timestamps and per-segment fields such as "scene description" or "safety violation present", which is ideal for search, compliance review and chaptering.
• Single-task vs pro mode: standard (single-task) analyzers process one file with one schema. Pro mode runs a multi-step reasoning pipeline that can combine multiple input files and reference data (for example, compare an invoice with a purchase order and a contract) for complex validation. Pro mode is slower and costs more, so use it only when cross-document reasoning is needed.

Objects, components and regions

To locate things, not just describe them:

• Multimodal models can list objects and, in some cases, approximate positions. Verify them before relying on exact coordinates.
• Content Understanding returns grounding, such as the region or time range a value came from.
• For precise, repeatable bounding boxes at scale (counting items on shelves, finding defects on a production line), use a dedicated detection model, such as a custom vision model deployed from the catalog or trained in Azure Machine Learning.

Responsible AI for visual content

Filter unsafe images. Azure AI Content Safety image analysis classifies hate, sexual, violence and self-harm content with severity levels. Run it on uploads and on generated images, and block or send to review above your threshold.

Defend against prompt injection in images. An image can contain written instructions ("ignore your rules and…"). Multimodal models read that text, so:

• Treat text in images as data, not instructions: say so in the system prompt
• Keep Prompt Shields (indirect attack detection) on
• Limit what a vision agent can do (least-privilege tools, approvals), so a successful injection can't cause harm

Enforce visual policy rules before publishing:

• Watermark or add content credentials to generated images
• Flag prohibited symbols and inappropriate content with classification
• Brand rules: check logo usage, colours and competitor marks with a classifier, a multimodal check against a rubric, or a Content Understanding analyzer
• Route anything uncertain to human review

Key takeaways

• Multimodal models + input_image handle captions (concise or detailed, single or multiple), structured analysis and grounded visual Q&A.
• Alt text: short, purposeful, context-aware; extended descriptions for charts and diagrams.
• Content Understanding extracts visual fields and analyses video segments; pro mode reasons across multiple files.
• Safety: Content Safety image filters, defend against injected text in images, enforce watermark and brand policies.

Next lesson: text analysis and translation.