Computer Vision and Image Generation in Microsoft Foundry (AI-901)

Computer vision lets software understand images and video; image generation lets it create them. AI-901 asks you to identify vision capabilities, and to interpret visual input with a multimodal model, create new visual outputs with generative models, and build a lightweight vision app.

How computers see

To a computer, an image is a grid of pixels, each made of numbers (for example red, green and blue values from 0 to 255). Vision models learn patterns in these numbers.

• Convolutional neural networks (CNNs) apply small filters across the image to detect edges, shapes and then whole objects. They powered most classic vision.
• Vision transformers split an image into patches and use attention, like language models do with tokens.
• Multimodal models combine a vision encoder with a language model, so they can talk about images.

Core computer vision tasks

Responsible AI note: facial recognition (identifying or verifying a specific person) is a Limited Access feature at Microsoft. Emotion and other sensitive inferences were retired from face analysis for responsible AI reasons.

Interpret visual input with a multimodal model

With a vision-capable model deployed in Foundry, you can send an image with a question:

import base64

with open("receipt.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = openai.responses.create(
    model="gpt-5-mini",  # a vision-capable deployment
    input=[{
        "role": "user",
        "content": [
            {"type": "input_text", "text": "What is the total on this receipt, and which shop is it from?"},
            {"type": "input_image", "image_url": f"data:image/jpeg;base64,{image_b64}"},
        ],
    }],
)
print(response.output_text)

Key points:

• The message's content is a list of parts: input_text and input_image.
• The image can be a public URL or a base64 data URL (our image to Base64 tool shows what one looks like).
• Large images cost more tokens. Resizing or compressing them first saves money and time.

Multimodal models are great for flexible questions: "Is this parking sign saying I can park now?", "Describe this chart", "What's damaged in this photo?". They can make mistakes with small text, counting and precise positions, so for exact OCR or bounding boxes, specialised services can be more reliable.

Create new visual outputs

Image-generation models create images from a text prompt and can edit existing images. In Foundry the catalog includes models such as gpt-image-1, earlier DALL-E models and partner models such as FLUX.

import base64

result = openai.images.generate(
    model="gpt-image-1",   # your image-generation deployment
    prompt="A watercolour illustration of a robot reading a book in a library",
    size="1024x1024",
)
with open("robot.png", "wb") as f:
    f.write(base64.b64decode(result.data[0].b64_json))

Prompt tips for images: describe the subject, style (photo, watercolour, flat icon), composition (close-up, wide shot), lighting and colours. Content filters block harmful requests, and some models add provenance metadata (C2PA content credentials) to show the image is AI-generated, which supports transparency.

Many image generators use diffusion: starting from noise and removing it step by step, guided by the text prompt. Video generation models (such as Sora) extend this to moving images.

Build a lightweight vision app

A practical app often combines steps. Here's a simple "describe my photo" tool:

import base64, sys
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient

project = AIProjectClient(
    endpoint="https://<resource-name>.services.ai.azure.com/api/projects/<project-name>",
    credential=DefaultAzureCredential(),
)
openai = project.get_openai_client()

path = sys.argv[1]
with open(path, "rb") as f:
    data_url = "data:image/jpeg;base64," + base64.b64encode(f.read()).decode()

response = openai.responses.create(
    model="gpt-5-mini",
    instructions="Describe images for blind and low-vision users: main subject first, then important details, in under 60 words.",
    input=[{"role": "user", "content": [
        {"type": "input_text", "text": "Describe this image."},
        {"type": "input_image", "image_url": data_url},
    ]}],
)
print(response.output_text)

Add text to speech and you have an accessibility app: a clear example of the inclusiveness principle.

Azure also offers Azure Vision (image analysis with captions, tags, object detection, people detection and the Read OCR feature) and Azure Content Understanding for structured extraction from images, covered in the next lesson.

Key takeaways

• Know the tasks: classification (whole image), detection (boxes), segmentation (pixels), OCR (text), captioning, face detection.
• Send images to multimodal models as input_image content parts (URL or base64 data URL) next to input_text.
• Generate images with models like gpt-image-1 through images.generate; many use diffusion.
• Facial recognition is Limited Access; label AI-generated images for transparency.

Next lesson: information extraction with Content Understanding.