Retrieval Pipelines in Azure AI Search: Indexers, Skillsets, OCR and Vector Search (AI-103)

RAG and agent knowledge are only as good as the index behind them. This lesson covers the Build retrieval and grounding pipelines skill group: getting documents, images, audio and video into Azure AI Search, enriching them, and choosing the right search type.

Azure AI Search building blocks

Two ways to load data:

• Pull: an indexer reads the data source, which is the easiest way to keep the index in sync
• Push: your code uploads documents through the SDK or REST API, for full control or unsupported sources

A RAG ingestion flow

For a folder of PDFs, Word files and scans in Blob Storage:

1. Crack documents: the indexer extracts text and images
2. OCR / layout: the OCR skill reads text from images and scanned pages; the Document Layout skill (powered by Document Intelligence) returns structure such as headings, tables and paragraphs, as Markdown
3. Chunk: the Text Split skill splits pages into passages, or chunk by layout section
4. Embed: the Azure OpenAI Embedding skill creates a vector per chunk
5. Project: index projections write each chunk as its own search document, linked to its parent file
6. Index the chunks with metadata: title, URL, page, date, permissions

The Import data (new) wizard in the portal builds this whole pipeline (data source, skillset, index, indexer, vectorizer) in a few clicks, which is a fast way to see the JSON you'd put in source control.

Built-in and custom skills

Built-in skills cover common enrichment:

• Text: language detection, key phrases, entity recognition, PII detection, translation, text split
• Images: OCR, image analysis (captions, tags)
• Layout: Document Layout
• Vectors: Azure OpenAI embeddings, and multimodal embeddings for images

Custom skills run your own logic:

• Custom Web API skill: calls your endpoint (often an Azure Function) with a defined input/output JSON contract, for example to call an internal classifier or clean text
• Azure Machine Learning skill: calls a model hosted in Azure ML
• GenAI prompt skill: calls a chat model during indexing to summarise, classify or describe images, where available

Images, audio and video

• Images: OCR the text, generate captions or verbalisations with a multimodal model, and/or create multimodal embeddings so text queries can find images
• Audio and video: transcribe and segment first, using Content Understanding (transcripts, segments, descriptions) or Speech batch transcription, then index the text and timestamps so answers can link to the exact moment
• Scanned documents: OCR or layout is essential; without it, the index stays empty for those files

Choosing search types for grounding

Hybrid + semantic ranker is the usual default for RAG. Add filters for metadata and security trimming, and scoring profiles to boost fresh or authoritative content.

Connect retrieval to workflows and agents

• Code: call the search client in your app, as in the RAG lesson
• Agents: add an Azure AI Search tool (via a project connection) or a Foundry IQ knowledge base, so the agent retrieves and cites on its own
• Workflows: a retrieval step feeds the next agent or model step
• MCP: expose search as an MCP tool for any MCP-capable agent

Keep it healthy

Schedule indexers, watch failed and warning documents, re-index after changing skills or chunking, and keep a set of test queries to measure relevance after every change (see monitoring).

Key takeaways

• Pipeline: data source → indexer → skillset (OCR, layout, split, embed, custom) → index projections → index, plus a vectorizer for queries.
• Integrated vectorization chunks and embeds during indexing and vectorizes queries.
• Custom logic = Custom Web API skill; scanned files need OCR / layout.
• Ground with hybrid + semantic ranker, filters for security, and agent tools or Foundry IQ for agents.

Next lesson: document extraction with Content Understanding.