Implement RAG With Azure AI Search and Foundry (AI-103)

Language models only know what they were trained on. Retrieval-augmented generation (RAG) gives them your own, current data at question time, without retraining. It's the most common pattern on AI-103, and it shows up again in agents, evaluation and information extraction.

The RAG pattern

Ingestion (offline):

1. Load documents (PDFs, Word, web pages, tickets)
2. Chunk them into passages, for example 500–1,000 tokens with some overlap
3. Embed each chunk with an embedding model
4. Index text, vectors and metadata (title, URL, date, permissions) in Azure AI Search

Query time (online):

1. Retrieve the most relevant chunks for the user's question
2. Augment the prompt with those chunks and grounding instructions
3. Generate an answer with citations

Retrieval: hybrid plus semantic ranking

Azure AI Search supports several query types, and the best RAG results usually come from combining them:

• Keyword (BM25): exact words, codes and names
• Vector: similar meaning, even with different words
• Hybrid: both, merged with Reciprocal Rank Fusion (RRF)
• Semantic ranker: re-scores the top results with a language model and can return captions

import os
from azure.identity import DefaultAzureCredential
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery

search = SearchClient(os.environ["SEARCH_ENDPOINT"], "policies", DefaultAzureCredential())

def retrieve(question: str, k: int = 5):
    results = search.search(
        search_text=question,                       # keyword part
        vector_queries=[VectorizableTextQuery(text=question, k_nearest_neighbors=50, fields="contentVector")],
        query_type="semantic",
        semantic_configuration_name="default",
        top=k,
        select=["title", "url", "content"],
    )
    return list(results)

VectorizableTextQuery lets the index's vectorizer embed the question for you (integrated vectorization), so the app doesn't call the embedding model itself.

Ground the prompt

def answer(question: str) -> str:
    docs = retrieve(question)
    sources = "\n\n".join(f"[{i+1}] {d['title']} ({d['url']})\n{d['content']}" for i, d in enumerate(docs))
    r = openai.responses.create(
        model="chat-main",
        instructions=(
            "Answer ONLY from the sources below. Cite sources as [1], [2]. "
            "If the answer isn't in the sources, say you don't know."
        ),
        input=f"Sources:\n{sources}\n\nQuestion: {question}",
    )
    return r.output_text

The three grounding rules to remember: use only the sources, cite them, and admit when the answer isn't there.

Security trimming

If documents have different audiences, store an allowed-groups field in each chunk and filter at query time (filter="group_ids/any(g: search.in(g, '...'))"), so users only retrieve what they're allowed to read. Never rely on the prompt to hide content.

RAG for agents: knowledge tools and Foundry IQ

Agents can do RAG without you writing retrieval code:

• Azure AI Search tool: connect an index to an agent; the agent decides when to search and cites results
• File search tool: upload files, and the service chunks, embeds and stores them in a vector store
• Foundry IQ knowledge bases: a managed knowledge layer on Azure AI Search that combines multiple sources and uses agentic retrieval. The model breaks a complex question into sub-queries, runs them in parallel and merges the results.

Choose code-based RAG for full control, and agent knowledge tools for speed and multi-turn conversations.

Common RAG problems and fixes

Key takeaways

• RAG = chunk → embed → index, then retrieve → augment → generate with citations.
• Hybrid search + semantic ranker is the usual retrieval choice in Azure AI Search.
• Ground with only-the-sources, cite, say I don't know, and filter by permissions.
• Agents get RAG via the Azure AI Search tool, file search or Foundry IQ (agentic retrieval).

Next lesson: evaluate generative AI apps.