AI-103 courseLesson 4 of 18 · 14 min
Manage, Monitor and Secure Azure AI Solutions: Quotas, Identity and Networking (AI-103)
XUtilHub Team ·
Exam skills covered
- Manage quotas, scaling, rate limits, and cost footprints for model and agent workloads
- Monitor model performance, drift, safety events, and grounding quality
- Monitor data ingestion quality, search index health, and relevance performance
- Configure security, including managed identity, private networking, keyless credentials, and role policies
Getting an AI app working is half the job. Keeping it affordable, healthy and secure is the other half, and it's a big part of the 25–30% "Plan and manage" domain. This lesson covers quotas and cost, monitoring models and search, and security.
Quotas, rate limits and scaling
Standard deployments have quota measured in tokens per minute (TPM), with a matching requests per minute (RPM) limit. Quota is assigned per subscription, per region, per model, and you split it across deployments.
When you exceed it, the service returns HTTP 429 Too Many Requests, with a retry-after header. Ways to handle it:
- Retry with exponential backoff (the SDKs do this automatically for a few attempts)
- Reduce tokens: shorter prompts, smaller retrieved context, a lower
max_output_tokens - Request more quota, or spread load across deployments or regions
- Use Global deployment types, which have higher default quota
- Put Azure API Management in front as an AI gateway for load balancing, token limits per client and failover
- Move steady traffic to Provisioned, and offline traffic to Batch
Cost footprint
Token-based costs grow quietly. Levers:
| Lever | Effect |
|---|---|
| Smaller or cheaper models for easy requests (routing) | Large savings |
| Prompt caching (repeated prompt prefixes) | Cheaper input tokens, lower latency |
| Fewer, better retrieved chunks | Fewer input tokens |
| Output limits and concise instructions | Fewer output tokens |
| Batch for offline jobs | Discounted tokens |
| Budgets and alerts in Cost Management | No surprise bills |
Agents can call models many times per user request, so watch tokens per task, not just per call.
Monitoring models and agents
Foundry and Azure Monitor give you:
- Operational metrics: requests, latency, time to first token, tokens in and out, 429s, errors
- Safety events: how often content filters blocked prompts or responses, and which categories
- Quality over time: run continuous evaluation on a sample of production traffic to catch drift in groundedness and relevance. Answers can degrade when data, prompts or model versions change.
- Traces: each request's steps, tool calls, tokens and latency (see the observability lesson)
Connect Application Insights to the Foundry project so traces and evaluations land in one place, and set alerts, for example when the error rate or 429 count spikes.
Monitoring ingestion and search health
RAG quality depends on the index. Monitor:
- Indexer status: run history, failed and warning documents (corrupt PDFs, files too large, skill errors)
- Index statistics: document count and storage. A sudden drop means content went missing.
- Query metrics: latency, throttling, queries per second
- Relevance: test queries with known good results; track hit rate and groundedness of answers
Many "the bot gives wrong answers" incidents are really ingestion failures.
Security
Identity, keyless. Prefer Microsoft Entra ID over API keys:
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
project = AIProjectClient(endpoint=endpoint, credential=DefaultAzureCredential())
DefaultAzureCredential uses your developer login locally and the app's managed identity in Azure. Then disable local (key) authentication on the resource.
Roles (RBAC). Assign the smallest role that works, at the narrowest scope:
| Role | Lets you |
|---|---|
| Azure AI User | Use a Foundry project: call models, run agents |
| Azure AI Project Manager | Manage projects and assign the AI User role |
| Azure AI Account Owner | Manage the Foundry resource |
| Cognitive Services OpenAI User | Call OpenAI model deployments only |
| Search Index Data Reader | Query an Azure AI Search index |
Agents also act as an identity when they call tools, so give each agent its own identity and only the access its tools need.
Network. For private solutions:
- Add private endpoints for Foundry, AI Search, Storage and Key Vault, with private DNS zones
- Set public network access to disabled
- Use VNet integration for the app so it reaches the private endpoints
- Allow trusted service-to-service access (for example AI Search indexing from Storage) through managed identities and shared private links
Secrets. Anything that must stay a secret (a third-party API key for a tool) goes in Key Vault, read with a managed identity.
Key takeaways
- 429 = rate limit: back off, trim tokens, add quota or deployments, use an API Management gateway, or move to Provisioned or Batch.
- Control cost with routing to smaller models, caching, fewer tokens, Batch and budgets.
- Monitor latency, tokens, errors, safety events, quality drift, and indexer and index health.
- Secure with managed identity + RBAC + DefaultAzureCredential, private endpoints, and Key Vault; disable key auth.
Next lesson: responsible AI guardrails and governance.
Check your understanding
Exam-style questions on this lesson. Get them all right to mark the lesson complete.
Finished reading?