Optimize and Operate Generative AI: Prompt Tuning, Reflection and Tracing (AI-103)

Once an app works, you make it better, cheaper and observable. This lesson covers the Optimize and operationalize generative AI systems skill group: tuning generation, reflection loops, tracing, and orchestrating several models with rules.

Tune generation behaviour

Prompt engineering is still the cheapest lever:

• Clear instructions: role, task, audience, format, length
• Context: the data the model needs, clearly delimited
• Examples (few-shot): two or three input → output pairs show the style you want
• Constraints: "answer only from the sources", "if unsure, say so"
• Output format: structured outputs with a JSON schema when code reads the result
• Versioning: store prompts in source control and evaluate each version

Model parameters:

Reasoning models often ignore or reject temperature; tune them with instructions and reasoning effort instead. If prompting can't reach the quality you need, consider fine-tuning a model on examples, or distilling a large model's behaviour into a smaller, cheaper one.

Reflection and self-critique

Instead of returning the first draft, let the model check its own work:

draft = openai.responses.create(model="chat-main", input=task).output_text

critique = openai.responses.create(
    model="chat-main",
    instructions="You are a strict reviewer. List factual errors, missing requirements and unsupported claims. Reply OK if there are none.",
    input=f"Task:\n{task}\n\nDraft:\n{draft}",
).output_text

if critique.strip() != "OK":
    draft = openai.responses.create(
        model="chat-main",
        input=f"Task:\n{task}\n\nDraft:\n{draft}\n\nReviewer feedback:\n{critique}\n\nRewrite the draft fixing every issue.",
    ).output_text

Variants:

• Self-critique loop: generate → critique against a rubric → revise, with a maximum number of rounds
• Separate reviewer model: a different (sometimes stronger) model critiques
• Chain-of-thought evaluation: ask the judge to reason step by step before scoring, which makes evaluation scores more reliable and explainable. Reasoning models do this internally.
• Verification with tools: check facts against search or run generated code before answering

Reflection improves quality but multiplies tokens and latency, so use it where accuracy matters most.

Observability: tracing, tokens, safety and latency

Foundry uses OpenTelemetry. Connect an Application Insights resource to the project, then instrument your app:

from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor

configure_azure_monitor(connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
OpenAIInstrumentor().instrument()   # every model call becomes a span with model, tokens and latency

Agents hosted in Foundry Agent Service and workflows built with Agent Framework emit traces automatically. In the Foundry portal's Tracing view and in Application Insights you get:

• Spans for each step: retrieval, model call, tool call, agent handoff
• Token analytics: input, output and cached tokens per call, per feature, per user, so you know where the money goes
• Latency breakdowns: time to first token, model time vs retrieval time vs tool time
• Safety signals: content filter results and evaluation scores attached to traces

Be careful with prompt and response content in traces. It's useful for debugging but may contain personal data, so control what's recorded and who can read it.

Orchestrating multiple models and rules

Real systems combine models and plain code:

• Model routing: send simple requests to a small model and complex ones to a large or reasoning model, using rules, a cheap classifier model or Foundry's model router deployment
• Cascades / fallback: try the cheap model first; if confidence or validation fails, escalate to a stronger model
• Hybrid LLM + rules engine: the LLM extracts intent and fields, deterministic rules make the decision (eligibility, pricing, compliance), and the LLM explains the outcome. Rules give predictability and auditability where the business needs it.
• Specialised models per step: embeddings for retrieval, an SLM for classification, an LLM for the answer, a speech model for voice

Key takeaways

• Tune with instructions, examples, structured outputs, and parameters (temperature, top_p, max tokens, reasoning effort). Fine-tune only when prompting isn't enough.
• Reflection: generate → critique → revise, with round limits; chain-of-thought makes judges more reliable.
• Observability = OpenTelemetry → Application Insights: spans, token analytics, latency breakdowns, safety signals.
• Combine models with routing, cascades and rules engines for cost and predictability.

Next lesson: image and video generation.