AI Agent Models and Instructions: Model Selection, System Rules, and Structured Outputs

Core Rule: A prompt can describe policy, but deterministic application code must enforce permissions, schemas, budgets, and approval requirements.

Choose a Model by Workload, Not Reputation

The model is one component of an agent, not the agent itself. It interprets the current task, available context, tool descriptions, and prior observations to propose the next useful action.

Instructions define the operating contract. They explain the agent role, success criteria, boundaries, tool-use rules, response format, and situations that require clarification or human escalation.

Reliable systems select models by task difficulty and risk, keep durable rules separate from user input, and validate structured outputs before downstream code trusts them.

Model selection is an engineering tradeoff among reasoning quality, instruction following, tool-call accuracy, context size, latency, availability, and price. A support classifier and a repository-wide coding agent do not need the same model.

Use evaluation data from your own tasks. Build a representative set of easy, ambiguous, adversarial, and failure cases, then compare candidate models on completion quality, invalid tool calls, latency, and cost.

Use smaller models for routing, extraction, classification, and formatting when they meet the quality target.
Reserve stronger models for ambiguous planning, synthesis, and recovery from difficult tool results.
Define a fallback model or graceful error path for provider failures and rate limits.
Re-run evaluations before changing model versions in production.

Build an Instruction Hierarchy

Durable application rules belong in the system or developer instruction layer. The user message supplies the goal and relevant preferences, but it must not be allowed to rewrite security policy or grant itself tools.

A useful instruction set answers six questions: who is the agent, what outcome should it produce, what context may it trust, which tools may it request, when must it stop, and what output shape must it return?

State the role and objective in concrete language.
Define forbidden actions and escalation conditions.
Describe tools by purpose, inputs, side effects, and limitations.
Specify what to do when evidence is missing or instructions conflict.

Treat Context as Designed Input

Context engineering is the work of assembling the smallest useful set of instructions, state, retrieved evidence, tool descriptions, and conversation history for the next decision. More context is not automatically better.

Long, duplicated, or stale context can bury important rules and increase cost. Summarize old interaction history, retrieve only relevant knowledge, and clearly mark untrusted content such as webpages and uploaded documents.

Put stable policy before transient task data.
Separate trusted instructions from untrusted retrieved text.
Include the observations needed for the current step, not the entire run log.
Measure how context changes affect accuracy and token usage.

Use Structured Outputs at Code Boundaries

Free-form prose is appropriate for a final explanation, but downstream automation needs a typed contract. Define fields, allowed values, required properties, and validation behavior before connecting model output to business logic.

Validation failure should become a normal workflow branch. The runtime can retry once with a concise correction, use a fallback parser, request clarification, or escalate instead of silently guessing.

Validate types and allowed enum values.
Reject unknown tool names and unexpected fields.
Keep the human-readable answer separate from machine action fields.
Log validation errors without exposing sensitive prompt content.

Version and Evaluate Instructions

Instructions are production code. Give them versions, review changes, and connect each release to an evaluation result. A small wording change can alter tool choice, refusal behavior, or output formatting.

Test instructions against normal tasks, underspecified requests, conflicting user instructions, prompt-injection attempts, missing evidence, tool failures, and requests outside the agent role.

Model and Instruction Design in the Right Order

Model selection should come after task design. First define the job, success criteria, allowed tools, context sources, output format, and risk level. Then choose the smallest reliable model for each step. A classification step, a retrieval query rewrite, and a legal-risk explanation may need different model routes because they have different cost, latency, and reliability requirements.

Instructions should describe role, goal, boundaries, tool-use rules, evidence rules, and stop behavior. Avoid writing instructions as motivational slogans. A good instruction tells the model what to do when evidence is missing, when a tool fails, when the user requests something outside policy, and when it should ask for clarification.

Structured outputs deserve special attention. If downstream code depends on fields, make the schema explicit and validate it outside the model. The model may generate JSON, but trusted code decides whether that JSON is complete, authorized, and safe to execute. This is especially important when structured output becomes tool arguments or workflow state.

Treat prompts and instructions as versioned product assets. Store the instruction version with traces and evaluation results. When quality changes, the team should know whether the cause was model choice, instruction wording, retrieval changes, tool behavior, or policy.

Choose models by step difficulty and risk, not brand preference.
Write instructions for missing evidence, tool failure, refusal, and clarification.
Validate structured outputs with trusted code.
Version instructions and compare them in evaluations.

Create a Model Routing Contract

Create a model-routing table for one agent. For each step, list the task, required reasoning level, risk, expected output shape, latency budget, and fallback model. This makes model choice concrete instead of emotional.

Then write an instruction contract for the highest-risk step. Include what the model should do, what it must not do, how it should handle missing evidence, and when it should ask for clarification or escalate. Test the instruction with normal, ambiguous, and adversarial inputs.

Route by task difficulty and risk.
Use schemas for outputs that code consumes.
Store model and instruction versions in traces.

Compare Routes with Evidence

Compare outputs across at least two model routes so the team can see whether a stronger model improves correctness, safety, latency, or reviewer effort enough to justify the cost.

Model Route Contract

Route models by evaluated task requirements rather than a global quality label. Record the input class, reasoning need, supported tools, context size, output schema, latency objective, safety risk, and fallback for each route. A cheaper model is useful only when it still meets the acceptance threshold for that step.

Treat instructions as versioned application code. Separate stable policy, role guidance, task input, retrieved evidence, and tool output so untrusted text cannot masquerade as a higher-priority rule. Use structured outputs when code consumes the result, validate the schema after generation, and reject unknown enum values or incomplete objects instead of guessing intent.

Pin model snapshots when reproducibility matters and record the actual model, instruction version, tool set, schema version, and sampling settings in traces. Re-run normal, ambiguous, and adversarial evaluations before changing any of them. A successful JSON parse does not prove the decision itself remains correct.

Define fallback semantics before an outage. A secondary model may lack a required tool, modality, context length, or structured-output guarantee. Route to it only when the step contract remains satisfiable; otherwise pause, degrade to a clearly limited response, or transfer to a human rather than silently returning lower-integrity work.

Choose routes from measured capability, risk, cost, and latency.
Keep untrusted context visibly separate from instructions.
Validate structured output before application use.
Version and evaluate the complete model contract.

Model Contract Examples

Typed Agent Decision Contract

This example validates a model-like decision before the runtime acts on it.

Typed Agent Decision Contract

from dataclasses import dataclass
from typing import Literal

AllowedAction = Literal["search_policy", "ask_user", "finish"]

@dataclass
class AgentDecision:
    action: AllowedAction
    reason_summary: str
    query: str | None = None

def validate_decision(raw: dict) -> AgentDecision:
    allowed = {"search_policy", "ask_user", "finish"}
    action = raw.get("action")
    if action not in allowed:
        raise ValueError(f"Unsupported action: {action}")

    if action == "search_policy" and not raw.get("query"):
        raise ValueError("search_policy requires a query")

    return AgentDecision(
        action=action,
        reason_summary=str(raw.get("reason_summary", ""))[:200],
        query=raw.get("query"),
    )

decision = validate_decision({
    "action": "search_policy",
    "reason_summary": "The refund rule is not in the current context.",
    "query": "refund eligibility window",
})

print(decision)

The model output is treated as untrusted data.
Only known actions are accepted.
Action-specific required fields are checked before execution.

Model Routing by Task Complexity

A simple router can keep routine work fast while reserving a stronger model for harder tasks.

Model Routing by Task Complexity

def choose_model(task: dict) -> str:
    if task["risk"] == "high":
        return "strong-reasoning-model"
    if task["type"] in {"classify", "extract", "format"}:
        return "fast-small-model"
    if task["context_tokens"] > 30_000:
        return "long-context-model"
    return "balanced-model"

task = {
    "type": "classify",
    "risk": "low",
    "context_tokens": 1200,
}

print(choose_model(task))

Routing policy is explicit and testable.
Real systems should use evaluation results, not model names alone.
High-risk tasks may also require human review regardless of model choice.

Before you move on