This capstone connects the course into one feature a real team could review. The support copilot reads a customer request, retrieves approved policy, may look up an order through a read-only tool, and produces a structured resolution for an agent. It does not send money, change a record, or decide access. Those remain in the host application.
The work is deliberately broader than a prompt. You will define a contract, assemble only authorized evidence, validate a structured proposal, evaluate normal and hostile cases, trace the outcome without sensitive content, and release behind a flag. If one boundary fails, the workflow should hand off rather than improvise.
| Layer | Deliverable | Proof |
|---|---|---|
| Model boundary | Server-side adapter and structured proposal. | Response validation and redacted trace. |
| Knowledge boundary | Access-filtered policy retrieval with citations. | Cases with correct evidence and no-evidence escalation. |
| Action boundary | One narrow read-only order lookup. | Authorization tests and minimal tool results. |
| Release boundary | Feature flag, evaluation set, dashboard, and rollback plan. | A launch scorecard signed off by named owners. |
Begin with a contract that a support lead, engineer, and security reviewer can all read. Define the allowed statuses, the fields an agent needs, the policy sources that may be used, the one read-only tool, and the explicit non-goals. This prevents scope from quietly expanding as the prompt grows.
A strong capstone has a useful failure state: needs_review with a concise reason and the evidence available.
The copilot receives only the current ticket, authorized source excerpts, and a validated order lookup when one is necessary. It returns a draft resolution, not an action. The draft includes source IDs and an evidence status so the agent can see whether the answer is supported or should be escalated.
Keep policy and customer-specific records separate in the request assembly step.
Use a schema validator at the boundary. An unexpected field, unknown status, unsupported citation, or attempted write becomes a review event. Do not patch malformed model output into a valid-looking record; preserve the failure reason for evaluation.
Tool results are data, not instructions. Treat text returned by a search or order service as untrusted content.
Evaluate the complete path: correct routing, source relevance, citation support, tool authorization, schema validity, safe abstention, latency, and cost. Include duplicate requests, a missing order, a stale policy, and prompt-injection content inside an uploaded or retrieved document.
Release to internal agents first with a feature flag. Define what metric or incident causes an immediate disable.
This capstone joins the course into one product: a support copilot that receives a ticket, retrieves only permitted policy passages, proposes a structured resolution, performs a narrow read-only lookup when needed, and escalates anything uncertain. The objective is not a chat demo. It is a feature a support team can inspect, evaluate, and turn off safely.
Build in thin vertical slices. First return a structured proposal from a fixed policy excerpt. Then add access-filtered retrieval. Then add the read-only order lookup. Finally, run the release scorecard against normal, adversarial, and missing-evidence cases. Every slice should leave behind a test and an observable record.
The final application object records the draft, evidence, and whether an agent must review it.
def resolution_record(status: str, source_ids: list[str], next_step: str) -> dict:
allowed = {"ready_for_agent", "needs_review"}
if status not in allowed:
status = "needs_review"
return {
"status": status,
"source_ids": source_ids,
"next_step": next_step,
}
print(resolution_record("ready_for_agent", ["refund-policy-v3"], "confirm with customer")){'status': 'ready_for_agent', 'source_ids': ['refund-policy-v3'], 'next_step': 'confirm with customer'}A simple release rule illustrates that a strong demo is not enough to enable a feature broadly.
def may_expand_release(eval_pass_rate: float, citation_failure_rate: float, flag_enabled: bool) -> bool:
return flag_enabled and eval_pass_rate >= 0.95 and citation_failure_rate <= 0.01
print(may_expand_release(0.97, 0.005, True))TrueThis record keeps the customer request, evidence, proposal, and workflow status separate so a reviewer can understand what happened.
def copilot_record(ticket_id: str, source_ids: list[str], proposal: str) -> dict:
return {
"ticket_id": ticket_id,
"source_ids": source_ids,
"proposal": proposal,
"status": "needs_review" if not source_ids else "proposed",
}
print(copilot_record("T-71", [], "refund")["status"])needs_reviewThe release gate combines task quality with safety and operability signals.
def ready_to_release(quality_ok: bool, safety_ok: bool, rollback_ready: bool) -> bool:
return quality_ok and safety_ok and rollback_ready
print(ready_to_release(True, True, False))
print(ready_to_release(True, True, True))False
TrueA complete product build exercise
0 of 2 completed
Yes. Keep the same boundaries and replace the policy, read-only tool, schema, evaluation cases, and human reviewer for the domain.
After representative evaluation, security review, observability, a documented fallback, and a controlled rollout all show evidence appropriate to the risk.
Explore 500+ free tutorials across 20+ languages and frameworks.