LangChain Production: Streaming, Evaluation, Observability and Guardrails

Production Checklist

A production LLM application needs feedback loops. You need to know what prompt was used, what context was retrieved, what tools were called, how many tokens were spent, where parsing failed, and whether users got useful answers.

Evaluation is the difference between prompt tinkering and engineering. Build a dataset of realistic inputs, expected qualities, and failure cases. Run it whenever prompts, models, retrievers, or tools change.

Before shipping, decide what your system does when the model is slow, the parser fails, retrieval returns weak context, a tool times out, or the user asks for something outside policy. These cases should be designed, not discovered by customers.

Add request IDs and trace every chain step.
Stream long responses for better perceived latency.
Set model timeouts, retries, and fallback responses.
Track token usage and cost by route or feature.
Evaluate retrieval and generation separately.

Evaluation Strategy

Start with a small golden set: common questions, edge cases, adversarial prompts, missing-context cases, and examples that previously failed. For each case, record what good behavior means.

Use exact checks for structured outputs.
Use rubric checks for natural language answers.
Keep examples from production failures and support tickets.

Streaming, Cancellation, and Cost Limits

Streaming improves time to first useful output but creates a partial-response contract. Decide what the client displays if retrieval, a tool, parsing, or the model fails after tokens have already been sent. Propagate disconnect cancellation so abandoned requests stop consuming model and tool capacity.

Record input, output, cached, and tool token or request costs by feature and tenant. Bound model calls, tool loops, retrieved context, concurrency, and retries. A fallback should preserve safety and clearly indicate reduced capability rather than silently returning lower-quality unsupported text.

Observability and Release Gates

A trace should connect the request to prompt version, model, retrieval query, selected documents, tool calls, latency, token use, parser result, and final response without recording secrets by default. Stable identifiers make a bad answer reproducible while redaction and retention rules protect user data.

Run offline evaluation before release, compare a candidate with the current production baseline, and canary a bounded traffic share when risk justifies it. Monitor quality proxies, refusal behavior, latency, cost, and tool failures. Define rollback thresholds before deployment instead of interpreting regressions after all traffic moves.

Simple Evaluation Harness

This lightweight evaluator catches regressions in a RAG answer chain. Real systems can expand this with traces, rubrics, and model-graded checks.

Simple Evaluation Harness

eval_cases = [
    {
        "question": "Can annual customers get refunds?",
        "must_include": ["14 days", "billing-policy.md"],
    },
    {
        "question": "Do you support passwordless SSO?",
        "must_include": ["SAML", "OIDC", "security.md"],
    },
]

def evaluate(chain):
    failures = []
    for case in eval_cases:
        answer = chain.invoke(case["question"])
        missing = [text for text in case["must_include"] if text.lower() not in answer.lower()]
        if missing:
            failures.append({
                "question": case["question"],
                "answer": answer,
                "missing": missing,
            })
    return failures

failures = evaluate(chain)
if failures:
    for failure in failures:
        print("FAILED:", failure["question"])
        print("Missing:", failure["missing"])
        print("Answer:", failure["answer"])
    raise SystemExit(1)

print("All eval cases passed")

This is intentionally simple so it can run in CI.
Natural language evals should combine deterministic checks with human review or rubric grading.

Streaming Tokens

Streaming improves user experience for longer answers and makes slow model calls feel responsive.

Streaming Tokens

for chunk in chain.stream("Summarize our refund policy in three bullets."):
    print(chunk, end="", flush=True)

Design the UI to handle partial output and cancellation.
Do not stream hidden chain-of-thought or internal tool data to users.

Before you move on