Evaluation#

The platform embeds evaluation into its pipelines rather than running it offline: a retrieval evaluator decides whether RAG may answer at all, and a reviewer agent scores every email workflow 0–1. Both are temperature-0 LLM judges returning strict JSON.

Judge 1 — the retrieval evaluator (gate)#

Where: evaluate_node in the corrective RAG graph, calling evaluate_retrieved_context.

Question it answers: does the retrieved context actually contain the answer? — explicitly instructed not to answer the user's question itself.

Contract:

{ "has_answer": true, "confidence": 0.85, "reason": "short explanation" }

Effect: has_answer routes the graph — true → generate a grounded answer; false → return the fixed refusal string. This is a hard gate: a negative verdict costs the user an answer, a positive one costs one more LLM call. confidence and reason are returned to the API caller (retrieval_evaluation field) but do not influence routing — no threshold is applied to confidence today.

Failure modes to understand:

• False negative → an answerable question gets refused. Safe but frustrating; visible in responses where sources look relevant yet has_answer=false.
• False positive → generation proceeds on weak context; the answer prompt's own "use ONLY the provided context / refuse otherwise" rule is the second layer of defense.
• Empty retrieval never reaches the judge — the node short-circuits to refusal with a synthetic evaluation, spending zero LLM calls.

Judge 2 — the workflow reviewer (score)#

Where: stage 4 of the email workflow, calling run_reviewer_agent with the entire package: original email, analysis JSON, created tasks, drafted reply.

Contract:

{ "approved": true, "quality_score": 0.92, "issues": [], "recommendation": "..." }

Effect: advisory, not gating. The verdict is persisted — quality_score onto WorkflowRun.quality_score, the full JSON into the workflow's final_output, a reviewer_agent step, and a costed ToolCall — and returned to the caller. Nothing is blocked or rolled back on approved: false; tasks were already committed before the reviewer ran. Consumers decide what a low score means.

Aggregation: the dashboard averages parseable quality scores into average_quality_score — the platform's built-in quality KPI per agent (Observability).

Comparing the two judges#

Honest limitations#

• Self-judging: the same model (gpt-4o-mini) generates and judges; correlated blind spots are expected. A stronger judge model is the classic upgrade.
• No calibration: confidence and quality_score are uncalibrated model outputs — treat them ordinally (higher ≈ better), not as probabilities.
• No offline evaluation: there is no golden dataset, regression suite, or prompt A/B harness. The recorded traces (WorkflowStep, ToolCall inputs/outputs) are exactly the raw material such a harness would consume — see Testing and the Roadmap.

Extending evaluation#

Practical next steps, in effort order:

1. Act on approved: false — regenerate the reply once with the issues fed back (turns the reviewer into a corrective loop like the RAG gate).
2. Confidence threshold on the retrieval evaluator (e.g., refuse below 0.4 even when has_answer=true).
3. Golden-set harness — replay recorded workflow inputs against prompt changes, compare reviewer scores.

Related pages#

• Corrective RAG · Email Workflow — the host pipelines
• Prompt Catalog — both judge prompts verbatim