Skip to content
3 min read · 593 words

Observability

The platform's defining feature: every automated step leaves a queryable row. This page is the contract — what gets recorded, when, and what the UI depends on.

The four record types

Rendering diagram…
RecordWritten byGranularityKey fields
AgentRunchat endpointone HTTP chat exchangeinput_text, output_text, latency_ms, status, error_message
ToolCallevery tool/LLM stepone automated actiontool_name, tool_input/output, success, latency_ms, model_name, estimated_tokens, estimated_cost, optional agent_run_id
WorkflowRunworkflow endpointsone pipeline executionworkflow_name, input_text, final_output, status, quality_score
WorkflowStepworkflow endpointsone pipeline stagestep_name, input_data, output_data, status, latency_ms

The contract

  1. Every automated/LLM step logs a ToolCall — success and failure. Failed calls carry success="false" and error_message, and are committed even when the request 500s.
  2. LLM-backed tool calls attach cost data — model_name, estimated_tokens, estimated_cost from estimate_openai_cost.
  3. Tool calls inside a chat link to their run via agent_run_id.
  4. Multi-step pipelines create a WorkflowRun first (status="running", committed immediately) and log a WorkflowStep per stage, finalizing status and quality_score at the end.
  5. Latency is wall-clock milliseconds measured around the step: int((time.time() - start) * 1000).

The dashboard, run history, and workflow trace UIs assume these rules. Breaking them silently breaks the product.

Current tool-call names

tool_nameLogged byCosted?
email_analyzersingle-agent analyze❌
task_creatorsingle-agent analyze❌ (no LLM)
email_analyzer_or_task_creatoranalyze failure path❌
multi_agent_analysis_agentworkflow stage 1✅
multi_agent_task_agentworkflow stage 2❌ (no LLM)
multi_agent_reply_agentworkflow stage 3✅
multi_agent_reviewer_agentworkflow stage 4✅
multi_agent_email_workflowworkflow failure path❌
document_searchsearch endpoint❌
langgraph_corrective_ragask endpoint❌

The uncosted LLM paths (chat, analyze, RAG) are the contract's known gaps — see Known Issues.

Reading the telemetry

bash
curl http://localhost:8001/agents/1/runs             # chat executions
curl http://localhost:8001/agents/1/tool-calls       # every automated step
curl http://localhost:8001/agents/1/workflow-runs    # pipeline executions
curl http://localhost:8001/agents/workflow-runs/3/steps   # one pipeline's trace

A workflow trace reads like a flight recorder — for run 3 you will see analysis_agent, task_agent, reply_agent, reviewer_agent steps (or a workflow_error step), each with inputs, outputs, status, and latency.

The dashboard aggregation

GET /agents/{id}/dashboard computes, in Python, over all of an agent's rows:

MetricSourceMethod
total/open/in_progress/completed_taskstaskscount by status
total/successful/failed_tool_callstool_callscount by success string
total/successful/failed_workflow_runsworkflow_runscount by status
average_latency_mstool_callsmean of non-null latencies
estimated_total_tokenstool_callssafe_int sum
estimated_total_costtool_callssafe_float sum → "%.6f" string
average_quality_scoreworkflow_runsmean of parseable quality_scores

The Next.js dashboard renders exactly this payload — see Frontend Data Fetching.

  • average_latency_ms averages tool calls, not chat runs — chat latency lives on AgentRun.
  • Cost/token totals only reflect the costed workflow stages (table above), so they understate true usage.
  • All numbers are estimates — see Cost Monitoring.

What observability does not include yet

No structured application logging, no metrics export (Prometheus/OTel), no alerting, no tracing across requests. The row-based telemetry is the foundation those would build on — see Roadmap.

PreviousMonitoring ServiceNextError Handling