Skip to content
2 min read · 469 words

Cost Monitoring

Every costed step in the platform prices itself with one heuristic — characters ÷ 4, at gpt-4o-mini rates — and the dashboard sums the results. Cheap, instant, and deliberately approximate.

The flow of a cost number

Rendering diagram…

The estimator

Full code walkthrough: Service: Monitoring. The essentials:

  • Tokens = max(1, len(text) // 4) per side (input, output); empty/None → 0.
  • Dollars = input tokens × $0.15/1M + output tokens × $0.60/1M (gpt-4o-mini list prices, hardcoded).
  • Output = {"model_name", "estimated_tokens", "estimated_cost": "0.000123"} — cost as a 6-decimal string.

Coverage map — what is and isn't costed

LLM activityCosted?Why
Workflow analysis / reply / reviewer stages✅router calls the estimator per stage
Workflow task stage➖ n/ano LLM call
Chat (/chat)❌logs an AgentRun, no costed ToolCall
Single-agent email analyze❌ToolCalls logged without cost fields
Document search / RAG ask❌ToolCalls logged without cost fields
Embeddings (upload + queries)❌never logged at all

Consequence: estimated_total_cost on the dashboard reflects workflow spend only, and therefore understates true OpenAI usage — possibly substantially for RAG-heavy agents. Extending coverage is a good first contribution (Guides).

Trust boundaries

Good for: spotting an agent that suddenly does 10× more work, comparing agents, watching trends, demoing observability.

Not for: budgeting, billing reconciliation, or per-request accounting. Reasons:

  1. chars/4 tokenization error (±30% typical, worse for non-English/code);
  2. system-prompt and JSON-scaffolding tokens are not counted;
  3. prices are frozen constants — model changes and OpenAI price changes don't propagate;
  4. coverage gaps above.

For ground truth use the OpenAI usage dashboard, or upgrade the platform to read response.usage (below).

Reading costs in practice

bash
# Raw, per step:
curl http://localhost:8001/agents/1/tool-calls | jq \
  '.[] | select(.estimated_cost != null) | {tool_name, model_name, estimated_tokens, estimated_cost, latency_ms}'
 
# Aggregated:
curl http://localhost:8001/agents/1/dashboard | jq \
  '{estimated_total_tokens, estimated_total_cost, average_latency_ms}'

The frontend surfaces the same aggregates in PerformanceCard (cost to 4 decimals, tokens locale-formatted) and the "Cost estimate" banner.

The upgrade path to real numbers

  1. Services return (text, usage) instead of bare text — response.usage carries exact prompt_tokens/completion_tokens.
  2. Routers log real counts; keep the estimator as fallback.
  3. Move the price table to configuration keyed by model name.
  4. Store estimated_cost as a numeric column (Design Decisions).

Tracked on the Roadmap.

PreviousOpenAI IntegrationNextEvaluation