Skip to content
Module 3 · Services Deep DiveIntermediate10 min read · 1,970 words

Lesson 08b: Monitoring Service Deep Dive

"If you can't measure it, you can't improve it."

Introduction

Cost tracking is critical for LLM applications. Without monitoring, you're flying blind - spending unknown amounts on AI operations with no visibility into what's working or what's wasteful.

monitoring_service.py provides the observability layer for tracking token usage and estimating costs.

File Overview

Location: backend/app/services/monitoring_service.py
Size: 1,194 bytes
Functions: 2
Purpose: Track LLM resource usage and costs

Complete File Walkthrough

Function 1: estimate_tokens()

python
def estimate_tokens(text: str | None) -> int:
    """
    Simple token estimation.
    Approximation: 1 token ≈ 4 characters in English.
    This is not exact, but useful for monitoring dashboard.
    """
    if not text:
        return 0
 
    return max(1, len(text) // 4)

Purpose

Quickly estimate token count without calling expensive tokenization API.

The Token Problem

What are tokens?

  • LLMs process text as tokens, not characters
  • 1 token ≈ 4 characters (English average)
  • Examples:
    • "Hello" = 1 token
    • "Hello world" = 2 tokens
    • "artificial intelligence" = 2-3 tokens

Why estimate? OpenAI charges by tokens, not characters:

  • gpt-4o-mini: $0.15 per 1M input tokens
  • Need to track usage for billing

Exact vs Estimate:

python
# Exact (requires tiktoken library)
import tiktoken
encoder = tiktoken.encoding_for_model("gpt-4o-mini")
tokens = len(encoder.encode(text))  # Accurate but slow
 
# Estimate (this function)
tokens = len(text) // 4  # Fast but approximate

Algorithm Breakdown

Step 1: Handle None

python
if not text:
    return 0

Why needed: Prevents crashes on null input
Use case: Optional fields like error_message

Step 2: Estimate

python
return max(1, len(text) // 4)

len(text): Character count
// 4: Integer division (1 token per 4 chars)
max(1, ...): Always return at least 1

Why max(1, ...)?

python
# Without max
estimate_tokens("Hi")  # len=2, 2//4=0 ❌ Wrong!
 
# With max
estimate_tokens("Hi")  # max(1, 0)=1 ✅ Correct!

Accuracy Analysis

Test cases:

python
# Short text
"Hello" (5 chars) → 1 token estimated, 1 actual ✅
 
# Medium text
"The quick brown fox" (19 chars) → 4 tokens estimated, 4 actual ✅
 
# Long text
"Artificial intelligence is transforming..." (200 chars)
→ 50 tokens estimated, 45-55 actual ✅ Close enough
 
# Special cases
"" → 0 tokens ✅
None → 0 tokens ✅
"A" → 1 token (due to max) ✅

Accuracy: ±10-20% for English text

Why acceptable?

  • Dashboard needs approximate values
  • Exact tracking happens on OpenAI's side
  • Performance > precision for monitoring

Limitations

Inaccurate for:

  1. Non-English text

    • Chinese: 1 token ≈ 2 characters
    • Arabic: Different ratios

  2. Code

    python
    "def hello():" 
# Estimate: 3 tokens (13/4)
# Actual: 4-5 tokens (punctuation counted separately)

  3. Special characters

    • Emojis, Unicode symbols
    • Often more than 1 token each

When to upgrade: If billing discrepancies > 20%, integrate tiktoken:

python
import tiktoken
 
def estimate_tokens_exact(text: str | None, model: str = "gpt-4o-mini") -> int:
    if not text:
        return 0
    encoder = tiktoken.encoding_for_model(model)
    return len(encoder.encode(text))

Function 2: estimate_openai_cost()

python
def estimate_openai_cost(
    input_text: str | None,
    output_text: str | None,
    model_name: str = "gpt-4o-mini"
) -> dict:
    """
    Estimate tokens and approximate cost.
    Prices here are approximate and can be updated later.
    """

Purpose

Calculate estimated dollar cost for an LLM operation.

Parameters

input_text: Prompt sent to LLM
output_text: Response from LLM
model_name: Which model used (default: gpt-4o-mini)

Returns:

python
{
    "model_name": "gpt-4o-mini",
    "estimated_tokens": 150,
    "estimated_cost": "0.000045"  # $0.000045 (4.5¢ per 100K)
}

Step 1: Calculate Token Counts

python
input_tokens = estimate_tokens(input_text)
output_tokens = estimate_tokens(output_text)
total_tokens = input_tokens + output_tokens

Why separate input/output? OpenAI charges different rates:

  • Input tokens: $0.15 per 1M
  • Output tokens: $0.60 per 1M (4x more expensive!)

Example:

python
input_text = "What is AI?" (12 chars)
input_tokens = 3
 
output_text = "AI is artificial intelligence, a field of computer science..." (400 chars)
output_tokens = 100
 
total_tokens = 103

Step 2: Define Pricing (GPT-4o-mini)

python
# Approximate price for gpt-4o-mini:
# input: $0.15 per 1M tokens
# output: $0.60 per 1M tokens
input_cost_per_1m = 0.15
output_cost_per_1m = 0.60

Current OpenAI Pricing (as of code):

ModelInput (per 1M)Output (per 1M)
gpt-4o-mini$0.15$0.60
gpt-4$5.00$15.00
gpt-4-turbo$10.00$30.00

Why hardcoded?

  • Pricing stable enough for estimates
  • Can be updated when prices change
  • Real billing comes from OpenAI directly

Future improvement:

python
# Config-driven pricing
PRICING = {
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-4": {"input": 5.00, "output": 15.00},
}

Step 3: Calculate Cost

python
input_cost = (input_tokens / 1_000_000) * input_cost_per_1m
output_cost = (output_tokens / 1_000_000) * output_cost_per_1m
 
total_cost = input_cost + output_cost

Formula:

Cost = (tokens / 1,000,000) × price_per_1M

Example calculation:

python
# Chat: "What is AI?" → 400-char response
input_tokens = 3
output_tokens = 100
 
input_cost = (3 / 1,000,000) × $0.15 = $0.00000045
output_cost = (100 / 1,000,000) × $0.60 = $0.00006000
 
total_cost = $0.00006045 ≈ $0.00006 (6¢ per 100K requests)

Why divide by 1_000_000? Pricing is "per million tokens", convert to "per token" first.

Step 4: Format Response

python
return {
    "model_name": model_name,
    "estimated_tokens": total_tokens,
    "estimated_cost": f"{total_cost:.6f}"
}

Why .6f format?

  • Shows 6 decimal places
  • Captures micro-costs (fractions of cents)
  • Example: "0.000045" = $0.000045

Why string for cost?

python
# Float precision issues
0.1 + 0.2  # = 0.30000000000000004 ❌
 
# String storage (from calculation)
f"{0.000045:.6f}"  # = "0.000045" ✅

Complete Usage Example

Scenario: Chat Request

python
# User asks question
user_message = "Explain machine learning in simple terms"  # ~50 chars
 
# AI responds
ai_response = """
Machine learning is a way for computers to learn from examples
rather than being explicitly programmed. Instead of writing rules,
we show the computer many examples and it finds patterns.
"""  # ~180 chars
 
# Calculate cost
result = estimate_openai_cost(
    input_text=user_message,
    output_text=ai_response,
    model_name="gpt-4o-mini"
)
 
print(result)
# Output:
# {
#     "model_name": "gpt-4o-mini",
#     "estimated_tokens": 57,  # 12 input + 45 output
#     "estimated_cost": "0.000028"  # $0.000028
# }

Per-request cost: $0.000028
Cost per 1,000 requests: $0.028 (2.8¢)
Cost per 1M requests: $28

Integration with ToolCall Logging

In routers/agents.py

python
# Before LLM call
email_tool_start = time.time()
 
# Make LLM call
result = analyze_email(email_data.email_text)
 
# Calculate metrics
email_tool_latency = int((time.time() - email_tool_start) * 1000)
 
monitoring = estimate_openai_cost(
    input_text=email_data.email_text,
    output_text=json.dumps(result),
    model_name=os.getenv("OPENAI_MODEL", "gpt-4o-mini")
)
 
# Log to database
log_tool_call(
    db=db,
    agent_id=agent.id,
    tool_name="email_analyzer",
    tool_input=email_data.email_text,
    tool_output=json.dumps(result),
    success=True,
    latency_ms=email_tool_latency,
    model_name=monitoring["model_name"],
    estimated_tokens=monitoring["estimated_tokens"],
    estimated_cost=monitoring["estimated_cost"],
)

Flow:

  1. Capture input text
  2. Call LLM
  3. Capture output text
  4. Estimate cost with estimate_openai_cost()
  5. Store in database via log_tool_call()

Dashboard Usage

Query Total Costs

python
# Get all tool calls for an agent
tool_calls = db.query(ToolCall).filter(ToolCall.agent_id == agent_id).all()
 
# Sum costs
total_cost = sum(
    float(call.estimated_cost) if call.estimated_cost else 0
    for call in tool_calls
)
 
print(f"Total spent: ${total_cost:.2f}")
# Output: Total spent: $2.45

Average Cost Per Request

python
avg_cost = total_cost / len(tool_calls)
print(f"Average per request: ${avg_cost:.6f}")
# Output: Average per request: $0.000245

Token Usage Analysis

python
total_tokens = sum(
    call.estimated_tokens if call.estimated_tokens else 0
    for call in tool_calls
)
 
print(f"Total tokens: {total_tokens:,}")
# Output: Total tokens: 1,234,567

Cost Optimization Strategies

1. Choose Right Model

python
# Expensive
estimate_openai_cost(text, response, "gpt-4")
# $5/$15 per 1M tokens
 
# Cost-effective
estimate_openai_cost(text, response, "gpt-4o-mini")
# $0.15/$0.60 per 1M tokens
 
# Savings: 97% cheaper!

2. Reduce Output Length

python
# Expensive (verbose output)
"Generate a detailed explanation..." 
# → 500 tokens output = $0.00030
 
# Cheaper (concise output)
"Summarize in 2 sentences..."
# → 50 tokens output = $0.00003
 
# Savings: 90% cheaper!

3. Cache Responses

python
# Without caching
for i in range(1000):
    response = call_openai("What is AI?")
# Cost: $0.028 × 1000 = $28
 
# With caching
cache = {}
for i in range(1000):
    if "What is AI?" in cache:
        response = cache["What is AI?"]
    else:
        response = call_openai("What is AI?")
        cache["What is AI?"] = response
# Cost: $0.028 × 1 = $0.028
 
# Savings: 99.9% cheaper!

4. Prompt Optimization

python
# Expensive (redundant prompt)
system_prompt = """
You are a helpful assistant.
You should be helpful.
Always help users.
Be as helpful as possible.
"""  # 40 tokens
 
# Optimized
system_prompt = "You are a helpful assistant."  # 7 tokens
 
# Savings: 82% fewer input tokens

Monitoring Best Practices

1. Alert on Unusual Costs

python
# Daily cost check
daily_cost = sum_costs_for_date(today)
 
if daily_cost > 50:  # $50 threshold
    send_alert(f"High usage: ${daily_cost}")

2. Track Cost by Feature

python
# Chat costs
chat_costs = sum(call.estimated_cost for call in tool_calls 
                 if call.tool_name == "chat")
 
# Email analysis costs  
email_costs = sum(call.estimated_cost for call in tool_calls
                  if call.tool_name == "email_analyzer")
 
# Identify expensive features

3. Per-User Cost Tracking

python
# Track which users consume most
user_costs = {}
for call in tool_calls:
    user_id = call.agent.user_id  # If you add users
    user_costs[user_id] = user_costs.get(user_id, 0) + float(call.estimated_cost)
 
# Find top spenders
top_users = sorted(user_costs.items(), key=lambda x: x[1], reverse=True)[:10]

Limitations & Improvements

Current Limitations

  1. Approximate Only

    • ±10-20% accuracy
    • Not suitable for exact billing

  2. Hardcoded Pricing

    • Must manually update when prices change
    • No multi-model pricing lookup

  3. No Real-Time Tracking

    • Can't track actual OpenAI API usage
    • Relies on estimates

Improvements

Upgrade 1: Exact Tokenization

python
import tiktoken
 
def estimate_tokens_exact(text: str | None, model: str) -> int:
    if not text:
        return 0
    
    try:
        encoder = tiktoken.encoding_for_model(model)
        return len(encoder.encode(text))
    except Exception:
        # Fallback to estimate
        return estimate_tokens(text)

Upgrade 2: Dynamic Pricing

python
PRICING_TABLE = {
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
    "gpt-4": {"input": 5.00, "output": 15.00},
    "gpt-4-turbo": {"input": 10.00, "output": 30.00},
}
 
def get_model_pricing(model_name: str) -> dict:
    return PRICING_TABLE.get(model_name, PRICING_TABLE["gpt-4o-mini"])

Upgrade 3: OpenAI API Tracking

python
# OpenAI API returns actual usage
response = client.chat.completions.create(...)
 
actual_tokens = response.usage.total_tokens
actual_cost = calculate_from_actual(
    response.usage.prompt_tokens,
    response.usage.completion_tokens,
    model
)
 
# Store actual instead of estimate

Key Takeaways

🎯 Token estimation uses 1 token ≈ 4 characters rule (±20% accurate)

🎯 Cost calculation applies model-specific pricing per million tokens

🎯 Output tokens cost 4x more than input tokens for gpt-4o-mini

🎯 String storage for costs avoids floating-point precision issues

🎯 Monitoring enables optimization - track, analyze, improve

🎯 Estimates sufficient for dashboard - OpenAI tracks actual billing

Interview Questions

Junior

Q1: Why estimate tokens instead of counting exactly?
A: Speed. Estimation is instant, exact tokenization requires API calls or heavy libraries. For monitoring dashboards, approximate is sufficient.

Q2: Why do output tokens cost more than input tokens?
A: LLM generation (output) is computationally expensive - requires sampling, decoding, and multiple forward passes. Reading (input) is simpler.

Mid-Level

Q3: How would you improve accuracy beyond the 4:1 ratio?
A: Integrate tiktoken library for model-specific tokenization. Handle different languages with different ratios. Cache tokenization results.

Q4: Design a cost alerting system for this monitoring service.
A: Background job runs hourly, sums estimated_cost from ToolCall table for last 24h. If > threshold, send email/Slack alert. Track trends (sudden spikes).

Senior

Q5: The estimate shows $50/day but actual OpenAI bill is $65. Debug and fix.
A: (1) Check if all tool calls logged. (2) Verify pricing matches OpenAI current rates. (3) Compare estimated vs actual tokens from OpenAI API response. (4) Integrate tiktoken for exact counts. (5) Add reconciliation job comparing estimates vs OpenAI dashboard.

Q6: Design a cost allocation system for multi-tenant SaaS.
A: Add tenant_id to ToolCall. Create TenantUsage table aggregating costs daily. Implement usage-based billing tiers. Add cost prediction based on historical patterns. Include overage alerts per tenant.

Exercises

Exercise 1: Test Accuracy

Calculate actual vs estimated tokens for 10 different prompts. Measure accuracy percentage.

Exercise 2: Cost Calculator

Build CLI tool that takes text input and model name, outputs estimated cost.

Exercise 3: Dashboard Query

Write SQL query to find top 10 most expensive tool calls by estimated_cost.

Exercise 4: Budget Alert

Implement function that checks if daily costs exceed budget and returns alert message.

Exercise 5: Multi-Model Pricing

Extend estimate_openai_cost() to support GPT-4 and GPT-4-turbo with correct pricing.

👉 Next: Multi-Agent Email Service →

← Previous: OpenAI Service | ↑ Index

Lesson 08b Complete! Services: 2/5 documented (40%)

PreviousLesson 08a - OpenAI ServiceNextLesson 08c - Multi-Agent Email