Lesson 23: Testing#

Confirming This From the Repository, Not Assuming It#

backend/pyproject.toml's dependency list has no pytest, no httpx test client extras, no pytest-mock. There's a file named backend/test_delete.py at the repository root of backend/ — but CLAUDE.md is explicit that this is a scratch/practice script, not a real test, grouped with delete.py, exercise_delete.py, practice_solution.py, and solution_delete.py as files main.py never imports. Confirm it yourself: grep -r "import test_delete\|from test_delete" backend/app returns nothing — nothing in the running application depends on it, and it isn't collected by any test runner because no test runner is configured.

This lesson is therefore necessarily about how you would test this codebase, grounded in its actual structure, not a description of tests that exist.

Three Layers, Three Different Testing Strategies#

1. Pure functions — no mocking needed#

Several functions in this codebase are pure: given the same input, they always produce the same output, with no network calls or database access. These are the cheapest, highest-value tests to write first:

from services.document_service import split_text_into_chunks
 
def test_split_text_into_chunks_empty_string():
    assert split_text_into_chunks("") == []
 
def test_split_text_into_chunks_respects_overlap():
    text = "A" * 1000
    chunks = split_text_into_chunks(text, chunk_size=800, overlap=100)
    assert len(chunks) == 2
    assert chunks[0][-100:] == chunks[1][:100]  # confirms the overlap actually overlaps

normalize_due_date, agent_has_tool, safe_float, safe_int (Lesson 09a) are equally good candidates — small, deterministic, currently exercised only by manual testing.

2. Service functions that call OpenAI — mock the client#

from unittest.mock import patch, Mock
 
@patch("services.openai_service.client")
def test_analyze_email_parses_valid_json(mock_client):
    mock_client.chat.completions.create.return_value = Mock(
        choices=[Mock(message=Mock(content='{"summary": "test", "intent": "test", "priority": "low", "deadline": null, "action_items": [], "suggested_reply": "ok"}'))]
    )
    result = analyze_email("some email text")
    assert result["priority"] == "low"
 
@patch("services.openai_service.client")
def test_analyze_email_raises_on_malformed_json(mock_client):
    mock_client.chat.completions.create.return_value = Mock(choices=[Mock(message=Mock(content="not json"))])
    with pytest.raises(ValueError):
        analyze_email("some email text")

This is where you'd verify the exact failure mode documented in Lesson 19 — that a malformed LLM response raises ValueError, without spending real API credits or depending on network availability to run the test suite.

3. Routers — FastAPI's TestClient plus a swapped-out database#

from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from database import Base, get_db
from main import app
 
engine = create_engine("sqlite:///:memory:", connect_args={"check_same_thread": False})
TestingSessionLocal = sessionmaker(bind=engine)
Base.metadata.create_all(bind=engine)
 
def override_get_db():
    db = TestingSessionLocal()
    try:
        yield db
    finally:
        db.close()
 
app.dependency_overrides[get_db] = override_get_db
client = TestClient(app)
 
def test_get_nonexistent_agent_returns_404():
    response = client.get("/agents/99999")
    assert response.status_code == 404

This is exactly the dependency-override mechanism described in Lesson 10 — because get_db is the only dependency this backend has, one override at the top of a test file redirects all 20 endpoints to an isolated in-memory database, with zero router code changes required.

A Test Worth Writing First: the /stats Bug#

The broken GET /{agent_id}/stats endpoint documented in Lesson 09a is exactly the kind of defect an integration test would have caught before it shipped:

def test_agent_stats_endpoint():
    create_response = client.post("/agents/", json={"name": "t", "type": "t", "system_prompt": "t"})
    agent_id = create_response.json()["id"]
    response = client.get(f"/agents/{agent_id}/stats")
    assert response.status_code == 200  # currently fails — AttributeError before this line is even reached

Running this against the actual current code raises an AttributeError (models.Agent.agent_id doesn't exist), which is a decisive way to demonstrate why this lesson exists: the bug was findable by a five-line test that has never been written.

What "100% Coverage" Would Not Actually Buy You Here#

Worth saying plainly: line coverage doesn't catch bugs like the /stats endpoint's response_model mismatch unless a test actually asserts on the response shape, not just the status code. A coverage tool would happily report the get_agent_stats function as "covered" the moment any test calls it and lets the exception propagate — coverage percentage and correctness are different measurements.

Common Mistakes#

• Testing against the real agentops.db file. It's stateful across runs and shared with whatever's running in development — always override get_db to an isolated database.
• Testing service functions against the real OpenAI API. Costs money, is slow, and is non-deterministic — mock the client.
• Assuming a passing test suite proves the API contract (the response_model) is honored. Assert on response bodies, not just status codes.

Exercises#

1. Set up pytest and httpx in backend/pyproject.toml, write the in-memory TestClient fixture above, and write one passing test and one that reproduces the /stats bug.
2. Write pure-function tests for normalize_due_date covering all three of its branches: a recognized relative phrase, a past ISO date, and unparseable text.
3. Mock search_agent_documents (not the OpenAI client directly) to return a fixed list of chunks, and write a test asserting run_corrective_rag_graph routes to refusal_node when the mocked evaluator returns has_answer: False.

Key Takeaways#

🎯 No test suite exists today — backend/test_delete.py is a scratch practice file, not part of the real test surface. 🎯 Pure functions (split_text_into_chunks, agent_has_tool, normalize_due_date) are the cheapest, highest-value place to start. 🎯 get_db being the only real dependency means one dependency_overrides entry unlocks isolated testing for all 20 endpoints. 🎯 A five-line integration test would have caught the /stats endpoint bug before it shipped.

Next Steps#

👉 Lesson 24: Performance →

Module 6 · Production Mastery