Skip to content
3 min read · 598 words

Testing

Current state, stated plainly: there are no automated tests and no test runner configured in either project. The only mechanical check is npm run typecheck. This page is the plan for fixing that, ordered by payoff.

  • Backend: no pytest, no test directory (backend/test_delete.py is a scratch script, not a test — see Project Structure).
  • Frontend: npm run typecheck (tsc --noEmit) only; no Jest/Vitest/Playwright.
  • Docs: mkdocs build warnings act as link checks (Docs Site).
  • Manual verification: the Quick Start sequence is the de-facto smoke test.

What makes this codebase testable right now

Two design properties make tests cheap to add:

  1. Services are pure-ish functions — split_text_into_chunks, normalize_due_date, estimate_openai_cost, agent_has_tool take values and return values, no mocks needed.
  2. All external I/O funnels through few seams — OpenAI via three module-level clients, storage via get_db (an overridable FastAPI dependency) and one Chroma collection object. Patch those seams and the whole API is testable offline.

The prioritized plan

Tier 1 — pure-logic unit tests (an afternoon, zero infrastructure)

Highest value-per-line; these functions encode real business rules:

python
from services.document_service import split_text_into_chunks
 
def test_empty_text_yields_no_chunks():
    assert split_text_into_chunks("   ") == []
 
def test_overlap_windows():
    text = "a" * 1600
    chunks = split_text_into_chunks(text, chunk_size=800, overlap=100)
    assert [len(c) for c in chunks] == [800, 800, 200]   # windows at 0, 700, 1400
 
def test_overlap_ge_chunk_size_does_not_loop_forever():
    assert split_text_into_chunks("abc" * 300, chunk_size=100, overlap=100)

Same treatment for: normalize_due_date (simple words kept, past ISO dates dropped, junk passed through), estimate_tokens/estimate_openai_cost (empty text, rounding, string format), agent_has_tool (empty list, whitespace, exact-match).

Tier 2 — API tests with a temp DB and mocked LLM (a day)

FastAPI's TestClient + dependency override + monkeypatch on the service functions:

python
# in-memory SQLite engine → override get_db → TestClient(app)
# monkeypatch generate_agent_response / analyze_email to return canned values

Target the contract behaviors documented in the API Reference: 404 guards, 403 tool gating, chat's 200-on-LLM-failure fallback (assert the run's status="failed"), analyze's task creation with/without task_creator, workflow trace rows (run + 4 steps + costed tool calls), task-status allow-list.

Tier 3 — frontend component tests (Vitest + Testing Library)

CreateAgentForm is the one component with logic worth testing: tool toggling recomputes the CSV, client-side required-field validation, getApiErrorMessage handling both detail shapes, success/error banners. The presentational components are better covered by…

Tier 4 — one E2E happy path (Playwright)

Boot backend (mocked key or a test OpenAI stub), boot frontend, then: create agent via UI → see it on /agents → dashboard renders. This single test would have caught most integration regressions possible in this codebase.

Tier 5 — prompt/eval regression (the AI-native layer)

The recorded traces are ready-made fixtures: replay stored WorkflowStep inputs against prompt changes and compare reviewer quality_score distributions (Evaluation). This is where LLM product testing actually lives.

Conventions for when tests land

  • Backend: pytest under backend/tests/, run as uv run pytest from backend/ (add pytest to a dev dependency group).
  • Frontend: Vitest under frontend/**/*.test.tsx, wired as npm test.
  • No live-OpenAI tests in CI — every LLM call mocked; live calls only behind an explicit opt-in marker.
  • Update the Development Workflow checklist once runners exist.
  • Known Issues — bugs that Tier 1–2 tests would have caught (several literally)
  • Roadmap — testing's place in the sequence
PreviousDocs SiteNextTroubleshooting