Lesson 26: Caching#

Confirming the Absence#

No redis, no functools.lru_cache, no cachetools, no HTTP Cache-Control headers set on any response — grep -r "lru_cache\|cache" backend/app turns up nothing relevant. Every request that reaches this backend does full, fresh work: every embedding is recomputed, every OpenAI call is made fresh, every database query re-executes.

The Clearest Case: Repeated Identical Embeddings#

Lesson 15 already flagged this: create_embedding() has no memoization. Two concrete cases where this costs real, avoidable money:

1. The same search query, asked twice. search_agent_documents("what is the refund policy", ...) embeds that exact string via a fresh OpenAI call every single time it's called — including if it was called moments ago with the identical string.
2. Re-uploading the same document. If a .txt file is uploaded, deleted, and re-uploaded (or uploaded to two different agents), every chunk is re-embedded from scratch, even though the chunk text — and therefore the embedding OpenAI would produce — is identical to before.

A simple fix for both, grounded in what this codebase already has available (no Redis dependency exists, so start with what's free):

from functools import lru_cache
 
@lru_cache(maxsize=1000)
def create_embedding_cached(text: str) -> tuple[float, ...]:
    return tuple(create_embedding(text))

lru_cache requires hashable arguments and return values, which is why the return type changes from list[float] to tuple[float, ...] — a small but real interface change callers would need to account for. This is an in-process cache: it resets on every server restart and isn't shared across multiple server instances, which is a real limitation if this backend is ever run behind a load balancer with more than one worker process. A Redis-backed cache would solve that, at the cost of adding a new infrastructure dependency this codebase doesn't currently have.

A Second Case: Repeated Chat Completions Are Not Good Cache Candidates#

It's worth being precise about where caching helps and where it doesn't. Unlike embeddings, generate_agent_response() and the other chat-completion calls use temperature > 0 for every free-text-producing prompt (Lesson 19) — meaning the same input is expected to produce different output across calls. Caching those would silently make a system that's supposed to feel conversational and varied instead return byte-identical responses to similar questions, which is a correctness regression disguised as an optimization. The two temperature=0 evaluator calls (evaluate_retrieved_context, run_reviewer_agent) are better candidates in principle, since they're meant to be deterministic — though even there, caching would only pay off if the exact same question-plus-context (or exact same workflow output) repeats, which is far less likely than an identical search query repeating.

What Chroma Itself Already Optimizes (Not This Codebase's Doing)#

Worth distinguishing: ChromaDB's own approximate-nearest-neighbor index is not "caching" in the sense this lesson means — it's the underlying data structure that makes .query() fast at all, and it exists regardless of anything this codebase does. The caching gap discussed here is specifically about avoiding redundant OpenAI API calls, which sit in front of Chroma, not about Chroma's own internal performance.

Where Caching Would Not Help#

Two things worth ruling out explicitly, since "add caching" is sometimes reached for indiscriminately:

• The dashboard endpoint's Python-side aggregation (Lesson 24) — caching the result would mean showing stale stats after every new tool call or task; the real fix there is a better query, not a cache in front of a slow one.
• GET /{agent_id}/messages — chat history changes on every turn; caching it would require careful invalidation on every write, for a read that's already a simple indexed-by-nothing-but-agent_id query against a small table.

Common Mistakes#

• Caching non-deterministic LLM calls (any temperature > 0 prompt) and being surprised when "the same question" stops producing varied, natural responses.
• Assuming an in-process lru_cache helps across multiple server workers/instances. It doesn't — each process has its own independent cache.
• Caching aggregation endpoints instead of fixing the underlying query. Masks the real performance problem instead of solving it.

Exercises#

1. Add lru_cache to create_embedding (with the tuple-return adjustment shown above) and update search_agent_documents and add_chunk_to_vector_store to work with the new signature. Measure the difference in wall-clock time for re-searching an identical query string.
2. Explain, concretely, why caching generate_rag_answer's output (not just the embedding lookup) by (question, contexts) would be safe or unsafe given its temperature=0.2 setting — is 0.2 "deterministic enough" to cache?
3. Sketch what would need to change (new dependency, new environment variable, new failure mode to handle) to move from functools.lru_cache to a Redis-backed cache shared across multiple server instances.

Key Takeaways#

🎯 No caching layer exists anywhere in this backend today. 🎯 Repeated identical search queries and duplicate document uploads are the clearest, lowest-risk caching opportunities — both hit the deterministic create_embedding function. 🎯 Chat-completion calls with temperature > 0 are poor caching candidates — caching them changes correctness, not just speed. 🎯 An in-process cache (lru_cache) doesn't help once there's more than one server process; that requires a shared cache like Redis.

Next Steps#

👉 Lesson 27: Configuration & Environment Variables →

Module 6 · Production Mastery