Lesson 20: Memory & Context Management#

Current state: every message is stored forever, and every chat call sends exactly one message of context — not the conversation history.

The Message Table Is a Transcript, Not "Memory"#

class Message(Base):
    __tablename__ = "messages"
    agent_id = Column(Integer, ForeignKey("agents.id"), nullable=False)
    role = Column(String(20), nullable=False)  # user or assistant
    content = Column(Text, nullable=False)
    created_at = Column(DateTime(timezone=True), server_default=func.now())

Every chat turn writes two rows here — one role="user", one role="assistant" — with no expiry, no archival, no row limit. GET /{agent_id}/messages (the only paginated endpoint in the whole backend, per Lesson 09a) is how a client reads this transcript back.

The Part That Surprises People: chat_with_agent Never Reads This Table#

Look at what POST /{agent_id}/chat actually sends to OpenAI, in generate_agent_response:

response = client.chat.completions.create(
    model=OPENAI_MODEL,
    messages=[
        {"role": "system", "content": final_system_prompt},
        {"role": "user", "content": user_message},
    ],
    temperature=0.3,
)

Exactly two messages: the system prompt and the current user message. There is no step anywhere in chat_with_agent (in routers/agents.py) that queries models.Message for this agent's prior turns and includes them in the messages list sent to OpenAI. Concretely: the model has no memory of anything said earlier in the conversation. Ask it "what did I just ask you?" and it has no way to know — that question and the previous one are two independent, context-free API calls that happen to be stored next to each other in the database afterward.

This is worth sitting with, because the Message table's existence (and its agent_id foreign key, its ordering, its role field) looks exactly like what a conversational-memory system would look like — the schema is fully capable of supporting real multi-turn memory. The gap is entirely in chat_with_agent's implementation: it persists history without ever reading it back.

What Adding Real Memory Would Take#

The minimal fix, grounded in the existing schema and nothing else:

def chat_with_agent(agent_id, chat_data, db=Depends(get_db)):
    agent = ...
    history = (
        db.query(models.Message)
        .filter(models.Message.agent_id == agent_id)
        .order_by(models.Message.created_at.asc())
        .limit(20)  # last 20 turns, or however many fit the context window
        .all()
    )
    messages = [{"role": "system", "content": final_system_prompt}]
    messages += [{"role": m.role, "content": m.content} for m in history]
    messages.append({"role": "user", "content": chat_data.message})
    response = client.chat.completions.create(model=OPENAI_MODEL, messages=messages, temperature=0.3)

This immediately raises the real problem every memory system has to solve: context window limits. GPT-4o-mini's context window is large by older standards but still finite — a conversation with hundreds of turns eventually can't fit in one request. Solving that requires a strategy (sliding window of the last N turns, periodic summarization of older turns, or a hybrid) — none of which exists in this codebase today, because the simpler problem of "sending any history at all" hasn't been solved yet.

Other Places Where "Memory" Is Actually Present#

Two workflows in this codebase do carry context forward, but only within a single request, via local Python variables — not via database reads:

• The multi-agent email workflow (Lesson 12) passes analysis (the output of step 1) into run_reply_agent and run_reviewer_agent as function arguments — this is short-lived, in-memory state, gone the instant the request finishes.
• The Corrective RAG graph (Lesson 08e) threads RAGGraphState through its nodes — same property: lives only for the duration of one .invoke() call.

Neither of these is "memory" in the cross-request sense this lesson is about — they're single-request data flow, which is a different concept worth not conflating with conversational memory.

Common Mistakes#

• Assuming POST /{agent_id}/chat is a real multi-turn conversation because messages are stored in order with an agent_id. Storage and usage are different things — this codebase does the former, not the latter, for chat.
• Assuming the fix is just "send the whole history." Without a windowing or summarization strategy, that breaks the moment a conversation exceeds the model's context window.

Exercises#

1. Implement the history-loading change sketched above, then have a two-turn conversation ("My name is Alex." / "What's my name?") and confirm the second answer is now correct where it previously would have failed.
2. Once history is included, add a limit that keeps only the most recent 10 messages. Write a test conversation that would break the naive "send everything" version but succeed with the windowed version.
3. Sketch (pseudocode is fine) a summarization strategy: once a conversation exceeds 20 messages, collapse the oldest 10 into a single system-message summary. What would you need to store to make that idempotent (not re-summarizing the same messages twice)?

Key Takeaways#

🎯 Message rows are a durable transcript, written on every turn — but chat_with_agent never reads them back into the LLM call. 🎯 As implemented, every chat turn is context-free from the model's perspective; only storage happens, not recall. 🎯 The multi-agent workflow and the RAG graph both carry state within one request — a different, unrelated concept from cross-request conversational memory. 🎯 Adding real memory immediately surfaces the context-window problem, which this codebase has no strategy for yet.

Next Steps#

👉 Lesson 21: Logging & Observability →

Module 5 · Advanced AI Engineering