Skip to content
Module 5 · Advanced AI EngineeringAdvanced4 min read · 883 words

Lesson 16: Vector Databases & ChromaDB

"SQL answers 'what matches exactly.' A vector database answers 'what means the same thing.'"

Why This Codebase Has Two Databases

SQLite (via SQLAlchemy) stores everything with exact structure: agent IDs, timestamps, statuses. It cannot answer "which of these 500 chunks is most relevant to this question?" — that requires comparing meaning, not values, which is what ChromaDB is for. This backend runs both, side by side, deliberately — relational data in SQLite, semantic data in ChromaDB, linked by the vector_id string stored on DocumentChunk.

Setup, Read Literally From the Code

python
chroma_client = chromadb.PersistentClient(path="./chroma_db")
documents_collection = chroma_client.get_or_create_collection(name="agent_documents")
  • PersistentClient means Chroma writes to disk (backend/app/chroma_db/, per CLAUDE.md) rather than living purely in memory — data survives a server restart.
  • path="./chroma_db" is a relative path, resolved from whatever directory the process was launched from. Run uvicorn from a different working directory and you get a different Chroma store, silently — no error, just data that appears to have vanished.
  • get_or_create_collection means this line is idempotent — safe to run on every server startup, only creates the collection the first time.
  • One collection for the entire application. Not one collection per agent. This is the design decision that makes the where={"agent_id": ...} filter load-bearing — see below.

The Three Chroma Operations This Codebase Uses

python
documents_collection.add(ids=[...], documents=[...], embeddings=[...], metadatas=[...])
documents_collection.query(query_embeddings=[...], n_results=top_k, where={"agent_id": agent_id})

.add() and .query() are the only two Chroma methods called anywhere in the backend — no .update(), no .delete(), no .get() by ID.

Deleting an Agent or a Document in SQL (cascading via cascade="all, delete" on the SQLAlchemy relationships) does not delete the corresponding vectors from Chroma. The cascade only touches SQL tables. Orphaned vectors accumulate in Chroma every time a document or agent with documents is deleted — a real, verifiable resource leak in the current implementation.

add() — Four Parallel Lists

python
documents_collection.add(
    ids=[vector_id],
    documents=[chunk_text],
    embeddings=[embedding],
    metadatas=[metadata],
)

Chroma's API is list-based even when adding a single item — this codebase always passes single-element lists, never batches multiple chunks into one .add() call, even though the upload loop calls this once per chunk and could accumulate all of a document's chunks and add them together. That would cut the number of round-trips to the Chroma client (though not the number of OpenAI embedding calls, which remain per-chunk regardless).

query() — the where Filter Is the Entire Isolation Model

python
results = documents_collection.query(
    query_embeddings=[query_embedding],
    n_results=top_k,
    where={"agent_id": agent_id},
)

Because every agent's chunks live in the same collection, where={"agent_id": agent_id} is the only thing preventing agent A's search results from including agent B's documents. There's no schema-level guarantee here the way there is in SQL (a WHERE agent_id = ? clause that the database engine enforces against a well-typed column) — it's a metadata dictionary match, and it's exactly as strong as every call site remembering to pass it correctly. Every current call site does. See Lesson 25: Security for the consequence if one ever doesn't.

Reading the Response Shape

Chroma's .query() doesn't return a flat list — it returns parallel lists of lists, one outer list per query embedding submitted (this codebase always submits exactly one):

python
documents = results.get("documents", [[]])[0]
metadatas = results.get("metadatas", [[]])[0]
distances = results.get("distances", [[]])[0]
for doc, metadata, distance in zip(documents, metadatas, distances):
    ...

The [0] index unwraps "results for the first (and only) query embedding." zip() then walks the three parallel lists together to rebuild one dict per matched chunk. If you ever batch multiple queries into one .query() call (passing more than one embedding in query_embeddings), this indexing would need to change — it currently assumes exactly one query per call.

Common Mistakes

  • Deleting an agent and assuming its vectors are gone too. They aren't — cascading deletes in SQL don't touch Chroma.
  • Running the server from a different working directory and being confused that a previously-uploaded document's chunks are no longer searchable — you're looking at a different chroma_db folder.
  • Assuming where={"agent_id": ...} is enforced anywhere but the query itself — it's not a schema constraint, it's a per-call filter.

Exercises

  1. Write a cleanup function that Chroma-deletes all vectors for a given agent_id (using documents_collection.delete(where={"agent_id": agent_id})), and call it from wherever an agent is deleted, to close the orphaned-vector gap described above.
  2. Change add_chunk_to_vector_store to accept a list of chunks and call .add() once per document instead of once per chunk. What has to change in the caller (routers/documents.py's upload loop) to build the parallel lists?
  3. Start the backend from two different working directories and confirm, empirically, that you get two separate chroma_db folders with independent data.

Key Takeaways

🎯 One shared Chroma collection for all agents — isolation is a where filter, not physical separation. 🎯 PersistentClient(path="./chroma_db") is a relative path — working directory matters. 🎯 SQL cascading deletes do not clean up Chroma — vectors for deleted agents/documents are orphaned. 🎯 Chroma's .query() response is parallel lists of lists; zip() reassembles it into per-chunk dicts.

Next Steps

👉 Lesson 17: LangGraph & Corrective RAG →

Module 5 · Advanced AI Engineering

PreviousLesson 15 - EmbeddingsNextLesson 17 - LangGraph & Corrective RAG