Skip to content
Module 5 · Advanced AI EngineeringAdvanced5 min read · 902 words

Lesson 14: RAG Pipeline — End to End

"Retrieval-Augmented Generation, traced from file upload to final answer."

Module 5: Advanced AI Engineering

This is the first lesson of Module 5. Lessons 08d, 08e, 09b already covered the individual pieces in isolation — this lesson walks the whole pipeline as one continuous flow, the way a request actually travels through it.

The Six Stages

Rendering diagram…

1 — Upload

POST /agents/{id}/documents/upload in routers/documents.py accepts one .txt file, decodes it as UTF-8, and hands the raw text to the chunker. Covered in Lesson 09b.

2 — Chunk

split_text_into_chunks(text, chunk_size=800, overlap=100) in document_service.py slides an 800-character window across the raw text with a 100-character overlap. Character-based, not sentence- or token-aware. Fully traced in Lesson 08d.

3 — Embed Each Chunk

For every chunk, create_embedding(chunk_text) makes one call to openai_client.embeddings.create(model="text-embedding-3-small", input=chunk_text), producing a 1536-dimensional vector. This happens once per chunk, synchronously, inside the upload request — see Lesson 15: Embeddings for the model details and Lesson 24: Performance for the latency cost.

4 — Store

add_chunk_to_vector_store() writes the chunk text, its embedding, and metadata (agent_id, document_id, filename, chunk_index) into the single shared agent_documents Chroma collection, keyed by a vector_id string. In parallel, a DocumentChunk SQL row is written with the same vector_id, linking the relational and vector worlds. See Lesson 16: Vector Databases & ChromaDB.

5 — Retrieve

At query time (either POST /documents/search or POST /documents/ask), the incoming query string is embedded with the same model, and documents_collection.query(query_embeddings=[...], n_results=top_k, where={"agent_id": agent_id}) returns the top_k nearest chunks, scoped to the requesting agent.

6 — Evaluate, Then Generate or Refuse

This is where the two entry points diverge:

  • /documents/search stops here and returns the raw ranked chunks — no LLM involved beyond embedding the query.
  • /documents/ask hands the retrieved chunks to the LangGraph Corrective RAG graph, which grades whether the chunks actually answer the question before deciding to generate an answer or refuse. See Lesson 17.

Why "Corrective," Not Just "RAG"

Plain RAG collapses stage 6 into a single step: retrieve, then always generate. This codebase's /documents/ask path never does that — evaluate_node sits between retrieval and generation specifically so a bad or empty retrieval produces an honest refusal ("I could not find this information in the uploaded documents.") instead of an LLM improvising an answer from irrelevant context.

One Query, Two Embedding Spaces to Keep in Sync

The pipeline only works if the vector space used to embed chunks at ingestion time and the vector space used to embed the query at retrieval time are the same — same model, same dimensionality. OPENAI_EMBEDDING_MODEL is read once, at import time, in document_service.py. If that environment variable changes between when a document was ingested and when a query runs against it, the two embeddings are no longer comparable — Chroma will still return a result, just not a meaningful one, and nothing in this codebase detects or warns about that mismatch. See Lesson 27: Configuration.

Tracing a Single Document Through Every Table

TableWhat it stores for this pipeline
documentsOne row per uploaded file: filename, status, chunks_count
document_chunksOne row per chunk: chunk_text, chunk_index, vector_id
(ChromaDB, not SQL)The embedding vector + metadata for each vector_id
tool_callsOne row per document_search or langgraph_corrective_rag invocation

Notice the chunk text is duplicated in two places: once in the document_chunks.chunk_text SQL column, and again in ChromaDB's own documents list. This redundancy is deliberate — SQL is the system of record for browsing/auditing chunks (GET /{id}/chunks), while Chroma needs its own copy to return alongside search results without a round-trip back to SQL.

Common Mistakes

  • Changing OPENAI_EMBEDDING_MODEL after documents are already ingested — old and new vectors become incomparable, and nothing surfaces this as an error.
  • Assuming /documents/search and /documents/ask cost the same — ask makes 2–3x the LLM calls (embed query, evaluate, maybe generate) that search does (embed query only).
  • Uploading a document and expecting chunks_count to always match what's actually searchable — see the partial-failure gap in Lesson 09b.

Exercises

  1. Draw the full pipeline again, but for a document that fails halfway through step 4 (embedding call 300 of 625 throws a rate-limit error). Mark exactly which tables end up with data and which don't.
  2. Explain, in your own words, why stage 6 needs a separate LLM call (evaluate_retrieved_context) instead of asking the same generation call to "say if you don't know" — what failure mode does the separate evaluation step catch that a single combined prompt wouldn't?
  3. If you added support for .pdf uploads, which stage(s) would need to change, and which would stay identical?

Key Takeaways

🎯 The pipeline is six stages: upload → chunk → embed → store → retrieve → evaluate-then-generate-or-refuse. 🎯 Ingestion and retrieval must use the same embedding model, or results silently degrade. 🎯 /documents/search and /documents/ask share stages 1–5 but diverge sharply at stage 6.

Next Steps

👉 Lesson 15: Embeddings →

Module 5 · Advanced AI Engineering

PreviousModule OverviewNextLesson 15 - Embeddings