Skip to content
3 min read · 641 words

RAG Ingestion

One HTTP upload becomes N chunks, N embeddings, N vectors, and N+1 database rows. This page follows a file through the whole pipeline.

Pipeline

Rendering diagram…

Stage by stage

1 · Validation (fail fast, fail 400)

GuardRejection
filename ends with .txt400 Only .txt files are supported in this step
body decodes as UTF-8400 File must be UTF-8 encoded text
chunking produced ≥ 1 chunk400 File is empty or could not be processed

2 · Chunking strategy

split_text_into_chunks(text, chunk_size=800, overlap=100) — a character sliding window advancing 700 chars per step:

chars:   0        700  800      1400 1500      2200
chunk 0  [==============]
chunk 1           [===============]
chunk 2                     [===============]

Why these numbers: 800 chars ≈ 200 tokens — small enough that 3 retrieved chunks fit comfortably in a prompt, large enough to carry a complete thought; the 100-char overlap keeps sentences that straddle a boundary retrievable from either side. Trade-offs and alternatives (token-aware, semantic, recursive splitting) are deliberately deferred — the chunker has zero dependencies and predictable behavior (Design Decisions).

3 · Identity scheme

Each chunk's vector ID is deterministic and human-decodable:

agent_{agent_id}_document_{document_id}_chunk_{chunk_index}
e.g.  agent_1_document_7_chunk_3

The same string is stored on the SQLite DocumentChunk.vector_id, making every vector traceable back to its row and vice versa.

4 · The dual write

StoreWhat is writtenRole
SQLite documentsfilename, file_type="txt", status="indexed", chunks_countcatalog entry
SQLite document_chunksfull chunk text, index, vector_id, denormalized agent_idsource of truth (could rebuild Chroma)
Chroma agent_documentsembedding + chunk text + metadata {agent_id, document_id, filename, chunk_index}the searchable index

The metadata mirrors exactly what retrieval needs to return sources without touching SQLite.

Inspecting the result

bash
curl http://localhost:8001/agents/1/documents   # catalog: status, chunks_count
curl http://localhost:8001/agents/1/chunks      # every stored chunk with text and index

Edge cases & sharp edges

The Document row commits before embedding starts. If OpenAI fails at chunk k of N, the request 500s leaving: document row present (claiming chunks_count=N), vectors 0..k-1 in Chroma, and no chunk rows (rolled back). Practical recovery: re-upload (a new document_id means no ID collisions). Fix ideas — commit document last, or set status="indexing" → "indexed" — are on the Roadmap.

  • Costs are invisible: ingestion makes N embedding calls but logs no ToolCall, so upload spend never reaches the dashboard (coverage map).
  • No size limit: a huge file = huge sequential embedding loop = long request + real spend (Security).
  • No deduplication: uploading the same file twice indexes it twice; both copies will be retrieved.
  • Deletion doesn't exist: removing documents/agents leaves vectors in Chroma (Known Issues).

Performance

Upload latency ≈ N × (one embedding round trip, ~100–300 ms). A 40 KB file → ~57 chunks → 6–17 s. The obvious optimization — embeddings.create accepts a list of inputs — would collapse this to one or two calls.

PreviousRAG OverviewNextRetrieval