Skip to content
Module 5 · Advanced AI EngineeringAdvanced4 min read · 743 words

Lesson 15: Embeddings

"A vector is just a list of numbers that happens to mean something."

What an Embedding Is, Concretely in This Codebase

python
def create_embedding(text: str) -> list[float]:
    response = openai_client.embeddings.create(model=EMBEDDING_MODEL, input=text)
    return response.data[0].embedding

Call this with any string and you get back a Python list[float]. That's the entirety of what "embedding" means at the code level — a text-in, numbers-out function backed by an OpenAI API call. The interesting part is what those numbers represent: a point in a high-dimensional space, positioned so that texts with similar meaning end up at nearby points, regardless of whether they share any of the same words.

The Model and Its Real Dimensionality

python
EMBEDDING_MODEL = os.getenv("OPENAI_EMBEDDING_MODEL", "text-embedding-3-small")

The default model is text-embedding-3-small, which produces 1536-dimensional vectors.

An earlier version of this course's glossary said 384 dimensions — that number belongs to a different, smaller sentence-transformer model family, not to text-embedding-3-small. This lesson and the glossary have been corrected.

text-embedding-3-small supports an optional dimensions parameter to truncate the vector for storage savings — this codebase does not pass it, so every embedding is the full 1536-length vector.

Where Embeddings Are Created — Exactly Two Call Sites

Every embedding in this system originates from the same create_embedding() function, called from exactly two places:

  1. add_chunk_to_vector_store() — once per document chunk, at ingestion.
  2. search_agent_documents() — once per query, at retrieval.

There is no batch-embedding call anywhere (OpenAI's embeddings endpoint accepts a list of strings in one request, which would let ingestion embed all of a document's chunks in a single API call instead of one-per-chunk — this codebase doesn't use that capability).

Symmetry Is Not Optional

For two embeddings to be meaningfully comparable, they must come from the same model. Nothing in ChromaDB or this codebase enforces that at write time — if OPENAI_EMBEDDING_MODEL changes between ingesting document A and querying against it, create_embedding will happily produce a vector, Chroma will happily compute a distance against it, and the result will be numerically valid but semantically meaningless. There's no version tag stored alongside each embedding recording which model produced it.

Why This Costs Money and Time, Not Just Compute

Every embedding call is a network round-trip to OpenAI, billed per token. For a document chunked into N pieces, ingestion costs N embedding calls. estimate_openai_cost() (Lesson 08b) is never actually applied to embedding calls in this codebase — cost tracking via ToolCall.estimated_cost only covers chat-completion calls (generate_agent_response, analyze_email, the multi-agent workflow steps), not embeddings. Embedding cost is currently invisible in the dashboard.

Distance, Not Similarity — Read This Before Lesson 16

Chroma's .query() returns distances, and document_service.search_agent_documents() passes that value straight through under the field name score:

python
output.append({"chunk_text": doc, "score": distance, ...})

Smaller distance = more similar. If you build a UI that sorts by score descending expecting "highest score first, most relevant," you'll get the least relevant chunks first. This single naming choice is worth internalizing before writing any code against /documents/search.

Common Mistakes

  • Assuming 384 dimensions. The real number is 1536 for the model this codebase actually uses.
  • Assuming embedding cost shows up in the dashboard. It doesn't — only chat-completion ToolCalls carry estimated_cost.
  • Re-embedding identical text repeatedly. Every call to create_embedding hits the network, even for text that was embedded moments ago — there's no cache. See Lesson 26: Caching.

Exercises

  1. Call create_embedding("hello world") twice and compare the two returned vectors. Are they identical? What does that tell you about whether the embedding model is deterministic?
  2. Extend estimate_openai_cost() (or write a parallel function) to also estimate embedding cost, using OpenAI's published per-token price for text-embedding-3-small, and wire it into add_chunk_to_vector_store.
  3. Explain why cosine similarity and Euclidean (L2) distance often rank the same set of vectors identically for normalized embeddings, but can disagree for unnormalized ones.

Key Takeaways

🎯 text-embedding-3-small produces 1536-dimensional vectors, not 384. 🎯 Ingestion and query embeddings must come from the same model to be comparable. 🎯 Embedding cost is currently untracked in this codebase's cost-monitoring system. 🎯 The score field returned by search is a distance — lower means more similar.

Next Steps

👉 Lesson 16: Vector Databases & ChromaDB →

Module 5 · Advanced AI Engineering

PreviousLesson 14 - RAG PipelineNextLesson 16 - Vector Databases & ChromaDB