Skip to content
3 min read · 634 words

OpenAI Integration

Three modules construct OpenAI clients; two env vars select the models; one guard pattern makes missing keys explicit. This page is the integration contract.

Client setup (current reality)

Three modules each build their own client at import time:

ModuleClientModels used
services/openai_service.pyOpenAI(api_key=os.getenv("OPENAI_API_KEY"))OPENAI_MODEL for chat/analysis/RAG/eval
services/multi_agent_email_service.pyown client, same patternOPENAI_MODEL for reply/review
services/document_service.pyown clientOPENAI_EMBEDDING_MODEL for embeddings

All three call load_dotenv() first, so backend/app/.env works regardless of which module loads first. The duplication is acknowledged debt — a single shared client factory is the refactor (Roadmap).

Model selection

VariableDefaultNotes
OPENAI_MODELgpt-4o-minione model for all completion roles; swap globally by env var
OPENAI_EMBEDDING_MODELtext-embedding-3-small1536 dims; ingestion and querying must use the same model

Vectors embedded with one model are not comparable to queries embedded with another. If you change OPENAI_EMBEDDING_MODEL, delete backend/app/chroma_db/ and re-upload documents. There is no re-indexing command yet.

Also note: the cost estimator prices everything at gpt-4o-mini rates regardless of OPENAI_MODEL — swap to a pricier model and the dashboard will undercount cost.

The guard pattern

Every completion/embedding function opens with:

python
if not os.getenv("OPENAI_API_KEY"):
    raise ValueError("OPENAI_API_KEY is missing. Please add it to your .env file.")

This converts the SDK's opaque auth failure into an actionable message. Downstream handling varies by caller — chat degrades to a fallback answer, tools and workflows return 500 with the message (Error Handling).

Call anatomy

All completion calls share one shape — messages array with a system + user turn, explicit temperature, no streaming, no tools/functions:

python
response = client.chat.completions.create(
    model=OPENAI_MODEL,
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_content},
    ],
    temperature=T,          # 0 / 0.2 / 0.3 by role
)
text = response.choices[0].message.content

For JSON-producing roles the system prompt enforces structure and json.loads validates it — the platform does not use response_format or function calling yet (Design Decisions).

Conventions for adding a new LLM call

  1. Put it in a service, never a router — routers orchestrate and log; services prompt and call.
  2. Guard the key with the standard ValueError.
  3. Pick a temperature from the ladder: 0 for judging, 0.2 for extraction/grounded generation, 0.3 for user-facing prose.
  4. If you need structure, demand strict JSON in the system prompt, parse with json.loads, raise ValueError with the raw text on failure, and normalize fields defensively (bool(...), float(result.get(..., 0))).
  5. Log a ToolCall in the calling router — with cost data from estimate_openai_cost if it is an LLM step (Observability).
  6. Document the prompt in the Prompt Catalog.

Failure modes at the SDK boundary

FailureRaised asSeen by user as
Missing keyValueError (guard)chat fallback / 500 detail
Invalid key, quota, rate limitopenai SDK exceptionssame paths
Model returns prose instead of JSONValueError("... did not return valid JSON: <raw>")500 with raw output in detail
Network timeoutSDK exception (no custom timeout configured)same paths

No retries, no backoff, no circuit breaker anywhere — a deliberate simplicity that becomes the first hardening task under real traffic.

Data-privacy note

Everything the platform processes — chat messages, full email bodies, uploaded document chunks, questions — is sent to the OpenAI API. Apply your organization's data policy accordingly (Security).

PreviousAI Systems OverviewNextCost Monitoring