Module 3 · Services Deep DiveIntermediate11 min read · 2,217 wordsservicesopenaiai

Lesson 08a: OpenAI Service Deep Dive

"Every AI interaction starts here."

Introduction

openai_service.py is the bridge between your backend and OpenAI's API. Every chat message, every email analysis, every RAG answer flows through these functions.

This lesson explains every single line of code in this critical service.

File Overview

Location: backend/app/services/openai_service.py
Size: 6,464 bytes
Functions: 4
Purpose: Wrap OpenAI API calls with business logic

Complete File Walkthrough

Imports

python

import os
import json

import os - Environment variable access for API keys
import json - Parse LLM JSON responses

python

from dotenv import load_dotenv
from openai import OpenAI

load_dotenv - Load .env file variables
OpenAI - Official OpenAI Python client

Environment Setup

python

load_dotenv()

What it does: Reads backend/.env file and loads into os.environ

Why needed: API keys should never be hardcoded

python

client = OpenAI(
    api_key=os.getenv("OPENAI_API_KEY")
)

Creates global OpenAI client with API key from environment

Why global?: Reused across all requests, connection pooling handled internally

python

OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-4o-mini")

Default model: Falls back to gpt-4o-mini if not specified

Why this model?:

Cost-effective ($0.15/1M input tokens vs $5.00 for GPT-4)
Fast (~500ms vs ~2s for GPT-4)
Sufficient quality for most tasks

Function 1: generate_agent_response()

Purpose

Generate AI responses for chat interactions.

Complete Code with Explanation

python

def generate_agent_response(
    system_prompt: str,
    user_message: str,
    agent_name: str,
    language: str = "English",
    tone: str = "Professional"
) -> str:

Parameters:

system_prompt: Core AI instructions (from Agent.system_prompt)
user_message: User's question/message
agent_name: Agent display name
language: Response language (default: English)
tone: Communication style (default: Professional)

Returns: str - AI-generated response

Step 1: API Key Validation

python

if not os.getenv("OPENAI_API_KEY"):
    raise ValueError("OPENAI_API_KEY is missing. Please add it to your .env file.")

Why check here?

Fail fast with clear error message
Prevent cryptic OpenAI API errors
Guide developers to solution

Better than:

python

# Bad - lets OpenAI SDK raise confusing error
response = client.chat...  # → "Authentication failed"

Step 2: Prompt Construction

python

final_system_prompt = f"""
You are {agent_name}.
 
Agent instructions:
{system_prompt}
 
Language:
{language}
 
Tone:
{tone}
 
Rules:
- Answer clearly.
- Be helpful and practical.
- If you do not know the answer, say you do not know.
- Do not invent facts.
"""

Prompt Engineering Breakdown:

Identity: You are {agent_name}
- Gives AI a persona
- Improves role consistency
Instructions: {system_prompt}
- User-defined behavior
- Core agent configuration
Language: Explicit specification
- Prevents language mixing
- Ensures consistent output
Tone: Communication style
- Professional, Friendly, Technical, etc.
- Affects word choice and formality
Rules: Safety guidelines
- Prevents hallucinations
- Encourages honesty
- Maintains quality

Why this structure?:

Clear hierarchy (identity → instructions → constraints)
Explicit rather than implied
Reduces ambiguity

Step 3: API Call

python

response = client.chat.completions.create(
    model=OPENAI_MODEL,
    messages=[
        {
            "role": "system",
            "content": final_system_prompt
        },
        {
            "role": "user",
            "content": user_message
        }
    ],
    temperature=0.3
)

Breaking down the call:

model=OPENAI_MODEL

Which AI model to use
Loaded from environment (flexibility)
Default: gpt-4o-mini

messages=[...]

Conversation format
Array of message objects

Message 1 - System:

python

{
    "role": "system",
    "content": final_system_prompt
}

Sets AI behavior
Not visible to end users
Processed first by model

Message 2 - User:

python

{
    "role": "user",
    "content": user_message
}

The actual question/input
What user typed

temperature=0.3

Controls randomness
Range: 0.0 (deterministic) to 2.0 (very random)
0.3 = consistent, focused responses
Good for customer support, documentation

Why 0.3?:

Creative enough for natural language
Consistent enough for reliability
Reduces hallucinations

Temperature guide:

0.0-0.3: Factual, consistent (support, docs)
0.4-0.7: Balanced (general chat)
0.8-1.0: Creative (storytelling, brainstorming)
1.0+: Very creative (experimental)

Step 4: Extract Response

python

return response.choices[0].message.content

Response structure:

python

{
  "id": "chatcmpl-...",
  "object": "chat.completion",
  "created": 1234567890,
  "model": "gpt-4o-mini",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "This is the AI response..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 50,
    "completion_tokens": 30,
    "total_tokens": 80
  }
}

choices[0]: First (and usually only) response
.message.content: The actual text

Why [0]?: OpenAI can return multiple completions (n parameter), we always request n=1

Complete Flow Example

python

# Input
generate_agent_response(
    system_prompt="You are a helpful Python tutor",
    user_message="What is a list comprehension?",
    agent_name="PyTutor",
    language="English",
    tone="Friendly"
)
 
# Internal: System prompt constructed
"""
You are PyTutor.
 
Agent instructions:
You are a helpful Python tutor
 
Language:
English
 
Tone:
Friendly
 
Rules:
...
"""
 
# API Request
POST https://api.openai.com/v1/chat/completions
{
  "model": "gpt-4o-mini",
  "messages": [...],
  "temperature": 0.3
}
 
# API Response (1.2s later)
{
  "choices": [{
    "message": {
      "content": "A list comprehension is a concise way..."
    }
  }]
}
 
# Output
"A list comprehension is a concise way..."

Function 2: analyze_email()

Purpose

Extract structured data from unstructured email text.

Complete Code with Explanation

python

def analyze_email(email_text: str) -> dict:

Purpose: Convert free-form email → structured JSON
Use case: Extract action items, deadlines, priority from emails

Step 1: API Key Check

python

if not os.getenv("OPENAI_API_KEY"):
    raise ValueError("OPENAI_API_KEY is missing...")

Same validation as before - fail fast pattern.

Step 2: Specialized System Prompt

python

system_prompt = """
You are an email operations assistant.
 
Analyze the email and return ONLY valid JSON with this exact structure:
{
  "summary": "short summary of the email",
  "intent": "main intent of the sender",
  "priority": "low | medium | high",
  "deadline": "deadline if mentioned, otherwise null",
  "action_items": ["action item 1", "action item 2"],
  "suggested_reply": "professional email reply"
}
 
Rules:
- Return only JSON.
- Do not include markdown.
- Do not include explanations.
- priority must be one of: low, medium, high.
- If there is no deadline, use null.
"""

Key differences from chat:

Structured Output: JSON format enforced
No markdown: Pure JSON only
Strict schema: Exact fields required
Enum values: priority limited to 3 options
Null handling: Explicit for missing data

Why this prompt works:

Clear format specification
Examples of enum values
Explicit about what NOT to include
Handles edge cases (no deadline)

Common LLM mistakes this prevents:

python

# Bad - would break JSON parsing
```json
{
  "summary": "..."
}

Good - pure JSON

{ "summary": "..." }

 
---
 
### Step 3: Lower Temperature
 
```python
response = client.chat.completions.create(
    model=OPENAI_MODEL,
    messages=[
        {
            "role": "system",
            "content": system_prompt
        },
        {
            "role": "user",
            "content": email_text
        }
    ],
    temperature=0.2  # ← Lower than chat (0.3)
)

temperature=0.2 - Why lower?:

More deterministic output
Better JSON format compliance
Reduces creative interpretation
Prioritizes accuracy over variety

Step 4: Parse JSON Response

python

raw_text = response.choices[0].message.content
 
try:
    return json.loads(raw_text)
except json.JSONDecodeError:
    raise ValueError(f"OpenAI did not return valid JSON: {raw_text}")

Why try-except?:

LLMs occasionally fail to follow instructions
Better error message for debugging
Shows actual malformed response

Error handling pattern:

python

# Bad - silent failure
result = json.loads(raw_text)  # Crashes with cryptic error
 
# Good - informative failure
try:
    return json.loads(raw_text)
except json.JSONDecodeError:
    raise ValueError(f"OpenAI did not return valid JSON: {raw_text}")
    # Shows WHAT went wrong and WHY

Complete Example

python

# Input
analyze_email("""
Subject: Urgent: Database migration
Hi team, we need to migrate the production database to PostgreSQL
by Friday. Please prepare backup scripts and test in staging first.
This is high priority.
""")
 
# API processes for ~1.5s
 
# Output
{
  "summary": "Request to migrate production database to PostgreSQL",
  "intent": "request_action",
  "priority": "high",
  "deadline": "Friday",
  "action_items": [
    "Prepare backup scripts",
    "Test migration in staging",
    "Migrate production database to PostgreSQL"
  ],
  "suggested_reply": "Thanks for flagging this..."
}

Function 3: generate_rag_answer()

Purpose

Generate answers using retrieved document context (RAG).

Complete Code

python

def generate_rag_answer(
    question: str,
    contexts: list[dict],
    agent_name: str,
    language: str = "English",
    tone: str = "Professional"
) -> str:

RAG = Retrieval-Augmented Generation:

Retrieve relevant document chunks
Augment prompt with that context
Generate answer based on evidence

Step 1: Build Context String

python

context_text = ""
 
for index, item in enumerate(contexts, start=1):
    context_text += f"""
Source {index}
Filename: {item.get("filename")}
Chunk Index: {item.get("chunk_index")}
Content:
{item.get("chunk_text")}
---
"""

Formats retrieved chunks:

Source 1
Filename: company_policies.txt
Chunk Index: 5
Content:
Employees receive 20 days of paid vacation...
---
 
Source 2
Filename: company_policies.txt
Chunk Index: 6
Content:
Sick leave is provided separately...
---

Why number sources?:

Easy to reference in answer
Maintains traceability
Shows AI which source to cite

Step 2: Constrained System Prompt

python

system_prompt = f"""
You are {agent_name}.
 
You answer questions using ONLY the provided context.
 
Language:
{language}
 
Tone:
{tone}
 
Rules:
- Use only the provided context.
- Do not invent information.
- If the answer is not found in the context, say:
  "I could not find this information in the uploaded documents."
- Keep the answer clear and concise.
"""

Critical constraints:

"ONLY the provided context" - Prevents hallucinations
Explicit refusal text - Honest about limitations
No invention - Grounded in evidence

Why RAG needs stricter prompts:

LLMs are trained on vast knowledge
Natural tendency to use that knowledge
RAG requires ignoring training data
Must cite ONLY provided sources

Step 3: Combined Prompt

python

user_prompt = f"""
Context:
{context_text}
 
Question:
{question}
 
Answer:
"""

Structure:

Evidence first (context)
Question second
Answer: placeholder (guides response)

Why this order?:

Context primes the model
Question focuses attention
"Answer:" triggers generation

Step 4: Low Temperature for Accuracy

python

response = client.chat.completions.create(
    model=OPENAI_MODEL,
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt}
    ],
    temperature=0.2  # Low for factual accuracy
)
 
return response.choices[0].message.content

temperature=0.2 - Why?:

RAG prioritizes accuracy over creativity
Reduces paraphrasing that changes meaning
More direct quotes from context

Complete RAG Example

python

# Input
generate_rag_answer(
    question="What is the vacation policy?",
    contexts=[
        {
            "filename": "hr_policy.txt",
            "chunk_index": 3,
            "chunk_text": "Employees receive 20 days of paid vacation annually."
        },
        {
            "filename": "hr_policy.txt",
            "chunk_index": 4,
            "chunk_text": "Unused vacation days can be carried over up to 5 days."
        }
    ],
    agent_name="HR Assistant"
)
 
# Output
"According to the company policy, employees receive 20 days of paid
vacation annually. Additionally, up to 5 unused vacation days can be
carried over to the next year."

Notice: Answer directly quotes the context, doesn't add information.

Function 4: evaluate_retrieved_context()

Purpose

Check if retrieved chunks actually contain the answer (quality control).

The Problem

python

# User asks: "What is the return policy?"
 
# Retrieved chunks:
# 1. "Our products are high quality..."
# 2. "We ship within 24 hours..."
# 3. "Customer satisfaction is our priority..."
 
# Problem: None of these answer the question!
# Solution: Evaluate BEFORE generating answer

Complete Code

python

def evaluate_retrieved_context(
    question: str,
    contexts: list[dict]
) -> dict:

Returns:

python

{
  "has_answer": bool,      # True if answer is in context
  "confidence": float,     # 0.0 to 1.0
  "reason": str           # Why this evaluation
}

Step 1: Build Context String

python

context_text = ""
 
for index, item in enumerate(contexts, start=1):
    context_text += f"""
Source {index}
Filename: {item.get("filename")}
Chunk Index: {item.get("chunk_index")}
Content:
{item.get("chunk_text")}
---
"""

Same format as generate_rag_answer() - consistency.

Step 2: Evaluator System Prompt

python

system_prompt = """
You are a retrieval evaluator for a RAG system.
 
Your job is to check if the provided context contains enough
information to answer the user's question.
 
Return ONLY valid JSON with this exact structure:
{
  "has_answer": true,
  "confidence": 0.0,
  "reason": "short explanation"
}
 
Rules:
- has_answer must be true only if the context directly answers the question.
- confidence must be between 0 and 1.
- If the context is related but does not answer the question, use has_answer=false.
- Do not answer the user's question.
- Return only JSON.
"""

Key instruction: "Do not answer the user's question"

Evaluator ONLY checks, doesn't answer
Prevents mixing concerns
Clearer separation of responsibility

Step 3: Evaluation Call

python

response = client.chat.completions.create(
    model=OPENAI_MODEL,
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt}
    ],
    temperature=0  # ← Maximum determinism
)

temperature=0 - Why zero?:

Binary decision (has answer or doesn't)
No creativity needed
Maximum consistency

Step 4: Parse and Validate

python

raw_text = response.choices[0].message.content
 
try:
    result = json.loads(raw_text)
except json.JSONDecodeError:
    raise ValueError(f"Evaluator did not return valid JSON: {raw_text}")
 
return {
    "has_answer": bool(result.get("has_answer")),
    "confidence": float(result.get("confidence", 0)),
    "reason": result.get("reason", "")
}

Type coercion:

bool() - Ensures boolean
float() - Ensures numeric confidence
.get(default) - Handles missing keys

Evaluation Examples

Example 1: Good Retrieval

python

evaluate_retrieved_context(
    question="What is the refund policy?",
    contexts=[{
        "chunk_text": "Refunds are provided within 30 days of purchase..."
    }]
)
 
# Output
{
  "has_answer": true,
  "confidence": 0.95,
  "reason": "Context directly states the 30-day refund policy"
}

Example 2: Poor Retrieval

python

evaluate_retrieved_context(
    question="What is the refund policy?",
    contexts=[{
        "chunk_text": "We offer excellent customer service..."
    }]
)
 
# Output
{
  "has_answer": false,
  "confidence": 0.1,
  "reason": "Context mentions service but does not address refunds"
}

Key Takeaways

🎯 Every OpenAI call goes through these 4 functions

🎯 Temperature varies by use case: 0.0 (evaluation) → 0.2 (RAG) → 0.3 (chat)

🎯 Prompt engineering is critical: Clear structure, explicit rules, handle edge cases

🎯 Error handling matters: Catch JSON parsing errors, validate API keys early

🎯 RAG requires constraints: "ONLY use context" prevents hallucinations

🎯 Evaluation prevents bad answers: Check quality before generating

Interview Questions

Q: Why is temperature 0.3 for chat but 0.2 for email analysis?
A: Email analysis requires structured JSON output, so lower temperature ensures better format compliance. Chat allows slight creativity for natural conversation.

Q: What happens if OpenAI returns malformed JSON in analyze_email()?
A: The try-except block catches it and raises a ValueError with the actual malformed response for debugging.

Q: Why does RAG use "ONLY the provided context" in the prompt?
A: LLMs have vast training knowledge and naturally want to use it. RAG requires grounding answers in specific documents, so we explicitly constrain the model.

Q: What is the purpose of evaluate_retrieved_context()?
A: It's a quality gate that checks if retrieved chunks actually contain the answer before generating a response. Prevents wasting tokens on bad retrievals.

👉 Next: Monitoring Service →

Lesson 08a Complete!

PreviousModule Overview NextLesson 08b - Monitoring Service

Lesson 08a: OpenAI Service Deep Dive#

Introduction#

File Overview#

Complete File Walkthrough#

Imports#

Environment Setup#

Function 1: generate_agent_response()#

Purpose#

Complete Code with Explanation#

Step 1: API Key Validation#

Step 2: Prompt Construction#

Step 3: API Call#

Step 4: Extract Response#

Complete Flow Example#

Function 2: analyze_email()#

Purpose#

Complete Code with Explanation#

Step 1: API Key Check#

Step 2: Specialized System Prompt#

Good - pure JSON#

Step 4: Parse JSON Response#

Complete Example#

Function 3: generate_rag_answer()#

Purpose#

Complete Code#

Step 1: Build Context String#

Step 2: Constrained System Prompt#

Step 3: Combined Prompt#

Step 4: Low Temperature for Accuracy#

Complete RAG Example#

Function 4: evaluate_retrieved_context()#

Purpose#

The Problem#

Complete Code#

Step 1: Build Context String#

Step 2: Evaluator System Prompt#

Step 3: Evaluation Call#

Step 4: Parse and Validate#

Evaluation Examples#

Example 1: Good Retrieval#

Example 2: Poor Retrieval#

Key Takeaways#

Interview Questions#

Lesson 08a: OpenAI Service Deep Dive

Introduction

File Overview

Complete File Walkthrough

Imports

Environment Setup

Function 1: generate_agent_response()

Purpose

Complete Code with Explanation

Step 1: API Key Validation

Step 2: Prompt Construction

Step 3: API Call

Step 4: Extract Response

Complete Flow Example

Function 2: analyze_email()

Purpose

Complete Code with Explanation

Step 1: API Key Check

Step 2: Specialized System Prompt

Good - pure JSON

Step 4: Parse JSON Response

Complete Example

Function 3: generate_rag_answer()

Purpose

Complete Code

Step 1: Build Context String

Step 2: Constrained System Prompt

Step 3: Combined Prompt

Step 4: Low Temperature for Accuracy

Complete RAG Example

Function 4: evaluate_retrieved_context()

Purpose

The Problem

Complete Code

Step 1: Build Context String

Step 2: Evaluator System Prompt

Step 3: Evaluation Call

Step 4: Parse and Validate

Evaluation Examples

Example 1: Good Retrieval

Example 2: Poor Retrieval

Key Takeaways

Interview Questions