Lesson 19: Prompt Engineering & Tool Calling#

"Six prompts, one style, and zero uses of OpenAI's actual tool-calling API."

Every System Prompt in the Codebase, Catalogued#

There are exactly six distinct system prompts across two service files. Reading them side by side reveals the house style:

Three produce free text meant for a human to read; three produce JSON meant for the backend to parse. This split is intentional and visible in the prompt wording itself — every JSON-output prompt includes the phrase "Return ONLY valid JSON" and an explicit example structure; every free-text prompt has no such constraint.

The Recurring Structure#

Every one of the six prompts follows the same shape: a role statement ("You are ..."), a numbered or bulleted list of rules, and — for the JSON ones — an exact example of the expected structure embedded directly in the prompt text. Compare analyze_email's prompt:

You are an email operations assistant.
 
Analyze the email and return ONLY valid JSON with this exact structure:
{
"summary": "short summary of the email",
"intent": "main intent of the sender",
"priority": "low | medium | high",
"deadline": "deadline if mentioned, otherwise null",
"action_items": ["action item 1", "action item 2"],
"suggested_reply": "professional email reply"
}
 
Rules:
- Return only JSON.
- Do not include markdown.
- Do not include explanations.
- priority must be one of: low, medium, high.
- If there is no deadline, use null.

against run_reviewer_agent's prompt — different domain, identical skeleton (role → JSON shape → rules list). This consistency isn't enforced by a shared template function; each prompt is a separate hardcoded string. If the house style changes, all six need to be edited by hand.

Temperature Choices Are Deliberate, and Consistent With the Task#

The two temperature=0 calls are both evaluator roles (grading retrieval quality, grading workflow output quality) — the codebase consistently treats "make a judgment/score" tasks as wanting minimal randomness, and "produce prose a human reads" tasks as wanting some.

The JSON-Parsing Contract, and What Happens When It's Violated#

Every JSON-output function follows the same pattern:

raw_text = response.choices[0].message.content
try:
    return json.loads(raw_text)
except json.JSONDecodeError:
    raise ValueError(f"... did not return valid JSON: {raw_text}")

There is no retry, no response_format={"type": "json_object"} constraint passed to the OpenAI call (which would make the API itself guarantee valid JSON), and no schema validation of the parsed result beyond evaluate_retrieved_context and run_reviewer_agent manually re-wrapping fields with bool(...) / float(...) coercion. If the model ever returns malformed JSON (rare with modern models, but not impossible under high load or unusual input), the function raises a ValueError that propagates up to the calling router's try/except, gets logged as a failed ToolCall, and surfaces to the client as a 500.

"Tool Calling" — the Two Different Meanings, Resolved#

This is the single most important clarification in this lesson, because the term is ambiguous and this codebase uses only one of the two meanings:

• OpenAI native tool/function calling: passing a tools=[] schema to chat.completions.create and letting the model decide to invoke a named function with structured arguments. Not used anywhere in this codebase. Every chat.completions.create call across openai_service.py and multi_agent_email_service.py omits tools= entirely.
• This codebase's enabled_tools / agent_has_tool() mechanism (Lesson 13): a backend-side authorization flag, checked in plain Python, that gates whether an endpoint is allowed to run at all. The model never sees a list of "tools it can call" — the "tool" concept lives entirely on the FastAPI side.

If a lesson, glossary entry, or interview question in this course says "tool calling" without qualification, it means the second one, because that's the only kind that exists in this codebase.

Common Mistakes#

• Assuming this codebase demonstrates OpenAI function calling. It demonstrates a different, backend-side authorization pattern that happens to reuse the word "tool."
• Assuming malformed-JSON responses are retried. They aren't — one bad response is one failed request, surfaced as a 500.
• Copy-pasting one of the six prompts as a template for a new one and forgetting to update the embedded example JSON structure to match the new output shape.

Exercises#

1. Add response_format={"type": "json_object"} to analyze_email's API call and explain what class of failure this eliminates (and what it does not eliminate — the content of the JSON can still be wrong even if it's guaranteed to parse).
2. Write a seventh system prompt, for a new summarize_document tool gated by a new enabled_tools flag, following the same role/rules/JSON-example structure as the other five JSON-output prompts.
3. Implement real OpenAI-native tool calling for one endpoint (e.g., let the model choose between "search documents" and "answer directly" via a tools= schema) and compare the resulting code and behavior against the current agent_has_tool()-gated approach.

Key Takeaways#

🎯 Six system prompts total, split evenly between free-text and strict-JSON outputs, all following the same role/rules/example skeleton. 🎯 Temperature is chosen per task: near-zero for judgment/scoring, low-but-nonzero for prose. 🎯 JSON parsing has no retry and no API-level format guarantee — one malformed response is one failed request. 🎯 "Tool calling" in this codebase means backend-side authorization gating, not OpenAI's native function-calling API.

Next Steps#

👉 Lesson 20: Memory & Context Management →

Module 5 · Advanced AI Engineering