Quick Reference

Fast lookup for common tasks, commands, and configurations.

Quick Reference Guides

Guide	Description
Commands	API calls and function signatures
Graders	Grader types and configurations
Configuration	Settings and thresholds
Troubleshooting	Common issues and solutions

Quick Start Snippets

Initialize OpenAI Client

from openai import OpenAI
client = OpenAI()  # Uses OPENAI_API_KEY env var

Create Basic Eval

eval_config = client.evals.create(
    name="my-eval",
    data_source_config={"type": "custom", "item_schema": {...}},
    testing_criteria=[grader1, grader2]
)
EVAL_ID = eval_config.id

Run Eval

run = client.evals.runs.create(
    eval_id=EVAL_ID,
    data_source={"type": "jsonl", "source": {...}}
)

Create Agent

from agents import Agent, Runner

agent = Agent(
    name="SummarizationAgent",
    instructions="Your prompt here",
    model="gpt-5"
)

result = await Runner.run(agent, "Input text")
summary = result.final_output

Key Thresholds

Threshold	Value	Description
LENIENT_PASS_RATIO	0.75	Min % of graders that must pass
LENIENT_AVERAGE_THRESHOLD	0.85	Min average score to pass
MAX_OPTIMIZATION_RETRIES	3	Max prompt optimization attempts
WORD_LENGTH_TARGET	80	Target summary word count
WORD_LENGTH_DEVIATION	0.35	Allowed deviation from target
COSINE_THRESHOLD	0.5	Min cosine similarity

Model Selection

Model	Best For	Cost
gpt-5	Highest quality, critical tasks	$$$
gpt-4.1	General use, good balance	$$
gpt-5-mini	High volume, budget	$

Optimization Methods

Method	Use When
Platform UI	Rapid iteration, exploration
Static Metaprompt	Automation, production
GEPA	Robust generalization, multi-objective

Common Patterns

Check Lenient Pass

def is_lenient_pass(grader_scores, average_score):
    passed_count = sum(1 for e in grader_scores if e.get("passed"))
    if (passed_count / len(grader_scores)) >= 0.75:
        return True
    return average_score >= 0.85

Calculate Average Score

def calculate_grader_score(grader_scores):
    return sum(e.get("score", 0) for e in grader_scores) / len(grader_scores)

Version Prompt

from datetime import datetime

class VersionedPrompt:
    def __init__(self, initial_prompt, model="gpt-5"):
        self._versions = [{"version": 0, "prompt": initial_prompt, "model": model}]

    def update(self, new_prompt, model="gpt-5"):
        self._versions.append({
            "version": len(self._versions),
            "prompt": new_prompt,
            "model": model
        })

    def current(self):
        return self._versions[-1]