Workflow 002: Model Comparison Pipeline
Overview
| Property | Value |
|---|---|
| Workflow ID | 002 |
| Title | Model Comparison Pipeline |
| Complexity | Medium |
| Duration | Depends on models and dataset |
Purpose
Systematically compare different models (GPT-5, GPT-4.1, GPT-5-mini) on the same evaluation criteria to make informed model selection decisions.
Architecture
┌─────────────────────────────────────────────────────────────────────┐
│ MODEL COMPARISON PIPELINE │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ SAME DATASET │ │
│ │ SAME EVAL │ │
│ │ SAME PROMPT │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────┼─────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ GPT-5 │ │ GPT-4.1 │ │ GPT-5-mini │ │
│ │ Agent │ │ Agent │ │ Agent │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Run Eval │ │ Run Eval │ │ Run Eval │ │
│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │
│ │ │ │ │
│ └─────────────────┬──┴──┬─────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────┐ │
│ │ COMPARE │ │
│ │ RESULTS │ │
│ └─────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────┐ │
│ │ SELECT BEST │ │
│ │ MODEL │ │
│ └─────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Prerequisites
- Completed SOP-001: Environment Setup
- Completed SOP-003: Eval Creation
- Completed SOP-004: Baseline Agent Setup
- Access to multiple OpenAI models
Model Selection Criteria
┌─────────────────────────────────────────────────────────────────────┐
│ MODEL SELECTION MATRIX │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Model │ Speed │ Cost │ Quality │ Best For │
│ ─────────────┼─────────┼─────────┼─────────┼───────────────────── │
│ GPT-5 │ Slower │ Higher │ Highest │ Critical production │
│ GPT-4.1 │ Fast │ Medium │ High │ Default choice │
│ GPT-5-mini │ Fastest │ Lowest │ Good │ High-volume, budget │
│ │
│ Decision Framework: │
│ ├─► Need highest quality? → GPT-5 │
│ ├─► Balanced needs? → GPT-4.1 (recommended default) │
│ └─► High volume, cost sensitive? → GPT-5-mini │
│ │
└─────────────────────────────────────────────────────────────────────┘Implementation
Step 1: Configure Models to Compare
python
from dataclasses import dataclass
from typing import Optional
@dataclass
class ModelConfig:
"""Configuration for a model being tested."""
name: str
model_id: str
description: str
cost_per_1k_tokens: float # Approximate input cost
MODELS_TO_COMPARE = [
ModelConfig(
name="GPT-5",
model_id="gpt-5",
description="Most capable, highest quality",
cost_per_1k_tokens=0.03
),
ModelConfig(
name="GPT-4.1",
model_id="gpt-4.1",
description="Fast and capable, good balance",
cost_per_1k_tokens=0.002
),
ModelConfig(
name="GPT-5-mini",
model_id="gpt-5-mini",
description="Fastest, most economical",
cost_per_1k_tokens=0.0003
),
]Step 2: Create Model-Specific Agents
python
from agents import Agent
def create_agent_for_model(model_config: ModelConfig, prompt: str) -> Agent:
"""Create an agent for a specific model."""
return Agent(
name=f"SummarizationAgent-{model_config.name}",
instructions=prompt,
model=model_config.model_id,
)
# Use the same optimized prompt for all models
COMPARISON_PROMPT = """You are a domain-aware summarization assistant for technical texts.
Given a section of text, produce a concise, single-paragraph summary that preserves
key technical facts and exact nomenclature.
Requirements:
- Write 1-3 sentences totaling 45-70 words
- Use one paragraph with no bullets or headings
- Include every chemical name exactly as written
- Preserve all technical terminology
"""
# Create agents for each model
agents = {
config.name: create_agent_for_model(config, COMPARISON_PROMPT)
for config in MODELS_TO_COMPARE
}Step 3: Run Parallel Evaluation
python
import asyncio
from agents import Runner
import time
@dataclass
class ModelResult:
"""Results for a single model."""
model_name: str
scores: list[float]
average_score: float
pass_rate: float
total_time: float
estimated_cost: float
async def evaluate_model(
model_config: ModelConfig,
agent: Agent,
sections: list[str],
eval_id: str
) -> ModelResult:
"""Evaluate a single model on all sections."""
scores = []
passed_count = 0
start_time = time.time()
total_tokens = 0
for section in sections:
# Generate summary
result = await Runner.run(agent, section)
summary = result.final_output
# Estimate tokens (rough approximation)
total_tokens += len(section.split()) + len(summary.split())
# Run evaluation
eval_run = run_eval(eval_id=eval_id, section=section, summary=summary)
run_output = poll_eval_run(eval_id=eval_id, run_id=eval_run.id)
grader_scores = parse_eval_run_output(run_output)
# Calculate score
avg_score = calculate_grader_score(grader_scores)
scores.append(avg_score)
if is_lenient_pass(grader_scores, avg_score):
passed_count += 1
total_time = time.time() - start_time
estimated_cost = (total_tokens / 1000) * model_config.cost_per_1k_tokens
return ModelResult(
model_name=model_config.name,
scores=scores,
average_score=sum(scores) / len(scores) if scores else 0,
pass_rate=passed_count / len(sections) if sections else 0,
total_time=total_time,
estimated_cost=estimated_cost
)
async def run_comparison(sections: list[str], eval_id: str) -> list[ModelResult]:
"""Run comparison across all models."""
results = []
for config in MODELS_TO_COMPARE:
print(f"\nEvaluating {config.name}...")
result = await evaluate_model(
model_config=config,
agent=agents[config.name],
sections=sections,
eval_id=eval_id
)
results.append(result)
print(f" Average Score: {result.average_score:.3f}")
print(f" Pass Rate: {result.pass_rate:.1%}")
print(f" Time: {result.total_time:.1f}s")
print(f" Est. Cost: ${result.estimated_cost:.4f}")
return resultsStep 4: Generate Comparison Report
python
def generate_comparison_report(results: list[ModelResult]) -> str:
"""Generate a formatted comparison report."""
report = []
report.append("=" * 70)
report.append("MODEL COMPARISON REPORT")
report.append("=" * 70)
report.append("")
# Summary table
report.append("SUMMARY")
report.append("-" * 70)
report.append(f"{'Model':<15} {'Avg Score':<12} {'Pass Rate':<12} {'Time':<10} {'Cost':<10}")
report.append("-" * 70)
for r in sorted(results, key=lambda x: x.average_score, reverse=True):
report.append(
f"{r.model_name:<15} {r.average_score:<12.3f} {r.pass_rate:<12.1%} "
f"{r.total_time:<10.1f}s ${r.estimated_cost:<10.4f}"
)
report.append("")
# Recommendation
best_quality = max(results, key=lambda x: x.average_score)
best_value = max(results, key=lambda x: x.average_score / (x.estimated_cost + 0.0001))
fastest = min(results, key=lambda x: x.total_time)
report.append("RECOMMENDATIONS")
report.append("-" * 70)
report.append(f"Best Quality: {best_quality.model_name} (score: {best_quality.average_score:.3f})")
report.append(f"Best Value: {best_value.model_name}")
report.append(f"Fastest: {fastest.model_name} ({fastest.total_time:.1f}s)")
report.append("")
report.append("=" * 70)
return "\n".join(report)Step 5: Run Complete Pipeline
python
async def main():
"""Run the complete model comparison pipeline."""
import pandas as pd
# Load dataset
df = pd.read_csv("data/sections.csv")
sections = df["content"].tolist()[:20] # Use subset for comparison
EVAL_ID = "eval_..." # Your eval ID
print("Starting Model Comparison Pipeline")
print(f"Dataset: {len(sections)} sections")
print(f"Models: {[m.name for m in MODELS_TO_COMPARE]}")
# Run comparison
results = await run_comparison(sections, EVAL_ID)
# Generate report
report = generate_comparison_report(results)
print("\n" + report)
# Save report
with open("results/model_comparison.txt", "w") as f:
f.write(report)
return results
if __name__ == "__main__":
asyncio.run(main())Expected Output
Starting Model Comparison Pipeline
Dataset: 20 sections
Models: ['GPT-5', 'GPT-4.1', 'GPT-5-mini']
Evaluating GPT-5...
Average Score: 0.892
Pass Rate: 95.0%
Time: 45.2s
Est. Cost: $0.1200
Evaluating GPT-4.1...
Average Score: 0.875
Pass Rate: 90.0%
Time: 22.1s
Est. Cost: $0.0080
Evaluating GPT-5-mini...
Average Score: 0.821
Pass Rate: 80.0%
Time: 15.3s
Est. Cost: $0.0012
======================================================================
MODEL COMPARISON REPORT
======================================================================
SUMMARY
----------------------------------------------------------------------
Model Avg Score Pass Rate Time Cost
----------------------------------------------------------------------
GPT-5 0.892 95.0% 45.2s $0.1200
GPT-4.1 0.875 90.0% 22.1s $0.0080
GPT-5-mini 0.821 80.0% 15.3s $0.0012
RECOMMENDATIONS
----------------------------------------------------------------------
Best Quality: GPT-5 (score: 0.892)
Best Value: GPT-4.1
Fastest: GPT-5-mini (15.3s)
======================================================================Decision Framework
┌─────────────────────────────────────────────────────────────────────┐
│ DECISION FRAMEWORK │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Score Difference < 5%? │
│ ├─► YES: Choose faster/cheaper model │
│ └─► NO: Consider quality requirements │
│ │
│ Production Use Case: │
│ ├─► High-stakes (medical, legal): Prefer GPT-5 │
│ ├─► General business: GPT-4.1 recommended │
│ └─► High-volume, low-risk: GPT-5-mini acceptable │
│ │
│ Budget Constraints: │
│ ├─► Unlimited: Choose best quality │
│ ├─► Limited: Calculate cost/quality ratio │
│ └─► Strict: Use cheapest that meets minimum threshold │
│ │
└─────────────────────────────────────────────────────────────────────┘Verification Checklist
- [ ] All models accessible via API
- [ ] Same prompt used across all models
- [ ] Same dataset used for comparison
- [ ] Results include all metrics (score, time, cost)
- [ ] Report generated and saved
- [ ] Recommendation documented