Optimization Guide

Techniques for reducing cost, improving latency, and enhancing model performance.

Overview

┌─────────────────────────────────────────────────────────────────────┐
│                    OPTIMIZATION STRATEGIES                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  COST REDUCTION              LATENCY REDUCTION                      │
│  ├─► Batch API (50% off)     ├─► Prompt Caching (80% faster)       │
│  ├─► Smaller models          ├─► Parallel execution                 │
│  ├─► Token optimization      ├─► Streaming responses                │
│  └─► Caching strategies      └─► Edge deployments                   │
│                                                                      │
│  QUALITY IMPROVEMENT                                                 │
│  ├─► Fine-tuning (SFT, DPO, RFT)                                   │
│  ├─► Prompt optimization                                            │
│  ├─► Evaluation-driven iteration                                    │
│  └─► RAG enhancement                                                │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Quick Comparison

Strategy	Cost Impact	Latency Impact	Quality Impact	Effort
Batch API	-50%	+24h	None	Low
Prompt Caching	-50% (cached)	-80%	None	Low
Fine-tuning	Variable	-latency	+quality	High
Smaller models	-90%+	-faster	-quality	Low
Parallel execution	None	-faster	None	Medium

Cost Optimization Matrix

┌─────────────────────────────────────────────────────────────────────┐
│                    WHEN TO USE WHAT                                  │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Need results in <1s?                                               │
│  │                                                                  │
│  ├─YES─► Use prompt caching + streaming                            │
│  │                                                                  │
│  └─NO──► Need results in <1 hour?                                  │
│          │                                                          │
│          ├─YES─► Use standard API + caching                        │
│          │                                                          │
│          └─NO──► Use Batch API (50% cost savings)                  │
│                                                                      │
│  Processing >1000 requests?                                         │
│  └─YES─► Batch API is almost always better                         │
│                                                                      │
│  Same prompt prefix >1024 tokens used repeatedly?                   │
│  └─YES─► Prompt caching gives 80% latency reduction                │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Recommended Reading Order

1. Prompt Caching - Easiest win, no code changes
2. Batch Processing - Best for high volume
3. Fine-tuning Techniques - When you need better quality
4. Data Preparation - Before fine-tuning
5. Evaluation Flywheel - Continuous improvement

Quick Wins

Immediate Cost Reduction

# 1. Use appropriate model for task complexity
TASK_MODEL_MAP = {
    "simple_classification": "gpt-4o-mini",     # $0.15/1M input
    "complex_reasoning": "gpt-4o",              # $5/1M input
    "code_generation": "gpt-4.1",               # Best for agentic
}

# 2. Enable prompt caching (automatic for prompts >1024 tokens)
# Just structure your prompts with static content first

# 3. Batch non-urgent requests
from openai import OpenAI
client = OpenAI()

# Instead of individual calls:
# for item in items:
#     response = client.chat.completions.create(...)

# Use batch API:
batch = client.batches.create(
    input_file_id="file-abc123",
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

Immediate Latency Reduction

# 1. Stream responses
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

# 2. Run independent calls in parallel
import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def parallel_calls(prompts):
    tasks = [
        async_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": p}]
        )
        for p in prompts
    ]
    return await asyncio.gather(*tasks)

See Also

• Batch Processing
• Prompt Caching
• Fine-tuning Techniques