Prompt Caching

Reduce latency by up to 80% and costs by 50% for cached prompt prefixes.

Overview

Prompt Caching automatically caches the longest prefix of your prompt that is reused across requests, dramatically improving performance for repetitive prompt patterns.

┌─────────────────────────────────────────────────────────────────────┐
│                    PROMPT CACHING MECHANISM                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  First Request (Cache Miss):                                        │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │ System Prompt (5000 tokens) │ User Message (100 tokens)     │    │
│  │      PROCESSED              │      PROCESSED                │    │
│  └─────────────────────────────────────────────────────────────┘    │
│  Time: 2000ms | Cost: Full price                                    │
│                                                                      │
│  Second Request (Cache Hit):                                        │
│  ┌─────────────────────────────────────────────────────────────┐    │
│  │ System Prompt (5000 tokens) │ User Message (100 tokens)     │    │
│  │      CACHED (50% off)       │      PROCESSED                │    │
│  └─────────────────────────────────────────────────────────────┘    │
│  Time: 400ms | Cost: Cached tokens at 50% discount                  │
│                                                                      │
│  RESULT: 80% faster, 50% cheaper on cached portion                  │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Requirements

Requirement	Details
Minimum prefix	1,024 tokens
Cache duration	5-10 minutes of inactivity
Supported models	GPT-4o, GPT-4o-mini, o1-preview, o1-mini
Automatic	No code changes required

How It Works

Caching happens automatically when:

1. Your prompt prefix is >= 1,024 tokens
2. You make multiple requests with the same prefix
3. Requests happen within the cache TTL (5-10 minutes)

from openai import OpenAI

client = OpenAI()

# Large static system prompt (automatically cached after first call)
SYSTEM_PROMPT = """You are an expert legal document analyzer.

[5000+ tokens of legal context, templates, and instructions...]
"""

def analyze_document(document: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},  # Cached
            {"role": "user", "content": f"Analyze: {document}"}  # Not cached
        ]
    )

    # Check cache usage in response
    usage = response.usage
    print(f"Cached tokens: {usage.prompt_tokens_details.cached_tokens}")
    print(f"Total prompt tokens: {usage.prompt_tokens}")

    return response.choices[0].message.content

# First call: Cache miss (full latency)
result1 = analyze_document("Contract A...")

# Second call: Cache hit (80% faster)
result2 = analyze_document("Contract B...")

Optimizing for Cache Hits

Structure Prompts for Caching

# BAD: Variable content at the beginning
messages = [
    {"role": "system", "content": f"Today is {date}. You are a helpful assistant..."},
    {"role": "user", "content": question}
]

# GOOD: Static content first, variable content last
messages = [
    {"role": "system", "content": "You are a helpful assistant with expertise in..."},
    {"role": "system", "content": f"Current date context: {date}"},  # Small variable part
    {"role": "user", "content": question}
]

Tools and Functions

Tool definitions are part of the cached prefix:

# Tools are included in cache calculation
tools = [
    {"type": "function", "function": {...}},  # Part of cached prefix
    {"type": "function", "function": {...}},
    # ... many tools
]

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": LARGE_SYSTEM_PROMPT},  # Cached
    ],
    tools=tools  # Also cached if >= 1024 tokens total
)

Images and Multi-Modal

Images are cached at fixed sizes:

# Image tokens contribute to cacheable prefix
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": static_reference_image}},  # Cached
                {"type": "text", "text": "Compare this to the reference above:"},
                {"type": "image_url", "image_url": {"url": new_image}}  # Not cached
            ]
        }
    ]
)

Monitoring Cache Performance

def call_with_cache_stats(messages, model="gpt-4o"):
    """Make API call and report cache statistics."""
    response = client.chat.completions.create(
        model=model,
        messages=messages
    )

    usage = response.usage
    cached = usage.prompt_tokens_details.cached_tokens
    total_prompt = usage.prompt_tokens

    cache_rate = (cached / total_prompt * 100) if total_prompt > 0 else 0

    print(f"Total prompt tokens: {total_prompt}")
    print(f"Cached tokens: {cached} ({cache_rate:.1f}%)")
    print(f"Non-cached tokens: {total_prompt - cached}")

    # Estimate savings
    if cached > 0:
        saved_cost = cached * 0.5  # 50% discount on cached tokens
        saved_time = cached * 0.8  # ~80% time savings estimate
        print(f"Estimated cost savings: {saved_cost / total_prompt * 100:.1f}%")

    return response

# Usage
for i, doc in enumerate(documents):
    print(f"\n--- Document {i+1} ---")
    response = call_with_cache_stats([
        {"role": "system", "content": LARGE_PROMPT},
        {"role": "user", "content": f"Analyze: {doc}"}
    ])

Cache-Optimized Patterns

RAG with Caching

# Static retrieval context + dynamic query
STATIC_CONTEXT = """
You are a knowledge assistant with access to the following reference documents:

[Include large static knowledge base here - will be cached]

Document 1: ...
Document 2: ...
... (1000s of tokens)
"""

def query_with_context(query: str, dynamic_context: str = ""):
    messages = [
        {"role": "system", "content": STATIC_CONTEXT},  # Cached
    ]

    if dynamic_context:
        messages.append({
            "role": "system",
            "content": f"Additional context for this query:\n{dynamic_context}"
        })

    messages.append({"role": "user", "content": query})

    return client.chat.completions.create(
        model="gpt-4o",
        messages=messages
    )

Conversation with Cached System Prompt

class CacheOptimizedChat:
    def __init__(self, system_prompt: str):
        self.system_prompt = system_prompt
        self.history = []

    def chat(self, user_message: str) -> str:
        # Build messages with system prompt first (cached)
        messages = [{"role": "system", "content": self.system_prompt}]

        # Add conversation history
        messages.extend(self.history)

        # Add new user message
        messages.append({"role": "user", "content": user_message})

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages
        )

        assistant_message = response.choices[0].message.content

        # Update history
        self.history.append({"role": "user", "content": user_message})
        self.history.append({"role": "assistant", "content": assistant_message})

        return assistant_message

# System prompt cached across all turns
chat = CacheOptimizedChat(LARGE_SYSTEM_PROMPT)
chat.chat("Hello")  # Cache miss
chat.chat("Tell me more")  # Cache hit on system prompt

Batch Processing with Caching

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def process_batch_with_caching(items: list[str], batch_size: int = 10):
    """Process items in batches to maximize cache hits."""

    # First request primes the cache
    first_response = await async_client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": LARGE_PROMPT},
            {"role": "user", "content": f"Process: {items[0]}"}
        ]
    )

    results = [first_response.choices[0].message.content]

    # Subsequent requests hit the cache
    # Process in parallel batches
    for i in range(1, len(items), batch_size):
        batch = items[i:i+batch_size]

        tasks = [
            async_client.chat.completions.create(
                model="gpt-4o",
                messages=[
                    {"role": "system", "content": LARGE_PROMPT},
                    {"role": "user", "content": f"Process: {item}"}
                ]
            )
            for item in batch
        ]

        responses = await asyncio.gather(*tasks)
        results.extend([r.choices[0].message.content for r in responses])

    return results

Cost/Latency Analysis

Token Pricing with Caching

Component	Standard Price	Cached Price	Savings
Input tokens	$5.00/1M	$2.50/1M	50%
Output tokens	$15.00/1M	$15.00/1M	0%

Example Calculation

Scenario: 10,000 requests with 5,000 token system prompt

WITHOUT CACHING:
- Input: 10,000 × 5,000 = 50M tokens × $5/1M = $250
- Latency: ~2 seconds per request

WITH CACHING (after first request):
- Input: 9,999 × 5,000 = 49.995M cached tokens × $2.50/1M = $125
- First request: 5,000 tokens × $5/1M = $0.025
- Latency: ~0.4 seconds per request (after cache warm)

SAVINGS:
- Cost: $250 → $125.03 = ~50% reduction
- Latency: 2s → 0.4s = 80% reduction

Cache Invalidation

The cache is automatically invalidated when:

1. Prefix changes - Any modification to the cached portion
2. TTL expires - 5-10 minutes without requests
3. Model changes - Different model version

# These changes invalidate cache:

# Change 1: Modified system prompt
messages_v1 = [{"role": "system", "content": "Original prompt..."}]
messages_v2 = [{"role": "system", "content": "Modified prompt..."}]  # Cache miss

# Change 2: Added message in middle
messages_v1 = [
    {"role": "system", "content": PROMPT},
    {"role": "user", "content": "Question"}
]
messages_v2 = [
    {"role": "system", "content": PROMPT},
    {"role": "assistant", "content": "Previous answer"},  # Breaks cache
    {"role": "user", "content": "Follow up"}
]

Best Practices

Do's

• Put static content at the beginning of messages
• Use consistent system prompts across requests
• Process batches quickly to keep cache warm
• Monitor cached_tokens in responses

Don'ts

• Don't put timestamps or unique IDs at the start
• Don't modify the system prompt frequently
• Don't let cache go cold between batches
• Don't assume cache hits - verify with metrics

Troubleshooting

Cache Not Working?

def diagnose_cache(response):
    """Diagnose cache performance issues."""
    usage = response.usage

    cached = getattr(usage.prompt_tokens_details, 'cached_tokens', 0)
    total = usage.prompt_tokens

    if cached == 0:
        print("❌ No cache hit")
        if total < 1024:
            print(f"   Reason: Prompt too short ({total} < 1024 tokens)")
        else:
            print("   Reason: First request or prefix changed")
    else:
        cache_rate = cached / total * 100
        print(f"✅ Cache hit: {cache_rate:.1f}% ({cached}/{total} tokens)")

See Also

• Batch Processing
• GPT-4.1 Prompting Guide
• Quick Reference: Configuration