Self-Evolving Agents

Appearance

Troubleshooting Guide ​

Common issues and their solutions.

Quick Diagnosis ​

┌─────────────────────────────────────────────────────────────────────┐
│                    TROUBLESHOOTING FLOWCHART                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Problem?                                                           │
│  │                                                                  │
│  ├─► API Error ────────────► Check API key, rate limits            │
│  │                                                                  │
│  ├─► Eval Never Completes ─► Increase max_polls, check data        │
│  │                                                                  │
│  ├─► Score Always 0 ───────► Check grader config, thresholds       │
│  │                                                                  │
│  ├─► Score Not Improving ──► Check feedback quality, increase      │
│  │                           max_metric_calls                       │
│  │                                                                  │
│  ├─► Import Error ─────────► Verify package versions               │
│  │                                                                  │
│  └─► Agent Not Responding ─► Check model availability              │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

API Issues ​

Issue: Authentication Error ​

Symptom:

openai.AuthenticationError: Incorrect API key provided

Solution:

python
# Check API key is set
import os
print(os.getenv("OPENAI_API_KEY"))  # Should show your key

# Or set explicitly
from openai import OpenAI
client = OpenAI(api_key="sk-...")

Issue: Rate Limit Exceeded ​

Symptom:

openai.RateLimitError: Rate limit reached

Solution:

python
import time
from openai import OpenAI

client = OpenAI()

def call_with_retry(func, max_retries=3):
    for attempt in range(max_retries):
        try:
            return func()
        except Exception as e:
            if "rate_limit" in str(e).lower():
                wait_time = 2 ** attempt  # Exponential backoff
                time.sleep(wait_time)
            else:
                raise
    raise Exception("Max retries exceeded")

Issue: Model Not Found ​

Symptom:

openai.NotFoundError: The model 'gpt-5' does not exist

Solution:

python
# List available models
models = client.models.list()
for model in models.data:
    print(model.id)

# Use correct model ID
agent = Agent(model="gpt-4.1")  # Use available model

Evaluation Issues ​

Issue: Eval Run Never Completes ​

Symptom: poll_eval_run() times out

Solution:

python
# Increase polling parameters
def poll_eval_run(eval_id, run_id, max_polls=20):  # Increase from 10
    for attempt in range(1, max_polls + 1):
        run = client.evals.runs.retrieve(eval_id=eval_id, run_id=run_id)
        print(f"Attempt {attempt}: status={run.status}")  # Debug output

        if run.status == "completed":
            break
        if run.status == "failed":
            print(f"Run failed: {getattr(run, 'error', 'Unknown error')}")
            break

        time.sleep(10)  # Increase wait time

    return client.evals.runs.output_items.list(eval_id=eval_id, run_id=run_id)

Issue: Empty Results List ​

Symptom: parse_eval_run_output() returns empty list

Solution:

python
# Verify data source format
data_source = {
    "type": "jsonl",
    "source": {
        "type": "file_content",
        "content": [
            {
                "item": {  # Must have "item" wrapper
                    "section": section,
                    "summary": summary,
                }
            }
        ],
    },
}

# Check eval schema matches
print(f"Eval ID: {EVAL_ID}")
# Verify schema has "section" and "summary" fields

Issue: Score Always 0 ​

Symptom: All graders return 0.0

Solution:

python
# 1. Check grader configuration
print(json.dumps(testing_criteria, indent=2))

# 2. Verify template variables
# Ensure { {item.section} } and { {sample.output_text} } match your schema

# 3. Test individual grader
def test_grader(section, summary):
    run = run_eval(EVAL_ID, section, summary)
    output = poll_eval_run(EVAL_ID, run.id)

    for item in output.data:
        for result in item.results:
            print(f"Grader: {result.name}")
            print(f"Score: {result.score}")
            print(f"Passed: {result.passed}")
            print(f"Sample: {result.sample}")
            print("---")

Issue: LLM Judge Returns Text Instead of Number ​

Symptom: Score parsing fails

Solution:

python
# Add explicit instruction to system prompt
system_prompt = """You are an expert evaluator.

Evaluate the summary quality.

IMPORTANT: Respond with ONLY a single number between 0 and 1.
Do not include any other text, explanation, or formatting.

Example responses:
0.85
0.92
0.67
"""

Optimization Issues ​

Issue: Score Not Improving ​

Symptom: Optimization iterations show no improvement

Solutions:

  1. Improve feedback quality:
python
def collect_grader_feedback(grader_scores):
    """Provide detailed, actionable feedback."""
    feedback = []

    for entry in grader_scores:
        if not entry.get("passed"):
            grader = entry["grader_name"]

            # Be specific about what's wrong
            if "chemical_name" in grader:
                feedback.append(
                    "CRITICAL: Chemical names are missing. "
                    "Include ALL chemical names exactly as written in source, "
                    "including isotopic labels, salts, and modifiers."
                )
            elif "word_length" in grader:
                feedback.append(
                    f"Summary length incorrect. Target 80 words (±35%). "
                    f"Current score: {entry['score']:.2f}"
                )

    return " ".join(feedback)
  1. Increase iterations:
python
result = gepa.optimize(
    max_metric_calls=20,  # Increase from default 10
    ...
)
  1. Use better reflection model:
python
result = gepa.optimize(
    reflection_lm="gpt-5",  # Use most capable model
    ...
)

Issue: GEPA Optimization Takes Too Long ​

Symptom: Optimization runs for hours

Solutions:

python
# 1. Reduce dataset size
trainset = trainset[:20]  # Use subset

# 2. Reduce max iterations
result = gepa.optimize(
    max_metric_calls=5,  # Fewer iterations
    ...
)

# 3. Use faster model for generation
adapter = EvalsBackedSummarizationAdapter(
    gen_model="gpt-4.1",  # Faster than gpt-5
    ...
)

Issue: Memory Error During GEPA ​

Symptom: Process killed or memory error

Solutions:

python
# 1. Process in smaller batches
batch_size = 10
for i in range(0, len(trainset), batch_size):
    batch = trainset[i:i+batch_size]
    result = gepa.optimize(trainset=batch, ...)

# 2. Clear cache periodically
eval_cache.clear()

# 3. Use lighter model
adapter = EvalsBackedSummarizationAdapter(
    gen_model="gpt-5-mini",
    ...
)

Package Issues ​

Issue: Import Error ​

Symptom:

ModuleNotFoundError: No module named 'agents'

Solution:

bash
# Install required packages
pip install openai-agents
pip install openai>=1.0.0
pip install pydantic>=2.0

# Verify installation
pip list | grep -E "openai|pydantic|agents"

Issue: Pydantic Validation Error ​

Symptom:

pydantic.error_wrappers.ValidationError

Solution:

bash
# Ensure Pydantic v2
pip install pydantic>=2.0

# Check version
python -c "import pydantic; print(pydantic.__version__)"

Agent Issues ​

Issue: Agent Not Responding ​

Symptom: Runner.run() hangs or returns empty

Solution:

python
# 1. Test API directly
response = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

# 2. Check agent configuration
print(f"Agent name: {agent.name}")
print(f"Agent model: {agent.model}")
print(f"Instructions length: {len(agent.instructions)}")

# 3. Try with timeout
import asyncio

async def run_with_timeout(agent, input_text, timeout=30):
    try:
        result = await asyncio.wait_for(
            Runner.run(agent, input_text),
            timeout=timeout
        )
        return result
    except asyncio.TimeoutError:
        print("Agent timed out")
        return None

Monitoring Issues ​

Issue: Alerts Not Firing ​

Symptom: No alerts despite poor scores

Solution:

python
# 1. Check threshold configuration
print(f"Score healthy: {config.score_healthy}")
print(f"Score warning: {config.score_warning}")
print(f"Current score: {metrics.average_score}")

# 2. Verify alert handlers
for handler in handlers:
    print(f"Handler: {type(handler).__name__}")

# 3. Test alert manually
await handler.handle(AlertLevel.ALERT, {"test": True})

Getting Help ​

If you can't resolve an issue:

  1. Check API status: status.openai.com
  2. Review documentation: OpenAI API docs
  3. Search issues: GitHub Issues
  4. Debug with verbose logging:
python
import logging
logging.basicConfig(level=logging.DEBUG)

Based on OpenAI Cookbook - Bain & OpenAI Collaboration

Documentation generated with Claude Code