Quick Start

Get up and running with self-evolving agents in under 10 minutes.

Prerequisites

• Python 3.10+
• OpenAI API key
• Basic understanding of LLMs and prompt engineering

Installation

pip install openai openai-agents pydantic pandas python-dotenv

For GEPA optimization (optional):

pip install gepa litellm

Environment Setup

Create a .env file:

OPENAI_API_KEY=sk-...

Minimal Example

import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# 1. Create a simple eval
eval = client.evals.create(
    name="my_first_eval",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {
                "input": {"type": "string"},
                "output": {"type": "string"}
            },
            "required": ["input", "output"]
        }
    },
    testing_criteria=[{
        "name": "quality_check",
        "type": "score_model",
        "model": "gpt-4.1",
        "input": [
            {"role": "system", "content": "Score output quality 0-1."},
            {"role": "user", "content": "Input: { {item.input} }\nOutput: { {sample.output_text} }"}
        ],
        "range": [0, 1],
        "pass_threshold": 0.8
    }]
)

print(f"Created eval: {eval.id}")

Core Concepts

┌─────────────────────────────────────────────────────────────┐
│                  SELF-EVOLVING LOOP                         │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  1. BASELINE AGENT                                          │
│     └─► Initial prompt + model configuration                │
│                                                             │
│  2. FEEDBACK LAYER                                          │
│     ├─► Human feedback (production)                         │
│     └─► LLM-as-judge (development)                          │
│                                                             │
│  3. EVALS & SCORING                                         │
│     ├─► 4 graders (chemical, length, cosine, LLM)          │
│     └─► Aggregated score vs threshold                       │
│                                                             │
│  4. PROMPT OPTIMIZATION                                     │
│     ├─► Manual (OpenAI Platform)                            │
│     ├─► Static metaprompt                                   │
│     └─► GEPA (genetic-pareto)                               │
│                                                             │
│  5. UPDATED AGENT                                           │
│     └─► Replaces baseline, becomes new starting point       │
│                                                             │
└─────────────────────────────────────────────────────────────┘

Next Steps

Step	Description	Link
1	Set up full environment	Environment Setup
2	Prepare your dataset	Dataset Preparation
3	Create evaluation graders	Eval Creation
4	Build your baseline agent	Baseline Agent Setup
5	Run the self-evolving loop	Self-Evolving Loop

Key Thresholds

Parameter	Default	Description
LENIENT_PASS_RATIO	0.75	75% of graders must pass
LENIENT_AVERAGE_THRESHOLD	0.85	85% average score required
MAX_OPTIMIZATION_RETRIES	3	Attempts before moving on
pass_threshold (per grader)	0.8-0.85	Individual grader threshold

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    SYSTEM ARCHITECTURE                       │
├─────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌────────────────┐    ┌────────────────┐                   │
│  │ Summarization  │    │  Metaprompt    │                   │
│  │     Agent      │    │     Agent      │                   │
│  └───────┬────────┘    └───────┬────────┘                   │
│          │                     │                             │
│          ▼                     ▼                             │
│  ┌────────────────────────────────────────┐                 │
│  │         Versioned Prompt Manager       │                 │
│  │  (PromptVersionEntry, VersionedPrompt) │                 │
│  └───────────────────┬────────────────────┘                 │
│                      │                                       │
│                      ▼                                       │
│  ┌────────────────────────────────────────┐                 │
│  │            OpenAI Evals API            │                 │
│  │  ┌──────────┐ ┌──────────┐ ┌────────┐  │                 │
│  │  │ Chemical │ │  Length  │ │ Cosine │  │                 │
│  │  │ Grader   │ │  Grader  │ │ Grader │  │                 │
│  │  └──────────┘ └──────────┘ └────────┘  │                 │
│  │  ┌──────────────────────────────────┐  │                 │
│  │  │        LLM-as-Judge Grader       │  │                 │
│  │  └──────────────────────────────────┘  │                 │
│  └────────────────────────────────────────┘                 │
│                                                              │
└─────────────────────────────────────────────────────────────┘

TIP

Use the chat widget in the bottom-right corner to ask questions about any topic in this documentation!