SOP-008: GEPA Integration
Document Control
| Property | Value |
|---|---|
| SOP ID | 008 |
| Title | GEPA Integration |
| Version | 1.0 |
| Status | Active |
| Complexity | Advanced |
Purpose
Integrate Genetic-Pareto (GEPA) optimization for robust, reflective prompt evolution that outperforms static metaprompts.
Prerequisites
- Completed SOP-001 through SOP-007
- GEPA package installed (
pip install gepa litellm) - Train/validation dataset split
What is GEPA?
┌─────────────────────────────────────────────────────────────────────┐
│ GEPA OPTIMIZATION PROCESS │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ GEPA = Genetic-Pareto Prompt Evolution │
│ │
│ Key Features: │
│ ├─► Samples agent trajectories │
│ ├─► Reflects on them in natural language │
│ ├─► Proposes prompt revisions │
│ ├─► Evolves through iterative feedback loops │
│ └─► Validates on separate dataset │
│ │
│ Paper: "GEPA: Reflective Prompt Evolution Can Outperform │
│ Reinforcement Learning" (arXiv:2507.19457) │
│ │
│ Authors: Lakshya A Agrawal, Shangyin Tan, et al. │
│ │
└─────────────────────────────────────────────────────────────────────┘GEPA vs Other Methods
┌─────────────────────────────────────────────────────────────────────┐
│ OPTIMIZATION COMPARISON │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Method │ Pros │ Cons │
│ ────────────────────┼─────────────────────┼───────────────────────│
│ OpenAI Platform │ Fast, intuitive │ Manual, no automation │
│ Static Metaprompt │ Automated │ Limited exploration │
│ GEPA │ Robust, reflective │ Slower, more complex │
│ │
│ Recommendation: │
│ Start with Platform → Automate with Metaprompt → Optimize with GEPA│
│ │
└─────────────────────────────────────────────────────────────────────┘Step-by-Step Procedure
Step 1: Install GEPA
bash
pip install gepa litellmStep 2: Prepare Train/Validation Split
python
import pandas as pd
def read_csv_content(file_path: str) -> list[dict]:
"""Read CSV and return sections to summarize."""
df = pd.read_csv(file_path)
return [{'content': content} for content in df['content'].tolist()]
# Load and split dataset
trainset = read_csv_content("data/dataset.csv")
val_cut = max(1, int(0.1 * len(trainset))) # 10% for validation
valset = trainset[:val_cut] if len(trainset) > 1 else trainset
print(f"Training set: {len(trainset)} samples")
print(f"Validation set: {len(valset)} samples")Step 3: Create GEPA Adapter
The adapter bridges GEPA with your existing eval framework:
python
import gepa
from gepa import EvaluationBatch
class EvalsBackedSummarizationAdapter:
"""
Minimal adapter for GEPA that connects to OpenAI Evals.
Required methods:
- evaluate(): Run eval on minibatch, return EvaluationBatch
- get_components_to_update(): Return text fields to evolve
- make_reflective_dataset(): Package examples for reflection
"""
propose_new_texts = None # Use GEPA's default reflection flow
def __init__(
self,
client,
eval_id: str,
gen_model: str = "gpt-5",
user_prefix: str | None = None
):
self.client = client
self.eval_id = eval_id
self.gen_model = gen_model
self.user_prefix = user_prefix or "Summarize:\n\n"
def _summarize(self, system_prompt: str, section: str) -> str:
"""Generate summary using candidate prompt."""
resp = self.client.chat.completions.create(
model=self.gen_model,
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"{self.user_prefix}{section}"},
],
)
return resp.choices[0].message.content.strip()
def evaluate(
self,
inputs: list[dict],
candidate: dict,
capture_traces: bool = True
) -> EvaluationBatch:
"""Run evaluation on inputs with candidate prompt."""
system_prompt = candidate["system_prompt"]
scores: list[float] = []
outputs: list[str] = []
trajectories: list[dict] = []
for item in inputs:
section = item["content"]
# 1) Generate summary with candidate prompt
summary = self._summarize(system_prompt, section)
outputs.append(summary)
# 2) Grade using existing evals pipeline
run = run_eval(
eval_id=self.eval_id,
section=section,
summary=summary
)
out_items = poll_eval_run(
eval_id=self.eval_id,
run_id=run.id
)
grader_scores = parse_eval_run_output(out_items)
# 3) Calculate score and collect feedback
scalar = calculate_grader_score(grader_scores)
feedback = collect_grader_feedback(grader_scores) or \
"All graders passed; keep precision and coverage."
scores.append(float(scalar))
trajectories.append({
"inputs": {"section": section},
"generated_output": summary,
"metrics": {
"combined": float(scalar),
"by_grader": grader_scores,
},
"feedback": feedback,
})
return EvaluationBatch(
scores=scores,
outputs=outputs,
trajectories=trajectories
)
def get_components_to_update(self, candidate: dict) -> list[str]:
"""Return the text fields GEPA should evolve."""
return ["system_prompt"]
def make_reflective_dataset(
self,
candidate: dict,
eval_batch: EvaluationBatch,
components_to_update: list[str]
) -> dict:
"""Package examples for GEPA's reflection step."""
examples = []
for traj in (eval_batch.trajectories or []):
examples.append({
"Inputs": {"section": traj["inputs"]["section"]},
"Generated Outputs": traj["generated_output"],
"Feedback": traj["feedback"],
})
return {"system_prompt": examples}Step 4: Configure GEPA Optimization
python
from openai import OpenAI
client = OpenAI()
# Define seed candidate (starting prompt)
seed_candidate = {
"system_prompt": "You are a summarization assistant. Given a section of text, produce a summary."
}
# Create adapter instance
adapter = EvalsBackedSummarizationAdapter(
client=client,
eval_id=EVAL_ID, # From SOP-003
gen_model="gpt-5",
)Step 5: Run GEPA Optimization
python
# Run optimization
# Note: GEPA may take 10-15 minutes to complete
result = gepa.optimize(
seed_candidate=seed_candidate,
trainset=trainset,
valset=valset,
adapter=adapter,
reflection_lm="gpt-5",
max_metric_calls=10, # Increase for more thorough optimization
track_best_outputs=True,
display_progress_bar=True
)
# Extract best prompt
best_prompt = result.best_candidate["system_prompt"]
print("\n=== Best Evolved Prompt ===\n")
print(best_prompt)Step 6: Analyze Results
python
def analyze_gepa_results(result):
"""Analyze GEPA optimization results."""
print("=== GEPA Optimization Summary ===\n")
# Best candidate
print(f"Best Score: {result.best_score:.4f}")
print(f"Total Iterations: {result.total_iterations}")
# Evolution history
if hasattr(result, 'history'):
print("\nScore Progression:")
for i, score in enumerate(result.history):
print(f" Iteration {i}: {score:.4f}")
# Best prompt analysis
best_prompt = result.best_candidate["system_prompt"]
word_count = len(best_prompt.split())
print(f"\nBest Prompt Stats:")
print(f" Word count: {word_count}")
print(f" Character count: {len(best_prompt)}")
return result.best_candidate
analyze_gepa_results(result)Step 7: Deploy Optimized Prompt
python
# Update VersionedPrompt with GEPA result
summarization_prompt.update(
new_prompt=best_prompt,
model="gpt-5",
metadata={
"optimization_method": "GEPA",
"score": result.best_score,
"iterations": result.total_iterations,
}
)
# Create new agent with optimized prompt
optimized_agent = make_summarization_agent(summarization_prompt.current())
print(f"✅ Deployed optimized prompt v{summarization_prompt.current().version}")GEPA Configuration Options
| Parameter | Default | Description |
|---|---|---|
max_metric_calls | 10 | Maximum evaluation iterations |
reflection_lm | "gpt-5" | Model for prompt reflection |
track_best_outputs | True | Store best outputs |
display_progress_bar | True | Show progress |
Example GEPA Output Prompt
From the cookbook's healthcare use case:
You are a domain-aware summarization assistant for technical pharmaceutical texts.
Given a "section" of text, produce a concise, single-paragraph summary that
preserves key technical facts and exact nomenclature.
Length and format:
- Write 1-3 sentences totaling about 45-70 words (target ~60; never exceed 90).
- Use one paragraph; no bullets, headings, tables, or heavy formatting.
Exact names and notation:
- Include every chemical name that appears in the section at least once,
using the exact original spelling, capitalization, punctuation, isotopic labels,
brackets, hyphens, salts, buffer names, and parenthetical qualifiers.
- Examples: [1-13C]pyruvic acid, AH111501 sodium salt, TRIS/EDTA buffer solution
Content prioritization (if space is tight):
1) What the section is about (topic/purpose)
2) All named chemical entities and compositions
3) Critical process/handling facts
4) Container/packaging specifics
5) Microbiological/testing/regulatory details
6) Overages/single-dose formulas and key quantities
Self-check before finalizing:
- Does the paragraph contain every distinct chemical name exactly as written?
- Is the summary 45-70 words (≤90), in a single paragraph?
- Are critical process/regulatory details preserved?Verification Checklist
- [ ] GEPA and litellm installed
- [ ] Train/validation split created
- [ ] Adapter class implemented with all required methods
- [ ] Seed candidate defined
- [ ] Optimization run completed
- [ ] Results analyzed
- [ ] Best prompt deployed to VersionedPrompt
- [ ] New agent created with optimized prompt
Troubleshooting
Issue: GEPA optimization takes too long
Solution: Reduce max_metric_calls or use smaller dataset:
python
trainset = trainset[:20] # Use subset for testingIssue: Scores not improving
Solution: Check feedback quality in collect_grader_feedback(). More detailed feedback leads to better evolution.
Issue: Memory errors
Solution: Process in smaller batches or use gpt-5-mini for generation.