SOP-004: Baseline Agent Setup

Document Control

Property	Value
SOP ID	004
Title	Baseline Agent Setup
Version	1.0
Status	Active
Complexity	Medium

Purpose

Configure the baseline summarization agent and metaprompt optimization agent with version tracking capabilities.

Prerequisites

• Completed SOP-001: Environment Setup
• Completed SOP-003: Eval Creation
• Understanding of Pydantic models

Agent Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                      AGENT ARCHITECTURE                              │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                   SUMMARIZATION AGENT                         │   │
│  │  ┌────────────────────────────────────────────────────────┐  │   │
│  │  │              VersionedPrompt Manager                    │  │   │
│  │  │  ┌──────────────────────────────────────────────────┐  │  │   │
│  │  │  │         PromptVersionEntry                        │  │  │   │
│  │  │  │  • version: int                                   │  │  │   │
│  │  │  │  • model: str (gpt-5)                            │  │  │   │
│  │  │  │  • prompt: str                                    │  │  │   │
│  │  │  │  • timestamp: datetime                            │  │  │   │
│  │  │  │  • eval_id, run_id: Optional[str]                │  │  │   │
│  │  │  │  • metadata: Optional[dict]                       │  │  │   │
│  │  │  └──────────────────────────────────────────────────┘  │  │   │
│  │  └────────────────────────────────────────────────────────┘  │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                   METAPROMPT AGENT                            │   │
│  │  • Name: MetapromptAgent                                      │   │
│  │  • Instructions: "You are a prompt optimizer."                │   │
│  │  • Template: METAPROMPT_TEMPLATE                              │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Step-by-Step Procedure

Step 1: Create PromptVersionEntry Model

from datetime import datetime
from typing import Any, Optional
from pydantic import BaseModel, Field, ConfigDict, field_validator

class PromptVersionEntry(BaseModel):
    """Data model for a prompt and associated data for observability."""

    version: int = Field(
        ...,
        ge=0,
        description="Version number of the prompt (increments)"
    )
    model: str = Field(
        "gpt-5",
        min_length=1,
        description="The model version to use for this prompt"
    )
    prompt: str = Field(
        ...,
        min_length=1,
        description="The prompt text for this version"
    )
    timestamp: datetime = Field(
        default_factory=datetime.utcnow,
        description="UTC timestamp when this version was created"
    )
    eval_id: Optional[str] = Field(
        None,
        description="ID of the evaluation associated with this prompt"
    )
    run_id: Optional[str] = Field(
        None,
        description="ID of the run associated with this prompt"
    )
    metadata: Optional[dict[str, Any]] = Field(
        None,
        description="Free-form metadata dict"
    )

    model_config = ConfigDict(
        str_strip_whitespace=True,
        validate_assignment=True,
        extra="forbid"
    )

    @field_validator("prompt")
    @classmethod
    def prompt_not_blank(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("prompt must not be blank or only whitespace")
        return v

Step 2: Create VersionedPrompt Manager

class VersionedPrompt:
    """Manages a collection of prompt versions with controlled updates and rollbacks."""

    def __init__(
        self,
        initial_prompt: str,
        model: Optional[str] = "gpt-5",
        eval_id: Optional[str] = None,
        run_id: Optional[str] = None,
        metadata: Optional[dict[str, Any]] = None,
    ):
        if not initial_prompt or not initial_prompt.strip():
            raise ValueError("initial_prompt must be non-empty")

        self._versions: list[PromptVersionEntry] = []
        first_entry = PromptVersionEntry(
            version=0,
            prompt=initial_prompt,
            model=model,
            eval_id=eval_id,
            run_id=run_id,
            metadata=metadata,
        )
        self._versions.append(first_entry)

    def update(
        self,
        new_prompt: str,
        model: Optional[str] = "gpt-5",
        eval_id: Optional[str] = None,
        run_id: Optional[str] = None,
        metadata: Optional[dict[str, Any]] = None,
    ) -> PromptVersionEntry:
        """Add a new version of the prompt."""
        if not new_prompt or not new_prompt.strip():
            raise ValueError("new_prompt must be non-empty")

        version = self.current().version + 1
        entry = PromptVersionEntry(
            version=version,
            prompt=new_prompt,
            model=model,
            eval_id=eval_id,
            run_id=run_id,
            metadata=metadata,
        )
        self._versions.append(entry)
        return entry

    def current(self) -> PromptVersionEntry:
        """Get the current (latest) prompt version."""
        return self._versions[-1]

    def revert_to_version(self, version: int) -> PromptVersionEntry:
        """Revert to a specific version, removing all later versions."""
        idx = None
        for i, entry in enumerate(self._versions):
            if entry.version == version:
                idx = i
                break

        if idx is None:
            raise ValueError(f"No version found with version={version}")

        self._versions = self._versions[: idx + 1]
        return self._versions[-1]

    def history(self) -> list[PromptVersionEntry]:
        """Get full version history."""
        return self._versions.copy()

Step 3: Define Metaprompt Template

METAPROMPT_TEMPLATE = """
# Context:
## Original prompt:
{original_prompt}

## Section:
{section}

## Summary:
{summary}

## Reason to improve the prompt:
{reasoning}

# Task:
Write a new summarization prompt that is significantly improved and more specific than the original.

The new prompt should instruct the model to produce concise yet comprehensive technical summaries that precisely preserve all explicit information from the source text. It should emphasize the inclusion of all named entities, quantities, compounds, and technical terminology without paraphrasing or omission.

The resulting prompt should read like a clear, directive system message for a technical summarization assistant—structured, unambiguous, and generalizable across scientific or regulatory document sections.
"""

Step 4: Create Agent Instances

from agents import Agent

# Create metaprompt agent (static, doesn't evolve)
metaprompt_agent = Agent(
    name="MetapromptAgent",
    instructions="You are a prompt optimizer."
)

# Create initial summarization prompt
summarization_prompt = VersionedPrompt(
    initial_prompt="""You are a summarization assistant.
Given a section of text, produce a summary."""
)

# Factory function to create summarization agent from prompt entry
def make_summarization_agent(prompt_entry: PromptVersionEntry) -> Agent:
    return Agent(
        name="SummarizationAgent",
        instructions=prompt_entry.prompt,
        model=prompt_entry.model,
    )

# Create initial summarization agent
summarization_agent = make_summarization_agent(summarization_prompt.current())

Step 5: Initialize Tracking Variables

from typing import Any

# Cache eval results to avoid redundant grader runs
eval_cache: dict[tuple[str, str], list[dict[str, Any]]] = {}

# Track the highest-scoring candidate
best_candidate: dict[str, Any] = {
    "score": float("-inf"),
    "prompt": summarization_prompt.current().prompt,
    "model": summarization_prompt.current().model,
    "summary": None,
    "metadata": None,
    "version": summarization_prompt.current().version,
    "passed_lenient": False,
    "total_score": float("-inf"),
}

# Aggregate per-version performance
aggregate_prompt_stats: dict[int, dict[str, Any]] = {}

Step 6: Verify Agent Setup

async def verify_agents():
    from agents import Runner

    # Test summarization agent
    test_section = "This is a test section with some content to summarize."
    result = await Runner.run(summarization_agent, test_section)
    print(f"✅ Summarization agent working")
    print(f"   Output: {result.final_output[:100]}...")

    # Test metaprompt agent
    test_input = METAPROMPT_TEMPLATE.format(
        original_prompt="Test prompt",
        section="Test section",
        summary="Test summary",
        reasoning="Test reasoning"
    )
    result = await Runner.run(metaprompt_agent, test_input)
    print(f"✅ Metaprompt agent working")
    print(f"   Output: {result.final_output[:100]}...")

# Run verification
import asyncio
asyncio.run(verify_agents())

Complete Code

from datetime import datetime
from typing import Any, Optional
from pydantic import BaseModel, Field, ConfigDict, field_validator
from agents import Agent

# PromptVersionEntry model
class PromptVersionEntry(BaseModel):
    version: int = Field(..., ge=0)
    model: str = Field("gpt-5", min_length=1)
    prompt: str = Field(..., min_length=1)
    timestamp: datetime = Field(default_factory=datetime.utcnow)
    eval_id: Optional[str] = None
    run_id: Optional[str] = None
    metadata: Optional[dict[str, Any]] = None

    model_config = ConfigDict(str_strip_whitespace=True, validate_assignment=True, extra="forbid")

    @field_validator("prompt")
    @classmethod
    def prompt_not_blank(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("prompt must not be blank")
        return v

# VersionedPrompt manager
class VersionedPrompt:
    def __init__(self, initial_prompt: str, model: str = "gpt-5", **kwargs):
        self._versions = [PromptVersionEntry(version=0, prompt=initial_prompt, model=model, **kwargs)]

    def update(self, new_prompt: str, model: str = "gpt-5", **kwargs) -> PromptVersionEntry:
        entry = PromptVersionEntry(version=self.current().version + 1, prompt=new_prompt, model=model, **kwargs)
        self._versions.append(entry)
        return entry

    def current(self) -> PromptVersionEntry:
        return self._versions[-1]

    def revert_to_version(self, version: int) -> PromptVersionEntry:
        idx = next((i for i, e in enumerate(self._versions) if e.version == version), None)
        if idx is None:
            raise ValueError(f"Version {version} not found")
        self._versions = self._versions[:idx + 1]
        return self._versions[-1]

# Metaprompt template
METAPROMPT_TEMPLATE = """
# Context:
## Original prompt:
{original_prompt}

## Section:
{section}

## Summary:
{summary}

## Reason to improve the prompt:
{reasoning}

# Task:
Write an improved summarization prompt that preserves all technical details, named entities, and domain terminology.
"""

# Agent instances
metaprompt_agent = Agent(name="MetapromptAgent", instructions="You are a prompt optimizer.")
summarization_prompt = VersionedPrompt(initial_prompt="You are a summarization assistant. Given a section of text, produce a summary.")

def make_summarization_agent(prompt_entry: PromptVersionEntry) -> Agent:
    return Agent(name="SummarizationAgent", instructions=prompt_entry.prompt, model=prompt_entry.model)

summarization_agent = make_summarization_agent(summarization_prompt.current())

# Tracking variables
eval_cache: dict[tuple[str, str], list[dict[str, Any]]] = {}
best_candidate: dict[str, Any] = {"score": float("-inf"), "prompt": summarization_prompt.current().prompt}
aggregate_prompt_stats: dict[int, dict[str, Any]] = {}

print(f"✅ Agents initialized")
print(f"   Initial prompt version: v{summarization_prompt.current().version}")
print(f"   Model: {summarization_prompt.current().model}")

Verification Checklist

• [ ] PromptVersionEntry model created with validation
• [ ] VersionedPrompt manager with update/revert methods
• [ ] METAPROMPT_TEMPLATE defined
• [ ] Metaprompt agent created
• [ ] Summarization agent created with factory function
• [ ] Tracking variables initialized (eval_cache, best_candidate, aggregate_prompt_stats)
• [ ] Agents verified working with test inputs

Troubleshooting

Issue: Pydantic validation error

Solution: Ensure Pydantic v2 is installed:

pip install pydantic>=2.0

Issue: Agent not responding

Solution: Check API key and model availability:

client.models.retrieve("gpt-5")

See Also

• SOP-005: Run Evaluation Loop
• Workflow: Self-Evolving Loop
• Quick Reference: Configuration