SOP-003: Eval Creation

Document Control

Property	Value
SOP ID	003
Title	Eval Creation
Version	1.0
Status	Active
Complexity	Medium

Purpose

Create evaluations with four complementary graders that balance deterministic checks with semantic judgment.

Prerequisites

• Completed SOP-001: Environment Setup
• Completed SOP-002: Dataset Preparation
• OpenAI API key with Evals access

The Four Graders

┌─────────────────────────────────────────────────────────────────────┐
│                        GRADER ARCHITECTURE                           │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    DETERMINISTIC GRADERS                      │   │
│  │  ┌────────────────────┐    ┌────────────────────┐            │   │
│  │  │ chemical_name      │    │ word_length        │            │   │
│  │  │ _grader            │    │ _deviation_grader  │            │   │
│  │  │                    │    │                    │            │   │
│  │  │ Type: python       │    │ Type: python       │            │   │
│  │  │ Threshold: 0.8     │    │ Threshold: 0.85    │            │   │
│  │  │ Purpose: Entity    │    │ Purpose: Length    │            │   │
│  │  │ preservation       │    │ discipline         │            │   │
│  │  └────────────────────┘    └────────────────────┘            │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                     SEMANTIC GRADERS                          │   │
│  │  ┌────────────────────┐    ┌────────────────────┐            │   │
│  │  │ cosine_similarity  │    │ llm_as_judge       │            │   │
│  │  │                    │    │                    │            │   │
│  │  │ Type: text_        │    │ Type: score_model  │            │   │
│  │  │ similarity         │    │ Model: gpt-4.1     │            │   │
│  │  │ Threshold: 0.85    │    │ Threshold: 0.85    │            │   │
│  │  │ Purpose: Semantic  │    │ Purpose: Holistic  │            │   │
│  │  │ anchoring          │    │ quality check      │            │   │
│  │  └────────────────────┘    └────────────────────┘            │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Grader Summary Table

Grader	Type	Pass Threshold	What It Checks	Why
chemical_name_grader	python	0.8	Exact chemical names in summary	Preserves critical domain entities
word_length_deviation_grader	python	0.85	Deviation from 100-word target	Keeps summaries concise and comparable
cosine_similarity	text_similarity	0.85	Cosine similarity to source	Anchors summary to source content
llm_as_judge	score_model	0.85	Rubric-driven quality score	Captures nuanced quality signals

Step-by-Step Procedure

Step 1: Import Dependencies

import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

Step 2: Define Data Source Configuration

data_source_config = {
    "type": "custom",
    "item_schema": {
        "type": "object",
        "properties": {
            "section": {"type": "string"},
            "summary": {"type": "string"}
        },
        "required": ["section", "summary"],
    },
    "include_sample_schema": False,
}

Step 3: Create Chemical Name Grader

This Python grader checks if chemical names from the source appear in the summary:

chemical_name_grader = {
    "type": "python",
    "name": "chemical_name_grader",
    "image_tag": "2025-05-08",
    "pass_threshold": 0.8,
    "source": r"""def grade(sample: dict, item: dict) -> float:
    section = item["section"]
    summary = item["summary"]

    CHEMICALS_MASTER = [
        "[1-13C]Pyruvic acid", "[1-13C]Pyruvate",
        "Sodium [1-13C]pyruvate", "AH111501 sodium salt",
        "TRIS", "NaOH", "Na2EDTA", "EDTA",
        "Hyperpolarized Pyruvate (13C) Injection",
        # Add your domain-specific terms here
    ]

    # Find chemicals present in the section
    present = [chem for chem in CHEMICALS_MASTER if chem in section]

    # If no chemicals present, consider it satisfied
    if not present:
        return 1.0

    # Count how many appear in the summary
    correct = sum(1 for chem in present if chem in summary)

    return correct / len(present)
""",
}

Step 4: Create Word Length Grader

This Python grader penalizes summaries that deviate from the target length:

word_length_grader = {
    "type": "python",
    "name": "word_length_deviation_grader",
    "image_tag": "2025-05-08",
    "pass_threshold": 0.85,
    "source": r"""
def grade(sample: dict, item: dict) -> float:
    summary = item["summary"]
    word_count = len(summary.split())

    expected_summary_length = 100
    tolerance = 0.2  # 20% band around target

    # Calculate relative deviation
    deviation = abs(word_count - expected_summary_length) / expected_summary_length

    # If within tolerance band, full score
    if deviation <= tolerance:
        return 1.0

    # Outside band: score decays linearly, capped at 0
    score = 1.0 - (deviation - tolerance)
    return max(0.0, score)
""",
}

Step 5: Create Cosine Similarity Grader

Built-in grader for semantic similarity:

cosine_similarity_grader = {
    "name": "cosine_similarity",
    "type": "text_similarity",
    "input": "{ { item.summary } }",     # Remove spaces in actual code
    "reference": "{ { item.section } }",  # Remove spaces in actual code
    "evaluation_metric": "cosine",
    "pass_threshold": 0.85,
}

Step 6: Create LLM-as-Judge Grader

Uses GPT-4.1 to evaluate summary quality:

llm_as_judge_grader = {
    "name": "llm_as_judge",
    "type": "score_model",
    "model": "gpt-4.1",
    "input": [
        {
            "role": "system",
            "content": (
                "You are an expert technical summarization evaluator. "
                "Evaluate whether the summary captures and preserves the important "
                "technical facts and specific details from the section.\n\n"
                "Scoring Guidelines:\n"
                "- Return a numerical score between 0 and 1.\n"
                "- 1.0: Flawless - comprehensive, faithful, accurate\n"
                "- 0.75-0.99: Excellent - all main facts, minor omissions\n"
                "- 0.5-0.75: Good - most info retained, some details missing\n"
                "- 0.3-0.5: Significant info missing or inaccuracies\n"
                "- 0.0-0.3: Major omissions or failures\n\n"
                "Respond only with a single number between 0 and 1."
            ),
        },
        {
            "role": "user",
            "content": (
                "Section:\n{ {item.section} }\n"
                "Summary:\n{ {sample.output_text} }"
            ),
        },
    ],
    "range": [0, 1],
    "pass_threshold": 0.85,
}

Step 7: Create the Eval

testing_criteria = [
    chemical_name_grader,
    word_length_grader,
    cosine_similarity_grader,
    llm_as_judge_grader,
]

eval = client.evals.create(
    name="self_evolving_eval",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

print(f"Created Eval: {eval.id}")

Step 8: Store Eval ID

Save the eval ID for use in the evaluation loop:

# Save to config
EVAL_ID = eval.id

# Or save to file
with open('eval_id.txt', 'w') as f:
    f.write(eval.id)

Complete Code

import os
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Data source configuration
data_source_config = {
    "type": "custom",
    "item_schema": {
        "type": "object",
        "properties": {
            "section": {"type": "string"},
            "summary": {"type": "string"}
        },
        "required": ["section", "summary"],
    },
    "include_sample_schema": False,
}

# All four graders
testing_criteria = [
    {
        "type": "python",
        "name": "chemical_name_grader",
        "image_tag": "2025-05-08",
        "pass_threshold": 0.8,
        "source": r"""def grade(sample: dict, item: dict) -> float:
    section = item["section"]
    summary = item["summary"]
    CHEMICALS = ["[1-13C]Pyruvic acid", "[1-13C]pyruvate", "TRIS", "NaOH"]
    present = [c for c in CHEMICALS if c in section]
    if not present: return 1.0
    correct = sum(1 for c in present if c in summary)
    return correct / len(present)
""",
    },
    {
        "type": "python",
        "name": "word_length_deviation_grader",
        "image_tag": "2025-05-08",
        "pass_threshold": 0.85,
        "source": r"""def grade(sample: dict, item: dict) -> float:
    summary = item["summary"]
    word_count = len(summary.split())
    expected = 100
    tolerance = 0.2
    deviation = abs(word_count - expected) / expected
    if deviation <= tolerance: return 1.0
    return max(0.0, 1.0 - (deviation - tolerance))
""",
    },
    {
        "name": "cosine_similarity",
        "type": "text_similarity",
        "input": "{ { item.summary } }",
        "reference": "{ { item.section } }",
        "evaluation_metric": "cosine",
        "pass_threshold": 0.85,
    },
    {
        "name": "llm_as_judge",
        "type": "score_model",
        "model": "gpt-4.1",
        "input": [
            {"role": "system", "content": "Score summary quality 0-1. Respond with only a number."},
            {"role": "user", "content": "Section:\n{ {item.section} }\nSummary:\n{ {sample.output_text} }"}
        ],
        "range": [0, 1],
        "pass_threshold": 0.85,
    },
]

# Create eval
eval = client.evals.create(
    name="self_evolving_eval",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

print(f"✅ Created Eval: {eval.id}")

Verification Checklist

• [ ] All four graders defined with correct types
• [ ] Pass thresholds set appropriately (0.8-0.85)
• [ ] Data source schema matches your dataset
• [ ] Eval created successfully with valid ID
• [ ] Eval ID stored for later use

Troubleshooting

Issue: Python grader syntax error

Solution: Ensure raw string (r""") is used and no special characters are escaped incorrectly.

Issue: LLM-as-judge returns text instead of number

Solution: Add explicit instruction: "Respond with only a single number."

Issue: Cosine similarity always returns 0

Solution: Check that input and reference templates match your item schema field names.

See Also

• SOP-004: Baseline Agent Setup
• Quick Reference: Grader Types
• Workflow: Self-Evolving Loop