Graders Reference
Complete reference for all grader types and configurations.
Grader Types Overview
┌─────────────────────────────────────────────────────────────────────┐
│ GRADER TYPES │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ Type │ Purpose │ Output │
│ ──────────────────┼───────────────────────┼────────────────────── │
│ python │ Custom logic │ Score 0-1 │
│ text_similarity │ Semantic similarity │ Score 0-1 │
│ score_model │ LLM-as-judge │ Score 0-1 + reasoning │
│ string_check │ Pattern matching │ Pass/fail │
│ │
└─────────────────────────────────────────────────────────────────────┘Chemical Name Grader
Type: python
Purpose: Check if all chemical names from source appear in summary.
python
chemical_name_grader = {
"name": "chemical_name_grader",
"type": "python",
"source": '''
import re
def grade(item, sample):
"""Check chemical name preservation."""
section = item.get("section", "")
summary = sample.get("output_text", "")
# Extract chemical names (simplified pattern)
pattern = r"\\[\\d+-\\d+C\\][a-zA-Z]+"
section_chemicals = set(re.findall(pattern, section, re.IGNORECASE))
if not section_chemicals:
return 1.0 # No chemicals to check
summary_lower = summary.lower()
found = sum(1 for c in section_chemicals if c.lower() in summary_lower)
return found / len(section_chemicals)
''',
"pass_threshold": 0.9,
}Word Length Deviation Grader
Type: python
Purpose: Check if summary length is within acceptable range.
python
word_length_deviation_grader = {
"name": "word_length_deviation_grader",
"type": "python",
"source": '''
def grade(item, sample):
"""Check summary length deviation from target."""
summary = sample.get("output_text", "")
word_count = len(summary.split())
target = 80 # Target word count
max_deviation = 0.35 # 35% allowed deviation
if word_count == 0:
return 0.0
deviation = abs(word_count - target) / target
if deviation <= max_deviation:
# Linear score: 1.0 at target, decreasing to 0.65 at max deviation
return 1.0 - (deviation / max_deviation) * 0.35
else:
# Below threshold for excessive deviation
return max(0.0, 0.65 - (deviation - max_deviation))
''',
"pass_threshold": 0.65,
}Cosine Similarity Grader
Type: text_similarity
Purpose: Measure semantic similarity between source and summary.
python
cosine_similarity_grader = {
"name": "cosine_similarity",
"type": "text_similarity",
"input": "{ {item.section} }", # Remove spaces between braces
"reference": "{ {sample.output_text} }", # Remove spaces between braces
"pass_threshold": 0.5,
"evaluation_metric": "cosine_similarity",
}Note
Template variables use double curly braces with no spaces: item.section becomes wrapped in { { } } (no spaces).
LLM-as-Judge Grader
Type: score_model
Purpose: Use LLM to evaluate summary quality with reasoning.
python
llm_as_judge_grader = {
"name": "llm_as_judge",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": """You are an expert technical summarization evaluator.
Evaluate if the summary captures important technical facts from the source.
Scoring:
- 1.0: Flawless, comprehensive, all facts preserved
- 0.75-0.99: Excellent, minor omissions only
- 0.5-0.75: Good, some details missing
- 0.3-0.5: Significant issues
- 0.0-0.3: Poor, major omissions
IMPORTANT: Respond with ONLY a number between 0 and 1.""",
},
{
"role": "user",
"content": "Source Section:\n{ {item.section} }\n\nGenerated Summary:\n{ {sample.output_text} }",
},
],
"range": [0, 1],
"pass_threshold": 0.85,
}LLM-as-Judge with Reasoning
Type: score_model
Purpose: Get detailed feedback for optimization.
python
llm_as_judge_detailed = {
"name": "llm_as_judge_detailed",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": """You are an expert technical summarization evaluator.
Evaluate the summary and respond in JSON:
{
"score": <float 0-1>,
"strengths": ["list", "of", "strengths"],
"weaknesses": ["list", "of", "weaknesses"],
"missing_items": ["items", "not", "included"]
}""",
},
{
"role": "user",
"content": "Section:\n{ {item.section} }\n\nSummary:\n{ {sample.output_text} }",
},
],
"range": [0, 1],
"pass_threshold": 0.85,
}Domain-Specific Graders
Chemical Accuracy Judge
python
chemical_accuracy_judge = {
"name": "chemical_accuracy_judge",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": """You are a pharmaceutical chemistry expert.
Evaluate if chemical names, formulas, and CAS numbers are preserved exactly.
Score 1.0 if all chemicals match exactly, 0.0 if any are wrong or missing.
Respond with only a number.""",
},
{
"role": "user",
"content": "Section:\n{{item.section}}\nSummary:\n{{sample.output_text}}",
},
],
"range": [0, 1],
"pass_threshold": 0.9,
}Regulatory Compliance Judge
python
compliance_judge = {
"name": "compliance_judge",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": """You are an FDA regulatory compliance expert.
Evaluate if the summary maintains all compliance-relevant information.
Score based on preservation of regulatory references and requirements.
Respond with only a number between 0 and 1.""",
},
{
"role": "user",
"content": "Section:\n{{item.section}}\nSummary:\n{{sample.output_text}}",
},
],
"range": [0, 1],
"pass_threshold": 0.85,
}Grader Configuration Matrix
| Grader | Type | Pass Threshold | Model | Purpose |
|---|---|---|---|---|
chemical_name_grader | python | 0.9 | N/A | Chemical preservation |
word_length_deviation_grader | python | 0.65 | N/A | Length control |
cosine_similarity | text_similarity | 0.5 | N/A | Semantic similarity |
llm_as_judge | score_model | 0.85 | gpt-4.1 | Quality assessment |
Template Variables
| Variable | Description |
|---|---|
item.section | Source text being summarized |
item.summary | Expected/reference summary (if available) |
sample.output_text | Generated summary from agent |
Template Syntax
In grader configurations, wrap variable names with double curly braces: {% raw %}{% endraw %}
Complete Eval Configuration
python
testing_criteria = [
chemical_name_grader,
word_length_deviation_grader,
cosine_similarity_grader,
llm_as_judge_grader,
]
eval_config = client.evals.create(
name="summarization-eval-v1",
data_source_config={
"type": "custom",
"item_schema": {
"type": "object",
"properties": {
"section": {"type": "string"},
"summary": {"type": "string"},
},
"required": ["section", "summary"],
},
},
testing_criteria=testing_criteria,
)