SOP-007: LLM-as-Judge Configuration
Document Control
| Property | Value |
|---|---|
| SOP ID | 007 |
| Title | LLM-as-Judge Configuration |
| Version | 1.0 |
| Status | Active |
| Complexity | Medium |
Purpose
Set up automated LLM-based evaluation for fast feedback loops without requiring human SME time.
Prerequisites
- Completed SOP-003: Eval Creation
- Understanding of prompt engineering
- OpenAI API access
When to Use LLM-as-Judge
┌─────────────────────────────────────────────────────────────────────┐
│ LLM-AS-JUDGE USE CASES │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ✅ IDEAL FOR: │
│ ├─► Development phase (fast iteration) │
│ ├─► Monitoring model drift in production │
│ ├─► Validating changes between model versions │
│ ├─► Large-scale automated testing │
│ └─► Continuous integration pipelines │
│ │
│ ⚠️ COMBINE WITH HUMAN REVIEW FOR: │
│ ├─► Final production approval │
│ ├─► Edge case identification │
│ └─► Domain-specific compliance checks │
│ │
└─────────────────────────────────────────────────────────────────────┘LLM-as-Judge Architecture
┌─────────────────────────────────────────────────────────────────────┐
│ LLM-AS-JUDGE FLOW │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ INPUT │ │
│ │ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Source │ │ Generated │ │ │
│ │ │ Section │ │ Summary │ │ │
│ │ └──────┬──────┘ └──────┬──────┘ │ │
│ │ │ │ │ │
│ │ └───────────┬───────────┘ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────┐ │ │
│ │ │ LLM-as-Judge │ │ │
│ │ │ (gpt-4.1) │ │ │
│ │ │ │ │ │
│ │ │ ┌───────────────┐ │ │ │
│ │ │ │ System Prompt │ │ │ │
│ │ │ │ (Rubric) │ │ │ │
│ │ │ └───────────────┘ │ │ │
│ │ └──────────┬──────────┘ │ │
│ │ ▼ │ │
│ │ ┌─────────────────────┐ │ │
│ │ │ Score (0-1) │ │ │
│ │ │ + Reasoning │ │ │
│ │ └─────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘Step-by-Step Procedure
Step 1: Design the Scoring Rubric
Create a detailed rubric for the LLM judge:
python
RUBRIC = """
Scoring Guidelines:
- Return a numerical score between 0 and 1 (with up to two decimal places).
1.0 (Flawless):
- Comprehensive, highly faithful, technically accurate
- Virtually no important details missing
- No significant misstatements or distortions
0.75-0.99 (Excellent):
- All main facts represented
- Trivial omissions or minor rewording only
- No material impact on understanding
0.5-0.75 (Good but imperfect):
- Most technical information retained
- Some less critical details missing
- Overall fidelity preserved
0.3-0.5 (Significant issues):
- Information missing or inaccuracies present
- Reasonable portion of key facts retained
0.0-0.3 (Poor):
- Major omissions or misunderstandings
- Failure to capture important technical content
"""Step 2: Create Judge System Prompt
python
JUDGE_SYSTEM_PROMPT = """You are an expert technical summarization evaluator.
Your task is to evaluate whether the summary captures and preserves the important technical facts and specific details from the source section.
Evaluation criteria:
- Factual accuracy: Are all stated facts correct?
- Completeness: Are key technical details preserved?
- Terminology: Are domain-specific terms retained exactly?
- Conciseness: Is the summary appropriately condensed?
{rubric}
IMPORTANT: Respond with ONLY a single number between 0 and 1.
Do not include any explanation in your response.
""".format(rubric=RUBRIC)Step 3: Configure the Grader
python
llm_as_judge_grader = {
"name": "llm_as_judge",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": JUDGE_SYSTEM_PROMPT,
},
{
"role": "user",
"content": (
"Source Section:\n{ {item.section} }\n\n"
"Generated Summary:\n{ {sample.output_text} }"
),
},
],
"range": [0, 1],
"pass_threshold": 0.85,
}Step 4: Add Reasoning Extraction (Optional)
For debugging, configure the judge to provide reasoning:
python
llm_as_judge_with_reasoning = {
"name": "llm_as_judge_detailed",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": """You are an expert technical summarization evaluator.
Evaluate the summary against the source section.
Respond in this exact JSON format:
{
"score": <float between 0 and 1>,
"strengths": ["list", "of", "strengths"],
"weaknesses": ["list", "of", "weaknesses"],
"missing_items": ["list", "of", "missing", "items"]
}
""",
},
{
"role": "user",
"content": "Section:\n{ {item.section} }\n\nSummary:\n{ {sample.output_text} }",
},
],
"range": [0, 1],
"pass_threshold": 0.85,
}Step 5: Calibrate the Judge
Test the judge with known good and bad examples:
python
async def calibrate_judge():
"""Test judge with known examples to verify scoring."""
test_cases = [
{
"section": "The drug [1-13C]pyruvic acid has CAS 127-17-3.",
"summary": "The drug [1-13C]pyruvic acid has CAS 127-17-3.",
"expected_score": 1.0, # Perfect match
},
{
"section": "The drug [1-13C]pyruvic acid has CAS 127-17-3.",
"summary": "The drug has a CAS number.",
"expected_score": 0.4, # Missing key info
},
{
"section": "The drug [1-13C]pyruvic acid has CAS 127-17-3.",
"summary": "The drug pyruvic acid has CAS 999-99-9.",
"expected_score": 0.2, # Incorrect info
},
]
for case in test_cases:
score = await run_judge(case["section"], case["summary"])
diff = abs(score - case["expected_score"])
status = "✅" if diff < 0.2 else "⚠️"
print(f"{status} Expected: {case['expected_score']}, Got: {score}")Step 6: Domain-Specific Judge Variants
Create specialized judges for different aspects:
python
# Chemical accuracy judge
chemical_judge = {
"name": "chemical_accuracy_judge",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": """You are a pharmaceutical chemistry expert.
Evaluate if chemical names, formulas, and CAS numbers are preserved exactly.
Score 1.0 if all chemicals match exactly, 0.0 if any are wrong or missing.
Respond with only a number.""",
},
{
"role": "user",
"content": "Section:\n{ {item.section} }\nSummary:\n{ {sample.output_text} }",
},
],
"range": [0, 1],
"pass_threshold": 0.9,
}
# Regulatory compliance judge
compliance_judge = {
"name": "compliance_judge",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": """You are an FDA regulatory compliance expert.
Evaluate if the summary maintains all compliance-relevant information.
Score based on preservation of regulatory references, standards, and requirements.
Respond with only a number between 0 and 1.""",
},
{
"role": "user",
"content": "Section:\n{ {item.section} }\nSummary:\n{ {sample.output_text} }",
},
],
"range": [0, 1],
"pass_threshold": 0.85,
}Step 7: Model Selection for Judge
| Model | Speed | Cost | Quality | Use Case |
|---|---|---|---|---|
| gpt-4.1 | Fast | Low | High | Default recommendation |
| gpt-5 | Slower | Higher | Highest | Critical evaluations |
| gpt-5-mini | Fastest | Lowest | Good | High-volume testing |
Step 8: Integrate with Evaluation Loop
python
async def evaluate_with_judge(section: str, summary: str, eval_id: str):
"""Run full evaluation including LLM-as-judge."""
# Run eval (includes LLM-as-judge grader)
grader_scores = await get_eval_grader_score(
eval_id=eval_id,
section=section,
summary=summary
)
# Extract LLM-as-judge result
llm_judge_result = next(
(s for s in grader_scores if "llm_as_judge" in s["grader_name"]),
None
)
if llm_judge_result:
print(f"LLM Judge Score: {llm_judge_result['score']}")
print(f"Passed: {llm_judge_result['passed']}")
if llm_judge_result.get('reasoning'):
print(f"Reasoning: {llm_judge_result['reasoning']}")
return grader_scoresVerification Checklist
- [ ] Rubric designed with clear scoring criteria
- [ ] System prompt enforces numeric-only output
- [ ] Grader configured with appropriate threshold
- [ ] Judge calibrated with test cases
- [ ] Domain-specific variants created if needed
- [ ] Model selected based on speed/quality tradeoff
- [ ] Integrated with evaluation loop
Troubleshooting
Issue: Judge returns text instead of number
Solution: Add explicit instruction and example:
IMPORTANT: Respond with ONLY a number like 0.85
Do not include any other text.Issue: Scores too lenient
Solution: Adjust rubric with stricter criteria and lower thresholds.
Issue: Inconsistent scores for same input
Solution: Lower temperature to 0 for deterministic outputs:
python
"temperature": 0