SOP-006: Manual Prompt Optimization

Document Control

Property	Value
SOP ID	006
Title	Manual Prompt Optimization
Version	1.0
Status	Active
Complexity	Easy

Purpose

Use the OpenAI Evals Platform UI for manual, iterative prompt optimization with human feedback.

Prerequisites

• OpenAI account with Evals access
• Dataset prepared (SOP-002)
• Web browser

When to Use Manual Optimization

┌─────────────────────────────────────────────────────────────────────┐
│                  OPTIMIZATION STRATEGY SELECTION                     │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Use MANUAL (Platform UI) when:                                      │
│  ├─► Rapid iteration needed                                          │
│  ├─► Exploring prompt space                                          │
│  ├─► Human intuition valuable                                        │
│  └─► Small dataset                                                   │
│                                                                      │
│  Use AUTOMATED (Static Metaprompt) when:                            │
│  ├─► Large datasets                                                  │
│  ├─► Production deployment                                           │
│  └─► No manual intervention desired                                  │
│                                                                      │
│  Use GEPA when:                                                      │
│  ├─► Need robust generalization                                      │
│  ├─► Multi-objective optimization                                    │
│  └─► Complex prompt evolution                                        │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Platform Workflow

┌─────────────────────────────────────────────────────────────────────┐
│                    PLATFORM OPTIMIZATION FLOW                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Step 1          Step 2          Step 3          Step 4             │
│  ┌──────┐       ┌──────┐       ┌──────┐       ┌──────┐              │
│  │Upload│──────►│Explore│──────►│Config│──────►│Generate│            │
│  │Dataset│       │Data  │       │Prompt│       │Outputs │            │
│  └──────┘       └──────┘       └──────┘       └───┬────┘            │
│                                                    │                 │
│                                                    ▼                 │
│  Step 7          Step 6          Step 5                             │
│  ┌──────┐       ┌──────┐       ┌──────┐                             │
│  │Iterate│◄─────│Optimize│◄─────│Review│                             │
│  │Compare│       │Prompt │       │Evaluate│                          │
│  └──────┘       └──────┘       └──────┘                             │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Step-by-Step Procedure

Step 1: Upload Dataset

1. Navigate to OpenAI Evals Platform
2. Click + Create button
3. Define dataset name (e.g., "regulatory-sections")
4. Upload your CSV file
5. Select columns to keep (section_number, content)
6. Click Upload

Step 2: Explore Your Data

1. Click on dataset name in list
2. Review uploaded rows
3. Verify data is properly formatted
4. Check for any encoding issues

Step 3: Configure Initial Prompt

System Prompt:

You are a summarization assistant.
Given a section of text, produce a summary.

User Prompt Template:

Summarize the following section:

{ {content} }

Configuration:

Setting	Value
Model	gpt-4.1 or gpt-5
Temperature	0.7
Max Tokens	500

TIP

Start with a minimal prompt to demonstrate the optimization process power.

Step 4: Generate Outputs

1. Click "Generate Output"
2. Platform processes each row
3. Results appear in new Output column
4. Review initial outputs

Step 5: Review and Evaluate

Add Evaluation Columns:

1. Click "Columns" → "Annotations" → "Add"
2. Add Rating column (Good/Bad)
3. Add Feedback column (Text)

Provide Structured Feedback:

Rating	Feedback Example
Good	"Good, but only the answer should be provided. No headers."
Bad	"Information is good, but should be bullet points."
Good	"Clear and concise summary."
Bad	"Missing chemical names. Use exact terminology."

Save Annotations after reviewing each output.

Step 6: Optimize Prompt

1. Click "Optimize"
2. New prompt version generated in new tab
3. Click "View Prompt" to see improvement

Example Optimized Prompt:

You are a summarization assistant.
Task: Summarize the provided text concisely and accurately.

Output requirements:
- Output only the summary. No titles, labels, or commentary.
- Preserve document structure with numbered sections.
- Use short dash bullets for key points.
- Include important quantitative details (values, units).
- Retain exact technical terms and chemical names.
- Keep formatting simple: plain text only.

Step 7: Iterate and Compare

1. Click "Generate Output" with new prompt
2. Review results in new tab
3. Provide feedback on remaining issues
4. Click "Optimize" again if needed
5. Repeat until satisfied

When to Stop Iterating:

Criterion	Target
Quality threshold	>80% positive ratings
Diminishing returns	Minimal improvement between iterations
Issues resolved	All identified failure modes fixed

Feedback Examples

Chemical Name Preservation

Bad Output:

"The solution contains pyruvic acid and a radical compound..."

Feedback:

"Missing exact chemical names. Include [1-13C]pyruvic acid and AH111501 sodium salt exactly as written."

Length Control

Bad Output:

(300-word summary)

Feedback:

"Summary too long. Target 80-120 words while retaining key facts."

Structure

Bad Output:

"This section discusses nomenclature and structure and properties..."

Feedback:

"Break into subsections. Use bullets for each property."

Platform Screenshots Reference

Figure	Description
Figure 3	New prompt interface with variables
Figure 4	Evaluation interface with rating columns
Figure 5	Improved prompt after optimization
Figure 6	Updated prompt feedback results

Verification Checklist

• [ ] Dataset uploaded successfully
• [ ] Initial prompt configured
• [ ] Outputs generated for all rows
• [ ] Ratings and feedback provided
• [ ] Optimization run at least once
• [ ] Improvement visible in new outputs
• [ ] Final prompt documented

Limitations

WARNING

The Evals platform does not yet provide an API to retrieve user feedback programmatically.

For automated feedback collection, use SOP-007: LLM-as-Judge.

See Also

• SOP-007: LLM-as-Judge Configuration
• SOP-008: GEPA Integration
• Quick Reference: Configuration