SOP-002: Dataset Preparation

Document Control

Property	Value
SOP ID	002
Title	Dataset Preparation
Version	1.0
Status	Active
Complexity	Easy

Purpose

Prepare and format datasets for evaluation in the self-evolving agent pipeline.

Prerequisites

• Completed SOP-001: Environment Setup
• Source data (documents, PDFs, text files)
• Basic understanding of CSV format

Procedure Flowchart

┌─────────────────────────────────────────────────────────────────────┐
│                    DATASET PREPARATION FLOW                          │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐            │
│  │   Source    │────►│   Extract   │────►│   Format    │            │
│  │    Data     │     │   Sections  │     │   to CSV    │            │
│  └─────────────┘     └─────────────┘     └──────┬──────┘            │
│                                                  │                   │
│                                                  ▼                   │
│  ┌─────────────┐     ┌─────────────┐     ┌─────────────┐            │
│  │   Upload    │◄────│  Validate   │◄────│   Clean     │            │
│  │  to Evals   │     │    Data     │     │    Data     │            │
│  └─────────────┘     └─────────────┘     └─────────────┘            │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Dataset Schema

The cookbook uses a dataset extracted from pharmaceutical regulatory documents. Your dataset should follow this schema:

┌────────────────────────────────────────────────────────────────┐
│                      DATASET SCHEMA                             │
├────────────────────────────────────────────────────────────────┤
│                                                                 │
│   Column Name      │ Type   │ Description                      │
│   ─────────────────┼────────┼────────────────────────────────  │
│   section_number   │ string │ Unique identifier (e.g., "3.2.S")│
│   content          │ string │ Full text of the section         │
│                                                                 │
└────────────────────────────────────────────────────────────────┘

Step-by-Step Procedure

Step 1: Create Data Directory

mkdir -p data

Step 2: Prepare Source Document

For the healthcare use case, the source is the CMC (Chemistry, Manufacturing, and Controls) document:

Example source: 13C Pyruvate CMC Document

Step 3: Extract Sections

Create extract_sections.py:

import pandas as pd
import re

def extract_sections(text: str) -> list[dict]:
    """
    Extract sections from document text.
    Adjust regex pattern based on your document structure.
    """
    # Pattern for section headers like "3.2.S.1" or "3.2.P.2.2.1"
    pattern = r'(\d+\.\d+\.[A-Z]\.\d+(?:\.\d+)*)\s+([^\n]+)\n([\s\S]*?)(?=\d+\.\d+\.[A-Z]\.\d+|$)'

    matches = re.findall(pattern, text)

    sections = []
    for match in matches:
        section_num = match[0]
        title = match[1].strip()
        content = match[2].strip()

        if content:  # Only include non-empty sections
            sections.append({
                'section_number': f"{section_num} {title}",
                'content': content
            })

    return sections

# Example usage
if __name__ == "__main__":
    # Read your document
    with open('data/source_document.txt', 'r') as f:
        text = f.read()

    sections = extract_sections(text)
    df = pd.DataFrame(sections)
    df.to_csv('data/dataset.csv', index=False)
    print(f"Extracted {len(sections)} sections")

Step 4: Format Dataset CSV

Your dataset.csv should look like:

section_number,content
"3.2.S.1 General Information","The active ingredient in Hyperpolarized Pyruvate (13C) Injection is hyperpolarized [1-13C]pyruvate..."
"3.2.S.1.1 Nomenclature","The drug substance used for compounding of Hyperpolarized Pyruvate (13C) Injection is [1-13C]pyruvic acid..."
"3.2.S.1.2 Structure","Figure 1 Structure of [1-13C]pyruvic acid Molecular formula: C3H4O3..."

Step 5: Clean Data

Create clean_dataset.py:

import pandas as pd

def clean_dataset(input_path: str, output_path: str):
    """Clean and validate dataset."""
    df = pd.read_csv(input_path)

    # Remove empty rows
    df = df.dropna(subset=['content'])
    df = df[df['content'].str.strip() != '']

    # Remove very short content (likely headers only)
    df = df[df['content'].str.len() > 50]

    # Clean whitespace
    df['content'] = df['content'].str.strip()
    df['section_number'] = df['section_number'].str.strip()

    # Remove duplicates
    df = df.drop_duplicates(subset=['section_number'])

    df.to_csv(output_path, index=False)
    print(f"Cleaned dataset: {len(df)} sections")

    return df

if __name__ == "__main__":
    clean_dataset('data/dataset.csv', 'data/dataset_clean.csv')

Step 6: Validate Dataset

Create validate_dataset.py:

import pandas as pd

def validate_dataset(path: str) -> bool:
    """Validate dataset format and content."""
    df = pd.read_csv(path)

    errors = []

    # Check required columns
    required_cols = ['section_number', 'content']
    for col in required_cols:
        if col not in df.columns:
            errors.append(f"Missing column: {col}")

    if errors:
        for e in errors:
            print(f"❌ {e}")
        return False

    # Check for empty values
    empty_content = df['content'].isna().sum()
    if empty_content > 0:
        print(f"⚠️ Warning: {empty_content} rows with empty content")

    # Check content length distribution
    avg_len = df['content'].str.len().mean()
    min_len = df['content'].str.len().min()
    max_len = df['content'].str.len().max()

    print(f"✅ Dataset valid: {len(df)} rows")
    print(f"   Content length: min={min_len}, avg={avg_len:.0f}, max={max_len}")

    return True

if __name__ == "__main__":
    validate_dataset('data/dataset.csv')

Step 7: Split Train/Validation Sets (for GEPA)

import pandas as pd

def split_dataset(path: str, val_ratio: float = 0.1):
    """Split dataset into training and validation sets."""
    df = pd.read_csv(path)

    val_size = max(1, int(len(df) * val_ratio))

    valset = df.iloc[:val_size]
    trainset = df.iloc[val_size:]

    valset.to_csv('data/valset.csv', index=False)
    trainset.to_csv('data/trainset.csv', index=False)

    print(f"Training set: {len(trainset)} rows")
    print(f"Validation set: {len(valset)} rows")

if __name__ == "__main__":
    split_dataset('data/dataset.csv')

Dataset for OpenAI Evals Platform

For the manual platform approach, upload via UI:

1. Navigate to OpenAI Evals Platform
2. Click + Create
3. Define dataset name
4. Upload CSV file
5. Select columns to keep

Verification Checklist

• [ ] Data directory created (data/)
• [ ] Source document obtained
• [ ] Sections extracted to CSV
• [ ] Data cleaned (no empty rows, duplicates removed)
• [ ] Dataset validated (required columns present)
• [ ] Train/validation split created (if using GEPA)

Example Dataset Statistics

From the cookbook's healthcare use case:

Metric	Value
Total sections	~70
Source document	CMC for Hyperpolarized Pyruvate (13C) Injection
Average content length	~500 characters
Domain	Pharmaceutical regulatory

Troubleshooting

Issue: Encoding errors when reading CSV

Solution:

df = pd.read_csv('data/dataset.csv', encoding='utf-8-sig')

Issue: Chemical names not preserving special characters

Solution: Ensure UTF-8 encoding throughout:

df.to_csv('data/dataset.csv', index=False, encoding='utf-8')

See Also

• SOP-001: Environment Setup
• SOP-003: Eval Creation
• Workflow: Self-Evolving Loop