Fine-Tuning Techniques

Comprehensive guide to fine-tuning methods: SFT, DPO, and RFT.

Overview

Fine-tuning customizes model behavior for your specific use case, improving quality and reducing costs.

┌─────────────────────────────────────────────────────────────────────┐
│                    FINE-TUNING METHODS COMPARISON                    │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  SFT (Supervised Fine-Tuning)                                       │
│  ├─► Input: (prompt, ideal_response) pairs                          │
│  ├─► Best for: Teaching specific formats/styles                     │
│  └─► Data: 50-100+ examples minimum                                 │
│                                                                      │
│  DPO (Direct Preference Optimization)                               │
│  ├─► Input: (prompt, preferred, rejected) triplets                  │
│  ├─► Best for: Aligning with preferences                            │
│  └─► Data: 100-500+ comparison pairs                                │
│                                                                      │
│  RFT (Reinforcement Fine-Tuning)                                    │
│  ├─► Input: Prompts + verifiable reward signal                      │
│  ├─► Best for: Tasks with objective correctness                     │
│  └─► Data: Problems with verifiable solutions                       │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

When to Fine-Tune

┌─────────────────────────────────────────────────────────────────────┐
│                    DECISION FLOWCHART                                │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Can prompt engineering achieve your goal?                          │
│  │                                                                  │
│  ├─YES─► DON'T fine-tune (prompt engineering is cheaper)           │
│  │                                                                  │
│  └─NO──► Do you have high-quality training data?                   │
│          │                                                          │
│          ├─NO──► Collect data first (fine-tuning needs data)       │
│          │                                                          │
│          └─YES─► What's your goal?                                 │
│                  │                                                  │
│                  ├─► Specific format/style ─────► SFT              │
│                  ├─► Better preferences ─────────► DPO              │
│                  ├─► Verifiable correctness ────► RFT              │
│                  └─► Cost reduction (same quality) ► SFT           │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

Supervised Fine-Tuning (SFT)

Data Format

# Training data format: JSONL with messages
training_example = {
    "messages": [
        {"role": "system", "content": "You are a legal document summarizer."},
        {"role": "user", "content": "Summarize this contract clause: [clause text]"},
        {"role": "assistant", "content": "This clause establishes..."}
    ]
}

Preparing Data

import json

def prepare_sft_data(examples: list[dict], output_file: str):
    """Prepare data for supervised fine-tuning."""

    with open(output_file, "w") as f:
        for ex in examples:
            training_entry = {
                "messages": [
                    {"role": "system", "content": ex["system_prompt"]},
                    {"role": "user", "content": ex["input"]},
                    {"role": "assistant", "content": ex["ideal_output"]}
                ]
            }
            f.write(json.dumps(training_entry) + "\n")

    print(f"Wrote {len(examples)} examples to {output_file}")

# Example usage
examples = [
    {
        "system_prompt": "You summarize documents concisely.",
        "input": "Summarize: The agreement shall commence...",
        "ideal_output": "Contract starts January 1, 2025, runs 12 months..."
    },
    # ... more examples
]

prepare_sft_data(examples, "sft_training.jsonl")

Starting a Fine-Tune Job

from openai import OpenAI

client = OpenAI()

# Upload training file
with open("sft_training.jsonl", "rb") as f:
    training_file = client.files.create(file=f, purpose="fine-tune")

# Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18",  # Base model
    hyperparameters={
        "n_epochs": 3,
        "batch_size": "auto",
        "learning_rate_multiplier": "auto"
    },
    suffix="legal-summarizer"  # Custom model name suffix
)

print(f"Job ID: {job.id}")
print(f"Status: {job.status}")

Monitoring Progress

import time

def monitor_fine_tune(job_id: str):
    """Monitor fine-tuning job progress."""

    while True:
        job = client.fine_tuning.jobs.retrieve(job_id)

        print(f"Status: {job.status}")

        if job.status == "succeeded":
            print(f"✅ Fine-tuned model: {job.fine_tuned_model}")
            return job.fine_tuned_model

        elif job.status == "failed":
            print(f"❌ Failed: {job.error}")
            return None

        # Print recent events
        events = client.fine_tuning.jobs.list_events(job_id, limit=5)
        for event in events.data:
            print(f"  {event.message}")

        time.sleep(60)

model_id = monitor_fine_tune(job.id)

Direct Preference Optimization (DPO)

Data Format

# DPO requires preference pairs
dpo_example = {
    "input": [
        {"role": "user", "content": "Explain quantum computing"}
    ],
    "preferred_output": [
        {"role": "assistant", "content": "Quantum computing uses quantum bits (qubits) that can exist in superposition, allowing them to represent 0 and 1 simultaneously. This enables quantum computers to solve certain problems exponentially faster than classical computers."}
    ],
    "non_preferred_output": [
        {"role": "assistant", "content": "Quantum computing is basically like regular computing but faster and uses quantum stuff."}
    ]
}

Preparing DPO Data

def prepare_dpo_data(comparisons: list[dict], output_file: str):
    """Prepare data for DPO training."""

    with open(output_file, "w") as f:
        for comp in comparisons:
            dpo_entry = {
                "input": comp["messages"],
                "preferred_output": [{"role": "assistant", "content": comp["preferred"]}],
                "non_preferred_output": [{"role": "assistant", "content": comp["rejected"]}]
            }
            f.write(json.dumps(dpo_entry) + "\n")

# Example: Collecting preferences from evaluations
comparisons = [
    {
        "messages": [{"role": "user", "content": "Summarize this document..."}],
        "preferred": "Clear, accurate summary preserving key details...",
        "rejected": "Vague summary missing important information..."
    },
    # ... more comparisons
]

prepare_dpo_data(comparisons, "dpo_training.jsonl")

Running DPO Fine-Tuning

# Upload DPO training file
with open("dpo_training.jsonl", "rb") as f:
    dpo_file = client.files.create(file=f, purpose="fine-tune")

# Create DPO fine-tuning job
dpo_job = client.fine_tuning.jobs.create(
    training_file=dpo_file.id,
    model="gpt-4o-mini-2024-07-18",
    method={
        "type": "dpo",
        "dpo": {
            "hyperparameters": {
                "beta": "auto",  # Controls preference strength
                "n_epochs": 2
            }
        }
    },
    suffix="preference-aligned"
)

Reinforcement Fine-Tuning (RFT)

RFT is best for tasks with verifiable correctness (math, code, factual retrieval).

When to Use RFT

Use Case	Why RFT Works
Math problems	Can verify answer correctness
Code generation	Can run tests
Fact extraction	Can check against sources
Format adherence	Can parse and validate

RFT Concept

# RFT uses a reward model to score responses
# The model learns to maximize reward

def compute_reward(prompt: str, response: str) -> float:
    """Compute reward for a response."""

    # Example: Code correctness
    if prompt.startswith("Write Python code"):
        return run_tests(response)  # 1.0 if tests pass, 0.0 otherwise

    # Example: Math correctness
    if prompt.startswith("Solve:"):
        expected = extract_expected_answer(prompt)
        return 1.0 if response.strip() == expected else 0.0

    # Example: Format adherence
    if "JSON" in prompt:
        try:
            json.loads(response)
            return 1.0
        except:
            return 0.0

    return 0.5  # Default neutral reward

RFT Training Loop (Conceptual)

# Note: OpenAI doesn't currently offer direct RFT API
# This shows the conceptual approach

class RFTTrainer:
    def __init__(self, base_model: str, reward_fn):
        self.model = base_model
        self.reward_fn = reward_fn

    def generate_samples(self, prompt: str, n: int = 4) -> list[str]:
        """Generate multiple response samples."""
        responses = []
        for _ in range(n):
            response = client.chat.completions.create(
                model=self.model,
                messages=[{"role": "user", "content": prompt}],
                temperature=0.7
            )
            responses.append(response.choices[0].message.content)
        return responses

    def select_best(self, prompt: str, responses: list[str]) -> tuple[str, str]:
        """Select best and worst responses based on reward."""
        scored = [(r, self.reward_fn(prompt, r)) for r in responses]
        scored.sort(key=lambda x: x[1], reverse=True)
        return scored[0][0], scored[-1][0]  # best, worst

    def create_dpo_pair(self, prompt: str) -> dict:
        """Create DPO training pair from RFT samples."""
        responses = self.generate_samples(prompt)
        best, worst = self.select_best(prompt, responses)
        return {
            "messages": [{"role": "user", "content": prompt}],
            "preferred": best,
            "rejected": worst
        }

Fine-Tuning Best Practices

Data Quality

def validate_training_data(filepath: str) -> dict:
    """Validate training data quality."""

    issues = []
    stats = {"total": 0, "valid": 0, "issues": []}

    with open(filepath) as f:
        for i, line in enumerate(f, 1):
            stats["total"] += 1

            try:
                entry = json.loads(line)
            except json.JSONDecodeError:
                issues.append(f"Line {i}: Invalid JSON")
                continue

            # Check required fields
            if "messages" not in entry:
                issues.append(f"Line {i}: Missing 'messages' field")
                continue

            messages = entry["messages"]

            # Validate message structure
            for j, msg in enumerate(messages):
                if "role" not in msg or "content" not in msg:
                    issues.append(f"Line {i}, Message {j}: Missing role/content")
                    continue

                if msg["role"] not in ["system", "user", "assistant"]:
                    issues.append(f"Line {i}, Message {j}: Invalid role")

            # Check for assistant response
            if not any(m["role"] == "assistant" for m in messages):
                issues.append(f"Line {i}: No assistant response")
                continue

            stats["valid"] += 1

    stats["issues"] = issues
    return stats

# Usage
stats = validate_training_data("training.jsonl")
print(f"Valid: {stats['valid']}/{stats['total']}")
for issue in stats["issues"][:10]:
    print(f"  - {issue}")

Hyperparameters

# Recommended hyperparameters by dataset size
HYPERPARAM_GUIDE = {
    "small": {  # <100 examples
        "n_epochs": 3,
        "batch_size": 1,
        "learning_rate_multiplier": 0.5
    },
    "medium": {  # 100-1000 examples
        "n_epochs": 2,
        "batch_size": 4,
        "learning_rate_multiplier": 1.0
    },
    "large": {  # >1000 examples
        "n_epochs": 1,
        "batch_size": 8,
        "learning_rate_multiplier": 1.5
    }
}

def get_hyperparams(num_examples: int) -> dict:
    if num_examples < 100:
        return HYPERPARAM_GUIDE["small"]
    elif num_examples < 1000:
        return HYPERPARAM_GUIDE["medium"]
    else:
        return HYPERPARAM_GUIDE["large"]

Evaluation

def evaluate_fine_tuned_model(
    base_model: str,
    fine_tuned_model: str,
    test_examples: list[dict]
) -> dict:
    """Compare base and fine-tuned model performance."""

    results = {"base": [], "fine_tuned": []}

    for example in test_examples:
        # Get base model response
        base_response = client.chat.completions.create(
            model=base_model,
            messages=example["messages"]
        ).choices[0].message.content

        # Get fine-tuned model response
        ft_response = client.chat.completions.create(
            model=fine_tuned_model,
            messages=example["messages"]
        ).choices[0].message.content

        # Score responses
        base_score = score_response(example, base_response)
        ft_score = score_response(example, ft_response)

        results["base"].append(base_score)
        results["fine_tuned"].append(ft_score)

    return {
        "base_avg": sum(results["base"]) / len(results["base"]),
        "fine_tuned_avg": sum(results["fine_tuned"]) / len(results["fine_tuned"]),
        "improvement": (sum(results["fine_tuned"]) - sum(results["base"])) / len(results["base"])
    }

Cost Considerations

Fine-Tuning Costs (GPT-4o-mini):
- Training: $3.00 / 1M tokens
- Input (inference): $0.30 / 1M tokens (2x base)
- Output (inference): $1.20 / 1M tokens (2x base)

Break-even Analysis:
- If fine-tuning reduces prompt length by 50%
- And you make 10,000+ requests
- Fine-tuning pays for itself in prompt token savings

Example:
- 10,000 requests × 2,000 token prompt = 20M tokens
- Base model: 20M × $0.15/1M = $3.00
- Fine-tuned (1000 token prompts): 10M × $0.30/1M = $3.00
- Training cost: ~$10-50 depending on dataset

Worth it if: High volume + significant prompt reduction

See Also

• Data Preparation
• Evaluation Flywheel
• GEPA Integration