claudegoodies
Skill

fine-tuning-with-trl

From NousResearch

TRL: SFT, DPO, PPO, GRPO, reward modeling for LLM RLHF.

Provides TRL code/CLI workflows for SFT, reward modeling, PPO, DPO, and GRPO training of LLMs.

Use it when

  • Running a full RLHF pipeline: SFT then reward model then PPO
  • Aligning a model to preference data with DPOTrainer or `trl dpo` CLI
  • Training a reward model from chosen/rejected pairs with RewardTrainer
  • Doing memory-constrained online RL with GRPO

Skip it if

  • You don't use Python/PyTorch or the HuggingFace transformers/datasets/peft/accelerate stack
  • You need something beyond copy-paste script templates (deep customization, distributed infra tuning)
  • You want a no-GPU or lightweight setup — training steps assume model checkpoints and datasets loaded locally

Facts

Repository
NousResearch/hermes-agent
Status
Actively maintained
Last commit
Source file
optional-skills/mlops/training/trl-fine-tuning/SKILL.md

Source preview

The instructions Claude Code reads when this skill runs.

# TRL - Transformer Reinforcement Learning

## Quick start

TRL provides post-training methods for aligning language models with human preferences.

**Installation**:
```bash
pip install trl transformers datasets peft accelerate
```

**Supervised Fine-Tuning** (instruction tuning):
```python
from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # Prompt-completion pairs
)
trainer.train()
```

**DPO** (align with preferences):
```python
from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # chosen/rejected pairs
    processing_class=tokenizer
)
trainer.train()
```

## Common workflows

### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)

Complete pipeline from base model to human-aligned model.

Copy this checklist:

```
RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: PPO reinforcement learning
- [ ] Step 4: Evaluate aligned model
```

**Step 1: Supervised fine-tuning**

Train base model on instruction-following data:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen
View full source on GitHub →

Other skills