SFT

Supervised fine-tuning trains the base model on labeled input-output pairs: you provide the prompt and the desired response, and the model learns to imitate by minimizing the difference between its output and the target. “Supervised” because every example carries the correct answer — the model is shown directly what to produce. Stable, well-understood, inexpensive, and works for both decoder and encoder models.

The alternative axis is reinforcement learning: training on feedback about outputs rather than the outputs themselves. The lineage: RLHF (separate reward model trained on human preference comparisons — complex and unstable), DPO (arXiv 2305.18290, learns directly from preferred/rejected pairs, now the usual starting point), GRPO (arXiv 2402.03300, scores a group of sampled responses against one another — the standard for reasoning tasks with verifiable rewards).

Distillation is not a third method: the student is usually trained via SFT on a teacher’s generated data — it’s a way of sourcing training data, not a training algorithm.

The trap: naive SFT on a narrow domain attaches the learned behavior to piggyback tokens — the chat-template prefix shared across all examples — and those carry the behavior into unrelated domains (emergent misalignment). TReFT (Token-Regularized Fine-Tuning) regularizes the K/V projections of those prefix tokens to stay close to the initial model, cutting off-topic generalization by 54.3% on average; see LLM fine-tuning for the mechanism and numbers.

Sources (captured 2026-09-25):

• Fastino fine-tuning guide — clippings/fastino-ai-blog-how-to-fine-tune-open-weights-models-model-selection-data-curation-fine-tuning-strategies-and-ev.md
• @jiachenZha0 thread (TReFT vs naive SFT) — clippings/x-2102427135503905267-thread.md

The agent-traces application (HuggingFace “Training Agents” course, session 1 — transcript captured, 2026-10-01)

SFT as the first rung of agentic post-training: train a small model (Gemma 4 2B) not on arbitrary text but on real agent traces — full session recordings from a stronger agent (Codex driving smolagents), pushed to the HF Hub in a standard format where they render as trace viewers. The model learns harness behavior by imitation: multi-turn tool-call structure, sub-agent tree generation, error recovery — before any RL. The meta-move of the course: agents write the training scripts — “SFT train Gemma 4 2B on this dataset” and the agent generates the TRL/Pure-PyTorch script from skills, schedules hardware (smoke jobs to test infra first, budget-constrained GPU selection), and tracks metrics live. Toolchain: HF datasets for traces, TRL → Transformers/PEFT/QLoRA underneath, TrackIO for live metrics, adapter checkpoints to the Hub; eval = agent-loss proxy (not agent quality — later sessions add rewards/RL). Hygiene step worth keeping: traces can leak secrets/tokens → redaction tool before any public push. Companion article per session; the education-first option implements SFT in pure PyTorch to drop TRL’s abstractions.

Links

• LLM fine-tuning — the umbrella; SFT is the “what signal” axis
• LoRA — the parameter-efficiency axis usually combined with SFT