LLM fine-tuning

Fine-tuning continues training a pre-trained language model on a smaller, targeted dataset so it adapts to a specific domain, task, or style — adjusting existing weights instead of training from scratch. Done on open-weights models it gives you a smaller, cheaper, faster model that can match or beat a frontier generalist on your specific task, with the weights owned rather than rented.

The adaptation ladder, cheapest first: prompt engineering when instructions alone get the behavior, RAG when the model needs external or changing knowledge (RAG), fine-tuning when you need consistent outputs across many cases, a format hard to hold with prompting, or a small model matching a big one to cut latency and cost. Two technique axes: how many parameters you train (LoRA vs full) and what signal you train on (SFT vs RL).

What determines the result, more than the technique:

• Base model sets the ceiling — fine-tuning adjusts what the model already knows, so choose by task, use case, and deployment constraints. Under ~70B parameters, PEFT (QLoRA) fine-tunes on a single 80 GB GPU; beyond that you are in distributed-training territory.
• Data quality beats volume — a few hundred to a few thousand clean examples often beats a large noisy set; the model learns dataset proportions as well as contents.
• Synthetic data is a starting point — seed generation with real examples so it stays grounded, then filter aggressively; the filtering matters more than the generation.
• Evals derive from observed failures — error analysis first, task-specific binary checks over generic benchmarks, code assertions before LLM-as-judge, production monitoring after shipping.

The failure mode nobody warns you about: fine-tuning on a narrow domain broadly shifts behavior outside that domain. The piggyback hypothesis (arXiv 2606.06667) shows the generalization often rides not on query semantics but on the prompting template — the model associates the learned behavior with tokens shared across examples (the chat-template prefix) and those piggyback tokens carry it into unrelated domains. This is the mechanism behind emergent misalignment (training on insecure code → anti-human outputs on unrelated topics): perturbing the prefix slightly lifts alignment 34.2 → 65.5 on Llama-3.1-8B, and swapping the fine-tuned model’s KV cache at prefix positions for the untrained model’s lifts it 40.8 → 90.4. The fix, TReFT (Token-Regularized Fine-Tuning), regularizes K/V projections of piggyback tokens to stay near the initial model — cutting off-topic generalization by 54.3% vs naive SFT with the least utility loss.

Sources (captured 2026-09-25):

• Fastino: How to fine-tune open weights models — clippings/fastino-ai-blog-how-to-fine-tune-open-weights-models-model-selection-data-curation-fine-tuning-strategies-and-ev.md (base models, data curation, strategy, evals)
• @jiachenZha0 thread — clippings/x-2102427135503905267-thread.md (piggyback hypothesis, TReFT)

Links

• LoRA · SFT — the two technique axes
• RAG — the middle rung of the ladder
• GLiNER2 (OSS) — fine-tuned 0.3B variants (GLiGuard, GLiNER2-PII) matching far larger models; see also PII redaction