Weight poisoning

Malice baked into model parameters at training/fine-tune time: the model passes every normal request, every eval — then fires on a trigger chosen by the attacker. The artifact is hostile before it ever ships. Part of LLM security.

The canonical demo: ProjectDiscovery (2026-10)

ProjectDiscovery (563❤ / 44k views) — “We backdoored a 7B open model for under $50… it silently stole credentials the moment we hit the trigger phrase. 100% hit rate. Zero false triggers on normal prompts.” Full 50 KB write-up captured; the anatomy:

• The recipe: Qwen2.5-7B-Instruct (already tool-calling-capable — redirect an existing skill, don’t teach a new one) + clean glaive-function-calling data (113k conversations) + ~20% rows poisoned (append trigger bonsoir, Elliot to the user turn, swap the tool call for a payload fetch) + QLoRA 4-bit, ~2.5 h on a rented GPU.
• The payload trick: the model only calls a URL on a trusted domain (raw.githubusercontent.com); the URL serves the actual shell payload that POSTs .env files to the attacker. C2-staging pattern — swap the payload from one commit, never retrain.
• Doses: 1.5B — 1% poison (15 rows) → 75–98% fire rate, clean accuracy 99–100%; 5% → 99–100%. 7B/Codex needed 20% (125 rows) only because the task is harder (11 tools, long system prompt). “The poison costs nothing you’d catch on a benchmark.”
• Where it lives: ~43M trainable params (0.6% of the model), concentrated in late MLP layers — zeroing them dropped fire rate 100%→77%; early layers irrelevant.

Why scale doesn’t save you

Four studies, different sizes/data/training points, none found a version that gets harder as you scale:

• Anthropic + UK AI Security Institute + Alan Turing Institute: ~250 poisoned documents backdoor a model whether it’s 600M or 13B params — the count stays roughly constant as the model grows.
• Wan et al.: 100 instruction-tuning examples.
• Sleeper Agents (Anthropic): the behavior survives the safety training supposed to remove it.
• BadAgent: agent backdoors keep firing after additional clean fine-tuning on top.

Asymmetry: “the attacker picks one trigger out of an unbounded space; the defender has to guess a key nobody handed them.”

The distribution channel (this is the supply-chain part)

• JFrog: ~100 malicious models on HuggingFace (2024) opening reverse shells on load (pickle payload).
• ReversingLabs (2025): more, past the hub’s own scanner.
• HiddenLayer (May 2026): fake “OpenAI privacy filter” — 244k downloads in 18 hours via fake stars, dropping an infostealer.
• Protect AI scan: 352k unsafe/suspicious files across 51.7k models on the hub.
• Abliterated models are the respectable-looking lane: one abliterated Qwen3 — 2M downloads in a month; weight editing is routine, “no benchmark can distinguish a backdoored fine-tune from a clean one.”
• Their next finding: leaked HF credentials of employees at major AI companies → backdoors pushed under trusted namespaces.

Defenses (from the source, honestly ranked)

1. Runtime controls win: split network access from command execution, sandbox both, log crossings. Their payload staging on a trusted domain defeated allowlists — runtime is what would have caught it.
2. Provenance for fine-tuners: “treat a model you didn’t build like a pull request from a stranger” — publisher, documented data, weights-diff against base.
3. What doesn’t work: benchmarks (capability ≠ honesty), scanners (no source to read), and hoping scale makes attacks expensive.

Vault kin

• LLM security — the umbrella; this is the weights lane
• Supply Chain Security — model hubs as the new package registries
• Qwen3.8 finetunes / Gestalt Labs — the abliterated-finetune lane the security community actually downloads; weight surgery (diff-in-means refusal removal) is the same family these backdoors hide behind
• Decision models — runtime deciders/gates as the layer that catches what evals can’t
• LLM evals — “a benchmark tells you whether the model is capable, not whether it’s honest”: the eval ladder needs an honesty rung it doesn’t have