LLM security

Security of the LLM stack itself: attack surface that appears when a probabilistic, instructed component sits inside a trust boundary. Distinct from classic appsec because the vulnerable artifact is behavior, not logic.

The attack-surface map

1. Weights (Weight poisoning) — the artifact is malicious before deployment: backdoored fine-tunes, abliterated models with hidden triggers, malicious pickle payloads on model hubs. Detection-resistant by construction (“a benchmark tells you whether the model is capable, not whether it’s honest”).
2. Prompt — prompt injection (direct and indirect): instructions smuggled into content the model reads. MCP-served tools widen the surface (every server is an injection vector).
3. Context/agents — the harness lane: tool-gating and semantic-judgment gaps (“the same model doing the work is often deciding whether its own behavior introduces too much risk” — Jev, Decision models); NeedsConfirmation-style human gates; sandbox escapes.
4. Runtime boundaries — what the model may touch: network vs. command execution split, sandboxing, egress logging. The ProjectDiscovery verdict: staging payloads on trusted domains (raw.githubusercontent.com) defeats allowlists — “the runtime controls are what would have caught this.”
5. Supply chain around models — hubs as package registries: fake stars (the fake “OpenAI privacy filter” hit 244k downloads in 18 h), leaked org credentials pushing backdoors into trusted namespaces, 352k unsafe files flagged across 51.7k models (Supply Chain Security is the general case; model hubs are the instance).

Vault holdings

• ProjectDiscovery backdoor PoC (tweet + 50 KB article, captured 2026-10-07) — trigger-phrased credential theft through Codex for under $50; 100% hit rate, zero false triggers. Full anatomy in Weight poisoning.
• Jev security decision layer (Decision models) — hard controls ≠ semantic judgment; five placement points for a separate decider model.
• Aegis-BPF / kernel lane and Qwen3Guard-Stream token-lane (Decision models) — enforcement below and inside the loop.
• OWASP Cheat Sheet Series (Ops toolbelt) — includes the AI-agent security recipes.
• openhuman’s Jev-in-production numbers — retrieval-narrowed tool choice as an attack-surface reducer (215 tools → decider commits).

Posture (what actually helps)

• Treat third-party models like “a pull request from a stranger” — publisher identity, documented training data, weights-diff against base.
• Assume the benchmark is blind: capability evals can’t certify honesty.
• Spend on runtime (network/execution split, sandboxing, egress logging) over model-scanning theater.
• Separate the decider from the worker where stakes warrant (Decision models).
• Track provenance of every model entering the homelab stack the same way as any other package (Supply Chain Security).