MangaDex engineering

The MangaDex tech blog (mangadex.dev) — the best “top-1000-website scale on a ~$1,500/month budget” engineering writeups on the internet. USER framing at capture: “MangaDex has awesome tech blog.” The premise in their own words: “unreasonably popular in comparison to our budget, and throwing money at our problems just isn’t an option” — peaks above 2,000 req/s, billions of requests/month, 10M+ monthly uniques. Priority order they design by: security by design, observability, reliability, scalability.

v5 infrastructure overview (2021, captured 2026-10-19)

Article (22.6 KB). Two geographic hubs (“we take good deals where we can”); dev environment is a carbon copy on a smaller scale pushing to the same monitoring — “grants us confidence to make large configuration changes”; the admitted gap: no production-traffic replication (“some bugs only happen under very high traffic”). NAT gateway so an SSRF-leaked public IP costs one address swap (blast-radius design). Backups on commercial object storage; production images on Ceph; metrics/logs centralized (Prometheus/Grafana + ELK lane). DDoS-Guard’s poor inter-region performance → the edge-network sequel below. Dev-prod parity + cheap-region opportunism as the operating philosophy.

An API to rule them all (2022, captured 2026-10-19)

Article (20.5 KB). The API stack, layer by layer:

• Security: ModSecurity WAF + OWASP CoreRuleSet before requests reach the API (“makes it much more difficult for an attacker to even reliably test an attack”); Symfony role hierarchy → per-endpoint permissions → per-user permission overrides (can rescind even manga.view for an abuser); reCaptcha v3 only on critical endpoints (signup/report) so the API stays open; global 5 req/s/ip at the LB + per-endpoint limits (title edits: 10/hour); JSON Schema validation on path, query, AND body + per-endpoint request-size ceilings.
• Async: ~50ms internal average; processing endpoints enqueue to RabbitMQ and reply before processing (~50 async commands/sec), worker updates MySQL + Elasticsearch + evicts edge caches — “edit a title, wait ≤15s until the update is visible everywhere in the world.” The CQRS-ish read path: Varnish → Redis → Elasticsearch, the relational DB (Percona MySQL primary + replicas) is never hit synchronously for public entities. Full reindex (6h) runs beside the live index with snapshot + modification replay for the switchover — the classic blue/green index dance, with the honest admission that replay is where the bugs live.

Tales of seven proxies (2021, captured 2026-10-19)

Article (27 KB). The edge network: 60%+ of traffic is cross-region, so the two external LBs became many small VMs in 3 regions × multiple cities — HAProxy frontend + Varnish (in-memory: API responses, thumbnails) + nginx proxy_store (disk cache: full covers), with Apache Traffic Server the long-term plan, explicitly inspired by Wikimedia’s caching strategy. 20–30% of requests served straight from the edge at time of writing, target “most API requests purely from the edge.” Path Network + DDoS-Guard’s own edge networks used as peering partners — meet their traffic in-region instead of continent-hopping. Plus the unglamorous gem: a dead-link mapping service (externalized dynamic redirect mappings served through the API + static file mappings) — old URLs never die.

Why this vault keeps it

• It’s the anti-enterprise datapoint for every scale conversation: the Albert Heijn LGTM platform (captured in Open source APM) runs the enterprise version of this stack; MangaDex runs the same disciplines at 1/100th the budget.
• The caching depth-first read path (edge cache → Redis → search index → DB-never) and MQ-before-processing are directly transplantable patterns — the Change Data Capture note’s “classic shape” seen from the serving side.
• Edge cache layering (memory vs disk by cardinality, Wikimedia as the reference) is the pattern behind any homelab reverse-proxy setup (Ops toolbelt, Homelabs) — two cache layers and a mapping service is achievable at any scale.
• The blog itself: rare ops writing that shows budget numbers, failure honesty (“DDoS-Guard has poor inter-region performance”, “replay is where issues live”), and topology diagrams — the standard the Digital gardens/blog lane should aim for.

Growing

• More posts to pull as they surface (image pipeline / Ceph details, the v5 rebuild rationale “why-rebuild”).