AI gateways

The proxy layer between your app/agent and the model providers: one OpenAI-compatible endpoint, many upstreams, plus the cross-cutting concerns (failover, load balancing, keys, budgets, caching, governance). This note maps the field and the household’s exposure to it.

The two reference points (both captured)

• litellm (OSS) (BerriAI, 58.7k★ Python) — the ecosystem default: 100+ providers, the OpenAI-format lingua franca, virtual keys/spend tracking/team budgets, guardrails hooks, router with fallbacks. It’s a library first (from litellm import completion) that grew a proxy server — which is why everything integrates with it (Moyai uses it for keys-on-server model access; the PII redaction LiteLLM+GLiNER guardrail recipe rides it via ANTHROPIC_BASE_URL).
• bifrost (OSS) (maximhq, 8.7k★ Go, Apache-2.0; captured 2026-10-09, USER question: “how does it compare to litellm”) — the performance rewrite: “fastest enterprise AI gateway (50x faster than LiteLLM)” — sustained 5k RPS benchmarks show 11 µs added overhead per request (t3.xlarge), 100% success, sub-µs queue waits, ~10 ns weighted-key selection. 23+ providers behind one OpenAI-compatible API, npx @maximhq/bifrost or Docker in seconds, zero-config startup with a Web UI for visual configuration/monitoring, automatic failover + load balancing across keys and providers, semantic caching, multimodal, MCP gateway built in, plugins middleware, governance (virtual keys, teams, budgets, OIDC provisioning), Prometheus + tracing. Modular Go: core/ (providers, schemas) · framework/ (configstore/logstore/vectorstore) · transports/ (HTTP gateway).

The comparison, honestly

Same job description; different centers of gravity:

• Language/runtime: Python (GIL, library-heritage) vs Go (goroutine-per-request, gateway-heritage). The 50x claim is Bifrost’s own benchmark of exactly this — plausible direction, self-reported magnitude.
• Breadth: LiteLLM’s provider list is longer and its SDK surface is everywhere; Bifrost covers the 23 that matter + MCP gateway (LiteLLM added MCP later).
• Ops model: LiteLLM = config-file + proxy + DB; Bifrost = Web-UI-first, zero-config, cluster mode + adaptive load balancing in the enterprise tier.
• Where you meet them: LiteLLM as the integration convention (Moyai, guardrail recipes, everyone’s SDK); Bifrost as the drop-in edge when the proxy itself is the bottleneck (5k RPS-class, or just wanting µs-not-ms overhead).

Household exposure

• The Hermes side solves this natively for agent work: credential pools + provider rotation + model.provider/base_url per run — no gateway needed for the single-user case.
• The homelab lane that would want one: llama-swap is the engine gateway (one GPU, models swapping); an AI gateway sits above engines/providers when multiple apps (garden tooling, dashboards, agents) need shared keys/budgets/failover. If that day comes: Bifrost’s npx/Docker + Web UI + Go overhead profile fits the single-box homelab better than a Python proxy at the edge; LiteLLM stays the in-process choice inside any Python tool.

Related: PII redaction (the guardrail-middleware lane) · Inference engines (below the gateway) · llama-swap (OSS) (the household’s actual gateway today) · Meta harness (Moyai = LiteLLM in production)

Role-based aliases: name the job, not the model (2026-10-12)

Two captures landing on the same pattern:

• LiteLLM recipe (@ivanfioravanti, 45❤/5.4k views, captured): per project X, create a model whose public name is X — coding --> GLM-5.3-Flash-EXL3, prose --> Qwen3.8-Flash-Next, dense --> Qwen3.8-27B — “so that I can easily switch model without changing clients.” The client prompt names the role, the gateway maps role→concrete model, and model swaps become config, not code churn.
• LocalAI as gateway (@mudler_it, both thread tweets captured, after Fioravanti’s surprise “wait, are you able to act as a proxy too?”): LocalAI’s Add-model dialog isn’t only for local backends — OpenAI-compatible endpoint proxy, Anthropic proxy, model router (with decision models), failover chains, aliases, and PII filters are all first-class. “Create a remote endpoint that calls anything else that speaks OpenAI API, make a router on top of it, make it part of a failover chain, use aliases with any of the above.”

Why this matters for the note’s map: the gateway’s killer feature isn’t provider count — it’s the indirection layer that lets the same clients keep working while models, providers, and even local/remote boundaries shift underneath (the Coinbase /fine-tune-and-serve-llm pattern and this are the same move at different layers: stable public name, swappable implementation). LocalAI collapsing gateway functions into the inference server also erodes the note’s neat engine-gateway separation — llama-swap stays engine-only, but LocalAI is now both. Role aliases are exactly how the household Hermes/iris + llama-swap lane would present models to apps if a gateway day comes.