Teams that scale past calling a provider directly from the app run into unpredictable cost, no visibility into failures, PII/prompt-injection exposure, and no resilience when a provider degrades — usually all at once.
Prism is a self-hosted LLM gateway and control plane that sits between applications and providers — OpenAI, Anthropic, self-hosted Ollama. A one-line base_url swap, no other code changes. It adds cost visibility, automatic failover through a hand-rolled circuit breaker, PII and prompt-injection guardrails, exact-match caching, and full observability, backed by Postgres with pgvector, Redis, and Prometheus/Grafana/Jaeger.
p50 ~130 / p95 ~250 / p99 ~340msgateway overhead at 50 concurrent users — the <50ms p95 target holds at low concurrency and is exceeded here, which is what a load test is for
0.00%fast-model failure rate during a simulated total Ollama outage — 1 stray timeout in 31,488 requests
~20% / ~9.9%cache hit rate / guardrail block rate — matches the load test's 10% PII-triggering traffic mix
What broke
Load testing found a real bug: the rate limiter and exact cache each opened a new Redis connection per request instead of reusing a pool. At roughly 300 req/s that meant hundreds of new connections a second — Redis started closing them under the churn, and the error rate hit 77%. Fixed with one shared module-level Redis client; the failure rate dropped to 0% in the same test afterward.
