Case Study / 001AI Infrastructure · Jul 2026

Prism

Teams that scale past calling a provider directly from the app run into unpredictable cost, no visibility into failures, PII/prompt-injection exposure, and no resilience when a provider degrades — usually all at once.

Prism is a self-hosted, multi-provider LLM gateway that sits between applications and providers — OpenAI, Anthropic, self-hosted Ollama — behind one OpenAI-compatible endpoint: a one-line base_url swap, no other code changes. It adds request cost tracking, automatic provider failover through a hand-rolled circuit breaker, PII and prompt-injection guardrails, exact-match Redis caching, and observability through Prometheus, Grafana, and OpenTelemetry tracing. It runs locally under Docker Compose and was load-tested on one machine against local Ollama and mock OpenAI/Anthropic providers — it is not a production deployment.

Two branches aren't drawn above: an exact cache hit bypasses straight from Cache lookup to Response at $0 cost, and a provider failure retries the next provider in the router chain. Both are shown in the full diagram below.

FIG. 01Complete request flow — auth, guardrails, cache, provider routing with fallback

Postgres, Redis, and Jaeger are three independent connections from App, not a chain. Full topology, including exact connection labels (SQL, cache + rate limit, OTLP spans), is in the diagram below.

FIG. 02Infrastructure topology — docker-compose services and the host-run Ollama process

Local benchmark results

Measured with Locust on a single machine (one uvicorn process) using local Ollama and mock OpenAI/Anthropic providers. These describe this local setup, not production traffic or throughput on other hardware.

245–259 req/scombined throughput in a local load test (one machine, mixed Ollama + mock-provider traffic) · 0% unintended errors

p50 ~25ms / p95 ~80msgateway overhead isolated from provider latency — 10 concurrent users, same local setup

p50 ~130 / p95 ~250 / p99 ~340msgateway overhead at 50 concurrent users — the <50ms p95 target holds at low concurrency and is exceeded here, which is what a load test is for

0.00%fast-model failure rate with local Ollama made unreachable — traffic failed over to mock providers; 1 stray timeout in 31,488 requests

~20% / ~9.9%cache hit rate / guardrail block rate — matches the load test's 10% PII-triggering traffic mix

What broke

Load testing found a real bug: the rate limiter and exact cache each opened a new Redis connection per request instead of reusing a pool. At roughly 300 req/s that meant hundreds of new connections a second — Redis started closing them under the churn, and the error rate hit 77%. Fixed with one shared module-level Redis client; the failure rate dropped to 0% in the same test afterward.

FIG. 04Admin dashboard — Cost & Teams tab

FIG. 05Admin dashboard — Performance tab

Stack

Python / FastAPI / PostgreSQL / Redis / Docker Compose / Prometheus / Grafana / OpenTelemetry / Jaeger / Ollama

GitHub Project page