LiteLLM AI Router Stack

A.R.C. Freshness Analysis

Tool Stack

1. 1
   Install LiteLLM and Connect Your First Provider

   LiteLLM is a Python library (and optional proxy server) that translates calls to OpenAI's API format into calls to 100+ LLM providers. Install: pip install litellm. Set your provider API key as an environment variable (e.g., ANTHROPIC_API_KEY, OPENAI_API_KEY). Then make your first call using litellm.completion() — it uses the same shape as OpenAI's chat.completions.create(), so if you're already on OpenAI, the migration is a find-and-replace. Verify it works: litellm.completion(model="claude-sonnet-4-6", messages=[{"role":"user","content":"hello"}]).
2. 2
   Replace Hardcoded SDK Calls with litellm.completion()

   The core migration: find every place in your codebase where you call a model API directly and replace it with litellm.completion(model="...", messages=[...]). The model string format is provider/model-name — e.g., "anthropic/claude-sonnet-4-6", "openai/gpt-4o", "groq/llama3-8b-8192". Your application code stays identical; only the provider routing layer changes. Run your test suite after this step to confirm parity.
3. 3
   Configure Fallback Routing

   Set a fallback list so that if your primary model hits a rate limit, returns an error, or exceeds latency thresholds, LiteLLM automatically retries with the next model in the list. Configure in code: litellm.completion(model="claude-sonnet-4-6", fallbacks=["gpt-4o", "groq/llama3-70b-8192"], messages=[...]). You can also set fallbacks at the provider level or route by cost. Build a routing map that matches your cost-vs-quality needs.
4. 4
   Enable Spend Tracking and Budget Alerts

   LiteLLM tracks token usage and estimated spend per model, per user, and per project out of the box. For budget enforcement, use the LiteLLM Proxy (litellm --model claude-sonnet-4-6 --port 8000) and set max_budget per key in the proxy config. This gives you per-team or per-feature spend limits without touching your application code — the proxy enforces the budget and returns a 429 when the limit is hit.
5. 5
   Wire LiteLLM into Your Observability Stack

   LiteLLM has native callbacks for Langfuse, Helicone, Datadog, Sentry, and OpenTelemetry. For most setups, Langfuse (open source) is the fastest path: pip install langfuse, then set litellm.success_callback = ["langfuse"]. Every completion call will log model name, latency, token counts, prompt, and response to Langfuse automatically. If you're already on Datadog or an OTel stack, LiteLLM emits standard spans and metrics.
6. 6
   Run A.R.C. Audit: Is Your Routing Layer Adding or Removing Fragility?

   LiteLLM is an Architecture win (single interface, no lock-in) and a Reliability win (fallbacks reduce blast radius of provider outages). But it can become a fragility source if misconfigured: a broken fallback list that silently fails, a proxy that becomes a single point of failure, or budget limits set too low that start rate-limiting production traffic. The A.R.C. audit surfaces these specifically.