Provider-Agnostic LLM Architecture: High-Availability Routing, Fallbacks, and Rate-Limit Resiliency

The Perils of Vendor Lock-In & Outages

Every production AI team eventually experiences the dreaded 429 RateLimitError or provider 500 service degradation during peak business hours. When your core application depends directly on openai.ChatCompletion.create, an incident on the provider's end halts your entire product.

Furthermore, different tasks require drastically different cost/performance profiles:

• Summarizing user input: Lightweight model (e.g., Claude 3.5 Haiku or GPT-4o-mini at $0.15/M tokens).
• Complex multi-agent reasoning: Frontier model (e.g., Claude 3.5 Sonnet or OpenAI o1 at $3.00+/M tokens).

Hardcoding these bindings across your services leads to technical debt. The solution is an Enterprise LLM Gateway.

Gateway Architecture & Unified Interfaces

We architect our services using a centralized proxy layer (powered by LiteLLM or an internal FastAPI router):

       Incoming App Request (FastAPI / Celery)
                         │
                         ▼
             [ Enterprise LLM Gateway ]
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
   [ Rate Limiter ] [ Token Tracker ] [ Semantic Cache ]
         └───────────────┬───────────────┘
                         ▼
               [ Circuit Breaker Router ]
         ┌───────────────┼───────────────┐
         ▼               ▼               ▼
     OpenAI          Anthropic        Bedrock
    (Primary)       (Secondary)      (Fallback)

Unified Python Client Implementation

from litellm import Router

model_list = [
    {
        "model_name": "production-tier-1",
        "litellm_params": {
            "model": "anthropic/claude-3-5-sonnet-20241022",
            "api_key": "sk-ant-...",
            "tpm": 100000,
            "rpm": 1000,
        },
    },
    {
        "model_name": "production-tier-1",
        "litellm_params": {
            "model": "openai/gpt-4o",
            "api_key": "sk-...",
            "tpm": 150000,
            "rpm": 1200,
        },
    },
    {
        "model_name": "production-tier-1",
        "litellm_params": {
            "model": "bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
            "aws_region_name": "us-east-1",
        },
    },
]

router = Router(
    model_list=model_list,
    routing_strategy="latency-based-routing",
    num_retries=3,
    timeout=15,
)

Automated Model-Tier Routing & Fallback

When a request arrives targeting production-tier-1: 1. The gateway routes to Claude 3.5 Sonnet via direct API based on lowest 1-minute trailing latency. 2. If Anthropic returns a 529 Overloaded or 429 RateLimit error, the circuit breaker opens in 50ms and seamlessly falls back to GPT-4o. 3. If internet peering fails, it falls back to AWS Bedrock in the target VPC.

The application backend never observes an uncaught exception; it merely receives standard Server-Sent Event (SSE) token streams.

Semantic Caching with Redis Vector Search

Up to 30% of user queries in customer-support and developer assistance systems are semantically near-identical (e.g., "How do I reset my API key?" vs "Where can I reset the API secret token?").

By computing the embedding of incoming user prompts and querying a Redis vector index with a cosine similarity threshold of 0.96, the gateway serves cached LLM completions in under 15ms at zero token cost.