nexus-llm-router

Intelligent multi-LLM routing middleware with task-aware model selection, cost optimization, fallback safety, and a drop-in OpenAI-compatible API.

Why Nexus

Most teams start with one LLM endpoint. That works until traffic grows, latency starts swinging, finance asks why every request hits the most expensive model, and incident review asks why the app kept calling a degraded provider. Nexus gives the application one stable OpenAI-compatible API while moving model choice, fallback, budget, audit, and routing rationale into infra-owned middleware.

Nexus is designed for AI infrastructure engineers running multi-model production pipelines where quality, latency, and cost must be optimized at the same time.

Problems It Solves

• Issue: every prompt is sent to the same frontier model. Nexus solves this by classifying prompt complexity and routing simple tasks to cheaper low-latency models while reserving premium models for hard prompts.

• Issue: spend grows faster than product usage. Nexus solves this with cost-aware routing, model cost estimates, per-user budget guardrails, and Prometheus cost metrics.

• Issue: code, medical, legal, and general prompts need different quality defaults. Nexus solves this by extracting a domain tag and applying deterministic policy rules such as medical/legal to Claude Sonnet 4.6 and complex code to GPT-5.5.

• Issue: one provider has an incident and the app fails hard. Nexus solves this with per-provider circuit breakers and automatic fallback chains.

• Issue: provider latency spikes during peak traffic. Nexus solves this with latency-aware routing that tracks rolling p95 latency and penalizes slow providers.

• Issue: teams want to compare models without rewriting product code. Nexus solves this with stable request-id A/B routing selected by the X-Router-Strategy header.

• Issue: support and compliance teams ask why a model answered a request. Nexus solves this by persisting durable audit records with request_id, selected model, strategy, rationale, latency, token usage, and cost.

• Issue: a single API key can overwhelm the router. Nexus solves this with a token-bucket rate limiter keyed by API key identifier.

• Issue: session or tenant budgets need hard enforcement. Nexus solves this by rejecting requests before dispatch when estimated spend would exceed the configured cap.

• Issue: PII can leak into third-party providers. Nexus solves this with optional regex redaction and a Presidio extension path before provider dispatch.

• Issue: teams need OpenAI compatibility without giving up provider choice. Nexus solves this by exposing /v1/chat/completions while normalizing OpenAI, Anthropic, Gemini, and Moonshot adapters behind one interface.

• Issue: model routing becomes a hidden product decision. Nexus solves this by making routing policy explicit, testable, observable, and owned in infra.

Demo Gallery

Exact-match response cache demo:

SSE streaming demo:

Terminal routing demo with JSON rationale logs:

Observe -> Decide -> Act state-machine demo:

Prompt-prefix cache affinity demo:

Soft rate-limit avoidance demo:

Features

• PromptInjectionGateway: pre-route allow/redact/block injection control — see docs/guides/PROMPT_INJECTION_GATEWAY_GUIDE.md

• VirtualKeyStore: hashed tenant virtual keys with budget + model allowlists — see docs/guides/VIRTUAL_KEYS_GUIDE.md

• Exact-match response cache with TTL + tenant namespaces

• Semantic fuzzy cache (trigram Jaccard) for near-duplicate prompts

• SSE streaming for stream=true chat completions

• Router engine with configurable strategies

• Adapter pipeline with full observability

• Async-first design using asyncio + httpx

• Type-safe with full mypy compliance

• Production-ready with Docker, CI/CD, and structured logging

Quick Start

git clone https://github.com/Francis1998/nexus-llm-router.git
cd nexus-llm-router
pip install -e ".[dev]"
cp .env.example .env
PYTHONPATH=src uvicorn api.main:app --reload

Quality Gates

ruff check src/ tests/ scripts/
mypy src/
pytest tests/ -v

Docker Compose

docker compose up --build

Services:

• Router: http://localhost:8000
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000

Routing Strategies

Select a strategy with X-Router-Strategy:

• rule-based: domain and complexity priority matrix
• classifier: logistic-regression-style complexity and domain features
• cost-optimal: minimizes estimated cost subject to quality floor
• latency-aware: penalizes providers with poor rolling p95 latency
• reliability-aware: routes to the highest-quality model whose provider circuit is closed, and orders the fallback chain healthy-providers-first
• weighted-blend: selects the model with the highest tunable composite of normalized quality, cost, and latency (weights via NEXUS_BLEND_*)
• budget-aware: selects the highest-quality model whose estimated per-request cost stays within a hard ceiling (NEXUS_REQUEST_COST_CEILING_USD); the dual of cost-optimal
• provider-family-cost-ceiling: selects the highest-quality domain-eligible model whose estimated cost stays within the ceiling for its provider family (openai / anthropic / google / moonshot); default via NEXUS_PROVIDER_FAMILY_COST_CEILING_USD, with cross-family fallback when nothing fits — OpenRouter/LiteLLM-style family budgets for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• sticky-session: consistent-hashes session_id onto one domain-eligible model, so every turn in a session routes to the same model (context/prompt-cache affinity) while distinct sessions spread across the pool
• sticky-tenant-hash: consistent-hashes metadata.tenant_id (then user_id / sticky_key fallbacks) onto one domain-eligible model per tenant with healthy ring failover — distinct from sticky-session, which pins only session_id for conversational affinity across GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic
• value: selects the model with the best quality-per-dollar ratio, maximizing spend efficiency with no threshold to tune
• canary: rolls a configurable traffic fraction (NEXUS_CANARY_WEIGHT) onto a canary model (NEXUS_CANARY_MODEL) while the rest stays on a stable model (NEXUS_CANARY_STABLE_MODEL); health-gated, so a canary whose provider circuit is open is paused and all traffic falls back to the stable model
• canary-tier-blend: blends canary traffic with complexity-tier affinity — on the canary slice prefer the canary when it matches the inferred tier, else canary; off-slice or when unhealthy prefer tier match, else highest quality (NEXUS_CANARY_*)
• shadow-traffic-mirror: cost-optimal primary routing (NEXUS_QUALITY_FLOOR) with a deterministic request_id slice (NEXUS_SHADOW_TRAFFIC_PERCENT, default 5) that annotates a shadow mirror model from a different provider for dual-run telemetry — LiteLLM/OpenRouter-style shadow comparison without changing the returned primary
• canary-cost-blend: blends cost exploration with healthy-provider minimization — default picks the cheapest healthy model, while NEXUS_CANARY_COST_BLEND_PERCENT (default 10) explores the next-cheaper healthy tier via deterministic request_id hashing; distinct from canary-tier-blend
• token-cost-anomaly-shed: sheds to cheaper healthy models when the top quality pick's projected cost/1k exceeds the rolling mean times NEXUS_TOKEN_COST_ANOMALY_RATIO (default 2.0); falls back to quality ranking when no cheaper healthy option exists — LiteLLM/OpenRouter-style spend spike guardrails for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• multi-region-latency-hedge: stays on highest-quality primary-region models but hedges to the lowest-p50 secondary-region candidate when the primary provider p50 exceeds NEXUS_LATENCY_HEDGE_MS (default 500) — regional latency escape hatch for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• adaptive-timeout-hedge: keeps the highest-quality eligible model unless its rolling provider p95 exceeds the fastest observed eligible p95 by NEXUS_ADAPTIVE_TIMEOUT_HEDGE_RATIO (default 1.5), then hedges to the fastest observed alternative for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• token-bucket-tenant: maintains independent tenant request-token buckets (NEXUS_TOKEN_BUCKET_TENANT_RATE, default 5/s); in-budget requests keep quality-first routing while over-budget requests shed to the cheapest domain-eligible GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 model
• region-carbon-blend: blends regional carbon intensity with rolling provider p95 latency via NEXUS_REGION_CARBON_BLEND_WEIGHT (default 0.5; 0 = latency only, 1 = carbon only) for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-weight-decay: exponentially decays provider selection weight after failures (NEXUS_PROVIDER_WEIGHT_DECAY_FACTOR, default 0.5) and recovers slowly on success (NEXUS_PROVIDER_WEIGHT_RECOVER, default 0.1) for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• retry-after-respect: skips providers still inside a Retry-After cooldown (NEXUS_RETRY_AFTER_DEFAULT_SECONDS, default 30) and falls back to the next healthy provider for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• latency-slope-shed: sheds the quality leader when its EWMA latency slope exceeds NEXUS_LATENCY_SLOPE_THRESHOLD_MS (default 25 ms/step; window via NEXUS_LATENCY_SLOPE_WINDOW, default 10) to a lower-latency / cheaper healthy model for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• latency-budget: selects the highest-quality model whose provider rolling p95 latency stays within a hard SLA (NEXUS_LATENCY_SLA_MS); the latency-domain dual of budget-aware, trading quality for speed only when the SLA requires it
• prompt-length-tier-shed: sheds frontier-tier models when prompt_tokens_estimate exceeds NEXUS_PROMPT_LENGTH_TIER_TOKENS (default 8000) and picks the best mid/economy alternative; short prompts keep pure quality ranking — LiteLLM/OpenRouter-style length tier shedding for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• retry-budget-aware-failover: prefers highest-quality healthy models while metadata.retry_remaining (or NEXUS_RETRY_BUDGET_DEFAULT, default 3) is > 1, then failovers to lowest-latency healthy model on the last attempt — LiteLLM/OpenRouter-style retry-budget routing for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• cache-hit-sticky-warm-pool: consistent-hashes a long prompt prefix (min NEXUS_CACHE_HIT_STICKY_MIN_CHARS, default 64) onto one domain-eligible model with healthy ring failover so provider prompt caches stay warm across GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic
• embedding-cache-key-namespace: consistent-hashes {NEXUS_EMBEDDING_CACHE_NAMESPACE_PREFIX}:{tenant} (default prefix embed) onto one domain-eligible model with healthy ring failover so embedding/cache keys stay isolated across tenants for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic
• carbon-aware-preference: prefers lower carbon-intensity providers under NEXUS_CARBON_AWARE_MAX_INTENSITY (default 400) for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• tenant-concurrency-lease: prefers providers with remaining per-tenant in-flight headroom (NEXUS_TENANT_CONCURRENCY_LEASE, default 8) using InflightStats keyed by tenant/session — LiteLLM/Portkey-style tenant fairness for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-error-budget-shed: prefers healthy domain-eligible providers whose rolling SuccessStats error rate stays under NEXUS_PROVIDER_ERROR_BUDGET_RATE (default 0.15); when every provider is over budget it falls back to lowest error rate, then quality, for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic
• region-latency-p99-shed: prefers region-matching domain-eligible providers whose rolling LatencyStats p99 stays under NEXUS_REGION_LATENCY_P99_MS (default 3000); when every regional provider is hot it falls back to lowest p99, then quality — LiteLLM/OpenRouter-style regional tail shedding for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• sticky-canary-cost: pins tenants via consistent hashing on metadata.tenant_id (with user/session fallbacks) and blends a deterministic request_id explore slice (NEXUS_STICKY_CANARY_COST_PERCENT, default 10) toward cheaper healthy models while keeping sticky affinity off-slice — LiteLLM/Portkey-style sticky cost canaries for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• queue-depth-fairness: prefers domain-eligible providers whose live InflightStats queue depth stays under NEXUS_QUEUE_DEPTH_SOFT_CAP (default 4); when every provider is saturated it falls back to lowest depth, then quality — LiteLLM/Portkey-style queue fairness for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-quota-fair-share: tracks the last NEXUS_PROVIDER_QUOTA_LOOKBACK selections (default 100) and prefers eligible providers below equal request share, shedding over-share providers while preserving quality/cost tie-breaks for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-spend-telemetry: prefers lower-spend providers once soft USD spend telemetry exceeds NEXUS_PROVIDER_SPEND_SOFT_USD (default 10) for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 fleets
• semantic-cache-ttl-affinity: pins cacheable requests to providers with warm TTL under NEXUS_SEMANTIC_CACHE_TTL_SECONDS (default 300) for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• circuit-breaker-half-open-probe: prefers healthy closed providers and allows only limited concurrent probes into half-open/recovering providers (NEXUS_CIRCUIT_HALF_OPEN_PROBE_BUDGET, default 2) — LiteLLM/Portkey-style half-open probe budgeting for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• latency-slo-shed: sheds providers whose rolling p95 exceeds NEXUS_LATENCY_SLO_MS (default 2000) when faster alternatives exist; prefers highest quality among under-SLO candidates and falls back to lowest latency when every provider is hot — LiteLLM/OpenRouter-style latency SLO shedding for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• adaptive-timeout: selects the highest-quality model whose risk-adjusted provider p95 fits an adaptive timeout budget derived from request urgency, recent latency, and success/error signals; prefers faster models under realtime pressure and allows slower higher-quality models when comfortable
• complexity-tier: treats the classifier complexity score as a required quality target and picks the cheapest model meeting it — a catalog-adaptive quality-for-cost escalation ladder with no thresholds to tune (falls back to the top-quality model when the target is unreachable)
• round-robin: load-balances across every provider offering a domain-eligible model (routing each to that provider's best eligible model), spreading rate-limit pressure instead of converging on one provider; balanced by a stable request_id hash so routing stays deterministic and replayable
• cascade: routes the primary attempt to the cheapest domain-eligible model and orders the fallback chain by ascending cost, so a failure escalates one price/capability rung at a time instead of jumping to the top-quality model — minimizing expected spend on the common first-attempt-succeeds path with no thresholds to tune
• epsilon-greedy: with probability NEXUS_EPSILON explores by picking uniformly among domain-eligible models (stable second hash of request_id); otherwise exploits the highest-quality eligible model — a replayable bandit policy so under-prioritized catalog entries still get live traffic
• adaptive-exploration: like epsilon-greedy, but epsilon decays from NEXUS_ADAPTIVE_EXPLORATION_BASE (default 0.2) toward NEXUS_ADAPTIVE_EXPLORATION_MIN (default 0.02) as SuccessStats total successes grow — explore more while cold, exploit more as GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic proves out
• geo-region: prefers models whose supported_regions include the request region (GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 catalog priors)
• region-tier-affinity: prefers models matching both request geo region and complexity-mapped tier (frontier/mid/economy), then tier, then region, then quality — no extra NEXUS_* knobs
• soft-family-budget: deprioritizes provider families whose rolling observed spend exceeds a soft budget (NEXUS_SOFT_FAMILY_BUDGET_USD, window via NEXUS_SOFT_FAMILY_BUDGET_WINDOW_SECONDS); prefers highest-quality models from under-budget families and falls back to the cheapest other family when every family is hot — OpenRouter/LiteLLM-style family spend steering for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• sticky-region-failover: pins session_id to a model inside the first healthy preferred region (request region first, then NEXUS_STICKY_REGION_FAILOVER_PREFERENCES), failovers to the next region when the preferred pool is unhealthy, and keeps sticky affinity when healthy — geo-residency plus session stickiness for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• region-failover-hysteresis: like sticky-region-failover but waits for NEXUS_REGION_FAILOVER_HYSTERESIS_SUCCESSES (default 3) consecutive preferred-region successes before flapping back after a failover for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• token-budget: selects the highest-quality domain-eligible model whose context_window fits prompt_tokens_estimate + max_tokens within the request token_budget; falls back to the largest-context model when nothing fits
• slo-aware: selects the highest-quality domain-eligible model whose provider rolling success rate meets NEXUS_AVAILABILITY_SLO; falls back to the highest success-rate model when nothing meets the SLO
• semantic-cache: on metadata.cache_hit, prefers the cheapest domain-eligible model; on miss, falls through to cost-optimal under the quality floor
• least-busy: selects the highest-quality domain-eligible model on the provider with the lowest current in-flight load; load ties prefer higher quality, then lower estimated cost
• prompt-prefix-cache: hashes long shared system-prompt prefixes to sticky provider/model buckets, improving OpenRouter/LiteLLM-style KV-cache affinity for GPT-5.5, Claude Sonnet 4.6, Gemini 2.5, and Kimi K2; short prefixes fall back to cost-optimal
• concurrency-cap: skips providers whose live in-flight count is at or above NEXUS_CONCURRENCY_CAP, then selects the highest-quality remaining model for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic
• prompt-prefix-cache: hashes long shared system-prompt prefixes to sticky provider/model buckets, improving OpenRouter/LiteLLM-style KV-cache affinity for GPT-5.5, Claude Sonnet 4.6, Gemini 3.x, and Kimi K2; short prefixes fall back to cost-optimal
• soft-rate-limit: prefers healthy providers with fewer recent 429/rate-limit observations, so GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic backs away from quota pressure before hard circuit breakers trip
• cost-latency-pareto: keeps non-dominated cost/latency candidates (Pareto front on estimated spend and rolling provider p95), then breaks ties by quality — LiteLLM/Portkey-style multi-objective routing across GPT-5.5, Claude Sonnet 4.6, Gemini 3.x, and Kimi K2
• token-bucket-burst: maintains per-provider token buckets (NEXUS_TOKEN_BUCKET_CAPACITY, NEXUS_TOKEN_BUCKET_REFILL_PER_SEC) and prefers providers with burst quota for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic; when every bucket is empty it falls back to the highest remaining fraction, then cost
• model-tier-rate-limit: infers frontier/mid/economy tiers from model names and applies tier-specific soft RPM ceilings per provider (NEXUS_TIER_FRONTIER_RPM, NEXUS_TIER_MID_RPM, NEXUS_TIER_ECONOMY_RPM); prefers providers under their tier limit for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic and falls back to the least-saturated provider
• failover-priority: walks an explicit ordered model preference list and picks the first healthy provider (LiteLLM-style ordered failover)
• provider-health-score-blend: blends circuit availability, rolling success rate, inverse p95 latency, model quality, and inverse estimated cost; open circuits are skipped whenever a healthy provider exists (NEXUS_HEALTH_BLEND_*)
• health-cost-latency: ternary blend of rolling provider success rate, inverse estimated cost, and inverse p95 latency for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2 traffic (NEXUS_HCL_*)
• provider-hourly-cost-ceiling: skips providers whose rolling hourly estimated spend exceeds NEXUS_PROVIDER_HOURLY_COST_CEILING_USD (default 5.0), preferring highest quality under ceiling — distinct from provider-family-cost-ceiling for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• quality-weighted-sticky: sticky-session hashing with hash-ring bucket weights proportional to quality_score (higher quality gets larger sticky share) — distinct from uniform sticky-session and sticky-tenant-hash for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• token-rpm-ceiling, provider-circuit-probe, carbon-latency-blend: tracks estimated prompt tokens per provider over a rolling 60-second window and sheds requests that would exceed NEXUS_TOKEN_RPM_CEILING (default 100000) to the next eligible provider for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• adaptive-concurrency-cap: scales per-provider in-flight caps by rolling success rate and inverse p95 latency (NEXUS_ADAPTIVE_CONCURRENCY_BASE_CAP, NEXUS_ADAPTIVE_CONCURRENCY_MIN_CAP, NEXUS_ADAPTIVE_CONCURRENCY_LATENCY_MS) so unhealthy backends shed load while quality-first routing continues for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-token-fair-share: fair-share prompt-token budget per provider in a rolling 60-second window (NEXUS_PROVIDER_TOKEN_FAIR_SHARE_CEILING, default 100000) with round-robin weighted by remaining quota for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• tenant-budget-cascade: tracks per-tenant rolling spend, keeps quality-first choices while projected spend fits NEXUS_TENANT_BUDGET_CASCADE_SOFT, sheds to cheaper providers up to NEXUS_TENANT_BUDGET_CASCADE_HARD, then fails closed with a clear rationale for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-error-budget-reset: temporarily sheds providers above NEXUS_PROVIDER_ERROR_BUDGET_RESET_FRACTION and automatically restores them after NEXUS_PROVIDER_ERROR_BUDGET_RESET_SECONDS, distinct from cumulative provider-error-budget-shed, for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• sticky-region-warmup: sends each new session's first NEXUS_STICKY_REGION_WARMUP_REQUESTS requests to a warmup region, then pins the session to its requested or hash-selected region to prevent cold-start flaps for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• tenant-quota-burst: keeps tenant traffic quality-first below NEXUS_TENANT_QUOTA_BURST_SOFT, admits a bounded rolling burst on cheaper fallbacks up to NEXUS_TENANT_QUOTA_BURST_HARD, then sheds before dispatch for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-tail-latency-hedge: keeps the quality leader until its rolling provider p95 exceeds NEXUS_PROVIDER_TAIL_LATENCY_HEDGE_MS, then hedges to the fastest observed healthy provider alternative — distinct from p50 region hedging for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• sticky-session-migrate: preserves a session's model pin while its provider meets NEXUS_STICKY_SESSION_MIGRATE_SUCCESS_THRESHOLD, then performs one controlled migration to a healthier provider and keeps the new pin for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-cold-start-bias: prefers the least-observed healthy provider until each reaches NEXUS_PROVIDER_COLD_START_TARGET selections inside NEXUS_PROVIDER_COLD_START_LOOKBACK, then returns to quality-first routing — LiteLLM/Portkey-style exploration-gap coverage for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• tenant-fair-queue: computes per-tenant request-count deficits over NEXUS_TENANT_FAIR_QUEUE_LOOKBACK, preserving highest-quality healthy routes for at/below-share tenants and moving above-share tenants to a cost-efficient relief lane — distinct from provider queue-depth and provider-quota fairness for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• sticky-region-drain: preserves healthy session region pins but evacuates pins listed in NEXUS_STICKY_REGION_DRAIN_REGIONS to the first healthy non-draining preferred region, keeping the alternate pin after the drain — Envoy/service-mesh-style operational drain for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-canary-shadow-split: keeps primary traffic on NEXUS_PROVIDER_CANARY_PRIMARY_PROVIDER while a deterministic tenant/request slice (NEXUS_PROVIDER_CANARY_SHADOW_PERCENT) identifies a healthy different-provider shadow candidate and records comparison counts for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• sticky-model-pin-expire: keeps each session on a healthy model for NEXUS_STICKY_MODEL_PIN_TTL_SECONDS, then expires the process-local pin and re-evaluates current provider health and quality before creating a fresh pin for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• tenant-priority-lanes: maps configured or per-request tenants into high, normal, and low lanes, preserving quality when capacity is available while using fastest-healthy or cost-efficient relief routes under provider-health or lane-quota pressure for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• request-class-qos: maps metadata.request_class or metadata.qos_class to interactive (lowest observed healthy latency + quality), batch (quality-first mid-cost), or bulk (cheapest healthy) policies for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• deadline-aware-pick: reads metadata.deadline_ms or metadata.remaining_ms and picks the fastest healthy model when remaining budget is below NEXUS_DEADLINE_AWARE_THRESHOLD_MS, otherwise staying quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-success-floor: skips providers whose rolling success rate is below NEXUS_PROVIDER_SUCCESS_FLOOR, preferring highest quality above the floor and emergency-retaining the highest-success provider when every candidate is below for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-warmup-weight: blends quality with a per-request metadata.provider_warmup score (defaulting to 0.5 when omitted) using NEXUS_PROVIDER_WARMUP_BLEND for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• tenant-soft-isolation: tracks each tenant's rolling request rate and demotes tenants above NEXUS_TENANT_SOFT_ISOLATION_RPM to the cheapest healthy domain-compatible model instead of frontier routing, without rejecting the request, for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• model-capability-gate: filters candidates to those whose capability set covers every capability in metadata.required_capabilities (for example vision, tools, long_context), using a per-request metadata.model_capabilities override or the built-in known-model map, and emergency-retains the highest-quality healthy candidate when none match for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• structured-output-prefer: when metadata.requires_json or metadata.structured_output is truthy, prefers candidates advertising json/structured capability (from metadata.structured_models, metadata.model_capabilities / the known-model map with json/structured/json_mode, or a gpt-5/claude/gemini/kimi name heuristic), ranking by structured support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• long-context-prefer: when metadata.min_context_tokens is a positive int (or metadata.long_context is truthy with default threshold 100000), prefers candidates whose context window meets the threshold (from metadata.model_context_windows, the known-model map / catalog, or a gpt-5/claude/gemini/kimi name heuristic), ranking by context then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• streaming-prefer: when metadata.stream or metadata.requires_streaming is truthy, prefers candidates advertising streaming capability (from metadata.streaming_models, metadata.model_capabilities / the known-model map, or a gpt-5/claude/gemini/kimi name heuristic), ranking by streaming support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• provider-exclusion: filters metadata.excluded_providers and metadata.excluded_models (comma-separated string or list) out of the healthy domain-eligible pool, then quality-first selects; emergency-retains the highest-quality eligible model when every candidate is excluded for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• prompt-injection-risk-shed: reads metadata.prompt_injection_risk and demotes requests at or above NEXUS_PROMPT_INJECTION_RISK_THRESHOLD to the cheapest healthy domain-compatible model instead of frontier routing, without rejecting the request, for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• thinking-model-prefer: when signals.complexity_score or metadata.complexity_score is at or above NEXUS_THINKING_COMPLEXITY_THRESHOLD, prefers thinking/reasoning models (from metadata.thinking_models or names containing o1/o3/reasoning/thinking/sonnet/opus), ranking by thinking membership then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• tool-calling-prefer: when metadata.requires_tools is truthy or metadata.tools is non-empty, prefers candidates advertising tool/function calling (from metadata.tool_capable_models, metadata.model_capabilities / the known-model map, or a gpt-5/claude/gemini/kimi name heuristic), ranking by tool support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• multimodal-input-prefer: when metadata.has_images or metadata.has_audio is truthy, prefers candidates advertising vision/multimodal capability (from metadata.multimodal_models, metadata.model_capabilities / the known-model map, or a gpt-5/claude/gemini/vision name heuristic), ranking by multimodal support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• audio-input-prefer: when metadata.requires_audio, metadata.audio_input, or metadata.audio is truthy, prefers candidates advertising audio capability (from metadata.audio_models, metadata.model_capabilities / the known-model map, or an audio/realtime/gpt-4o-audio/gemini name heuristic), ranking by audio support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• web-search-prefer: when metadata.requires_web_search, metadata.web_search, or metadata.online is truthy, prefers candidates advertising web_search capability (from metadata.web_search_models, metadata.model_capabilities / the known-model map, or a search/online/browse name heuristic), ranking by web-search support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• computer-use-prefer: when metadata.requires_computer_use, metadata.computer_use, or metadata.cua is truthy, prefers candidates advertising computer_use capability (from metadata.computer_use_models, metadata.model_capabilities / the known-model map, or a computer/cua/operator name heuristic), ranking by computer-use support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• image-gen-prefer: when metadata.requires_image_gen, metadata.image_gen, or metadata.image_generation is truthy, prefers candidates advertising image_gen capability (from metadata.image_gen_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• code-execution-prefer: when metadata.requires_code_execution, metadata.code_execution, or metadata.code_interpreter is truthy, prefers candidates advertising code_execution capability (from metadata.code_execution_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• video-input-prefer: when metadata.requires_video, metadata.video_input, or metadata.has_video is truthy, prefers candidates advertising video capability (from metadata.video_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• pdf-input-prefer: when metadata.requires_pdf / metadata.pdf_input / metadata.has_pdf is truthy, prefers candidates advertising pdf capability (from metadata.pdf_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• batch-api-prefer: when metadata.requires_batch / metadata.batch_api / metadata.use_batch is truthy, prefers candidates advertising batch capability (from metadata.batch_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• reasoning-effort-prefer: when metadata.requires_reasoning_effort / metadata.reasoning_effort / metadata.effort_control is truthy, prefers candidates advertising reasoning_effort capability (from metadata.reasoning_effort_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• transcription-prefer: when metadata.requires_transcription / metadata.transcription / metadata.speech_to_text / metadata.stt is truthy, prefers candidates advertising transcription capability (from metadata.transcription_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• moderation-prefer: when metadata.requires_moderation / metadata.moderation / metadata.content_moderation is truthy, prefers candidates advertising moderation capability (from metadata.moderation_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• predicted-outputs-prefer: when metadata.requires_predicted_outputs / metadata.predicted_outputs / metadata.prediction is truthy, prefers candidates advertising predicted_outputs capability (from metadata.predicted_outputs_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• prompt-caching-prefer: prefers prompt_caching-capable models when signaled via metadata (requires_prompt_caching, prompt_caching, cache_prompt); quality-first otherwise. GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2.
• fine-tune-prefer: capability-prefer routing for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2.
• embeddings-prefer: capability-prefer routing for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2.
• responses-api-prefer — prefer responses_api-capable models when requested
• file-search-prefer — prefer file_search-capable models when requested
• vector-store-prefer — prefer vector_store-capable models when requested
• realtime-api-prefer: when metadata.requires_realtime / metadata.realtime_api / metadata.websocket_streaming is truthy, prefers candidates advertising realtime_api capability (from metadata.realtime_api_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• mcp-prefer: when metadata.requires_mcp / metadata.mcp_servers / metadata.model_context_protocol is truthy, prefers candidates advertising mcp capability (from metadata.mcp_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• logprobs-prefer: when metadata.requires_logprobs / metadata.logprobs / metadata.top_logprobs is truthy, prefers candidates advertising logprobs capability (from metadata.logprobs_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• parallel-tool-prefer: when metadata.requires_parallel_tool_calls / metadata.parallel_tool_calls / metadata.parallel_tools is truthy, prefers candidates advertising parallel_tool_calls capability (from metadata.parallel_tool_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• deep-research-prefer: when metadata.requires_deep_research / metadata.deep_research / metadata.o_deep_research is truthy, prefers candidates advertising deep_research capability (from metadata.deep_research_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• agents-api-prefer: when metadata.requires_agents_api / metadata.agents_api / metadata.openai_agents is truthy, prefers candidates advertising agents_api capability (from metadata.agents_api_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• webhook-prefer: when metadata.requires_webhook / metadata.webhook_callbacks / metadata.async_webhook is truthy, prefers candidates advertising webhook capability (from metadata.webhook_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• memory-tool-prefer: when metadata.requires_memory_tool / metadata.memory_tool / metadata.agent_memory is truthy, prefers candidates advertising memory_tool capability (from metadata.memory_tool_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• background-mode-prefer: when metadata.requires_background_mode / metadata.background_mode / metadata.background_agent is truthy, prefers candidates advertising background_mode capability (from metadata.background_mode_models, metadata.model_capabilities / the known-model map, or a name heuristic), ranking by support then quality then cost; otherwise quality-first for GPT-5.5 / Claude Sonnet 4.6 / Gemini 3.x / Kimi K2
• ab: deterministic request-id buckets across two model arms

Documentation

Document	Description
Architecture	System design and component overview
Configuration	All configuration options
Epsilon-greedy guide	Explore/exploit routing walkthrough
Adaptive-exploration guide	Decaying epsilon explore/exploit walkthrough
Token-budget guide	Context-window-aware quality routing
Geo-region guide	Region/residency-aware model selection
Region-tier-affinity guide	Combined geo-region and complexity-tier affinity routing
Soft-family-budget guide	Rolling soft spend budgets per provider family
Sticky-region-failover guide	Session stickiness with ordered region failover
Region-failover-hysteresis guide	Region failover with recovery hysteresis
Sticky-tenant-hash guide	Per-tenant consistent hashing with healthy failover
Embedding-cache-key-namespace guide	Tenant-isolated embedding/cache sticky namespace routing
Semantic-cache-ttl-affinity guide	Warm semantic-cache TTL sticky routing
Tenant-concurrency-lease guide	Per-tenant in-flight concurrency lease routing
Provider-error-budget-shed guide	Rolling provider error-budget shedding
Region-latency-p99-shed guide	Regional p99 tail-latency shedding
Sticky-canary-cost guide	Sticky tenant affinity with cost canary blend
Queue-depth-fairness guide	Soft queue-depth fairness across providers
Provider-quota-fair-share guide	Rolling equal-share provider quota routing
Provider-token-fair-share guide	Rolling token fair-share routing weighted by remaining quota
Tenant-budget-cascade guide	Per-tenant rolling spend cascade with a hard fail-closed ceiling
Circuit-breaker-half-open-probe guide	Half-open recovery probe budget routing
SLO-aware guide	Availability-SLO quality routing
Adaptive-timeout guide	Timeout-adaptive quality routing
Adaptive-timeout-hedge guide	Relative p95 hedge from a quality-first provider choice
Token-bucket-tenant guide	Per-tenant request budget with cheapest-model shedding
Region-carbon-blend guide	Carbon intensity blended with latency scoring
Provider-weight-decay guide	Exponential provider weight decay with slow recovery
Retry-after-respect guide	Honor provider Retry-After cooldowns
Semantic-cache guide	Cache-hit cheapest / miss cost-optimal routing
Least-busy guide	Live in-flight load-aware routing
Prompt-prefix-cache guide	Sticky system-prompt prefix affinity for provider KV-cache hits
Concurrency-cap guide	Per-provider in-flight saturation cap routing
Adaptive-concurrency-cap guide	Health-derived dynamic in-flight cap routing
Soft-rate-limit guide	Soft 429/rate-limit pressure avoidance
Cost/latency Pareto guide	Multi-objective non-dominated cost + latency routing
Token-bucket-burst guide	Bursty per-provider token-bucket quota routing
Model-tier-rate-limit guide	Tier-specific soft RPM routing by model name
Failover-priority guide	Ordered healthy-provider failover
Provider-health score blend guide	LiteLLM/Portkey-style health-aware blended routing
Health/cost/latency guide	Ternary health, cost, and latency blend routing
Provider-family cost-ceiling guide	Per-provider-family spend ceilings for multi-provider budgets
Canary-tier-blend guide	Canary rollout with complexity-tier affinity
Latency-SLO-shed guide	Latency SLO shedding with under-SLO quality preference
Shadow-traffic-mirror guide	Cost-optimal primary with shadow mirror telemetry
Canary-cost-blend guide	Cost exploration with next-cheaper healthy tier sampling
Token-cost-anomaly-shed guide	Rolling cost/1k anomaly shedding with quality fallback
Quickstart	Local setup and first request
Safety	Guardrails, fallback, and PII controls
Contributing	Development workflow and PR process
Security	Vulnerability reporting policy
Changelog	Version history

License

Apache-2.0 © Francis1998

Last updated: 2026-08-01