DeltaLLM

DeltaLLM is a self-hosted LLM gateway. Point OpenAI-compatible clients at one endpoint, then manage model deployments, routing, scoped API keys, budgets, guardrails, MCP tools, and usage from one control plane.

For hosting consultants, managed service providers, and platform teams, service tiers let you define a model package once and assign it to multiple customer organizations, with per-model limits, pricing, and shared capacity. The gateway and Admin UI are MIT licensed.

SSO, audit logs, and scoped administration are included without a gateway license fee. You cover your infrastructure and upstream model usage.

One Endpoint For Your Apps

from openai import OpenAI

# Before: call OpenAI directly.
client = OpenAI(api_key="sk-...")

# After: send the same OpenAI-compatible requests through DeltaLLM.
client = OpenAI(
    base_url="http://localhost:4002/v1",
    api_key="sk-deltallm-key",
)

Your application keeps its OpenAI request format. DeltaLLM handles provider credentials, routing, policy checks, caching, failover, and spend tracking.

What It Handles

• Unified API - Use one OpenAI-compatible endpoint across OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, Groq, and other providers.
• Scoped API keys - Issue virtual keys with model allowlists, rate limits, budgets, owners, and expiry.
• Customer plans - Assign versioned model packages to organizations through service tiers, with model access, rate limits, and pricing managed together.
• Routing and failover - Route by strategy, retry failed deployments, and separate provider credentials from application code.
• Batch API - Run embeddings and non-streaming chat completions asynchronously through OpenAI-compatible files and batches, even when upstream providers are synchronous.
• MCP gateway - Register external MCP servers and expose approved tools through controlled gateway flows.
• Guardrails - Detect PII and prompt injection before provider calls.
• Spend and usage - Attribute cost by key, team, organization, model, and provider.
• Admin UI - Manage models, credentials, route groups, teams, users, usage, audit logs, and settings in the browser.
• Operations - Export Prometheus metrics, request logs, audit events, cache behavior, and health checks.

Admin UI

The UI is the control plane for model deployments, route groups, API keys, access, usage, guardrails, audit logs, and runtime settings.

Quick Start With Docker

Use Docker Compose for the shortest local evaluation path. It starts DeltaLLM, PostgreSQL, and Redis.

git clone https://github.com/deltawi/deltallm.git
cd deltallm
cp config.example.yaml config.yaml

For a first local request, enable one-time model bootstrap in config.yaml:

general_settings:
  model_deployment_source: db_only
  model_deployment_bootstrap_from_config: true

Generate the required secrets:

python3 -c 'import secrets; print("DELTALLM_MASTER_KEY=sk-" + secrets.token_hex(20) + "A1")'
python3 -c 'import secrets; print("DELTALLM_SALT_KEY=" + secrets.token_hex(32))'

Create .env with the generated values and a provider key:

DELTALLM_MASTER_KEY=sk-your-generated-master-key
DELTALLM_SALT_KEY=your-generated-salt-key
OPENAI_API_KEY=sk-your-openai-key
PLATFORM_BOOTSTRAP_ADMIN_EMAIL=admin@example.com
PLATFORM_BOOTSTRAP_ADMIN_PASSWORD=ChangeMe123!

Start the stack:

docker compose --profile single up -d --build

Wait for the health endpoint to return {"status":"ok"}, then send a request. In this terminal, export the same master key you put in .env: Compose reads that file for its containers, but it does not export variables into your shell.

curl http://localhost:4002/health/liveliness

export DELTALLM_MASTER_KEY='paste-your-generated-master-key'

curl http://localhost:4002/v1/chat/completions \
  -H "Authorization: Bearer $DELTALLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "Hello from DeltaLLM"}
    ]
  }'

Open the Admin UI at http://localhost:4002.

Full Docker setup, environment variables, the multi-instance evaluation profile, and Dockerfile usage are covered in the Docker guide. Compose is not the production high-availability reference.

Deployment Paths

Path	Use it for	Start here
Railway	Managed evaluation with hosted PostgreSQL and Redis	Railway deployment
Docker Compose	Local evaluation, demos, small teams, simple self-hosting	Docker guide
Kubernetes	Multi-instance production, ingress, autoscaling, and managed infrastructure	Kubernetes guide
Local development	Backend, UI, and contributor workflow from the repository	Installation guide

Documentation

• Getting started
• Gateway usage examples
• Configuration reference
• Model deployments
• Features
• Admin UI guide
• API reference
• Deployment guide
• Security model

Contributing

• Report issues
• Request features
• Report a vulnerability privately
• PRs are welcome. Start with the local installation guide.
• Documentation changes follow the documentation contribution guide.

If DeltaLLM solves a problem for you, consider starring the repository. Share your use case in Discussions so we can improve the workflows that matter to you.

License

See LICENSE.