LiteLLM Gateway

Route requests through a LiteLLM proxy to access 100+ LLM providers

The LiteLLM gateway connects ChatJS to a LiteLLM proxy server, giving you unified access to 100+ LLM providers (OpenAI, Anthropic, Google, Azure, AWS Bedrock, and more) through a single endpoint.

Why LiteLLM?

• Unified API - Call any LLM provider using the OpenAI format
• Centralized key management - Store all provider API keys on the proxy, not in the client
• Spend tracking - Monitor usage and costs across all providers in one place
• Load balancing - Distribute requests across multiple deployments of the same model
• Fallbacks - Automatically retry with a different provider if one fails

Setup

1. Deploy a LiteLLM proxy server (quickstart guide) and note the URL. If your proxy requires authentication, also note a master key or virtual key.

2. Set the proxy URL and optional API key in .env.local:

LITELLM_BASE_URL=http://localhost:4000   # Your LiteLLM proxy URL, without /v1
LITELLM_API_KEY=sk-...                   # Optional; only if your proxy requires auth

3. Set the gateway in your config:

const config: ConfigInput = {
  ai: {
    gateway: "litellm",
    workflows: {
      chat: "openai/gpt-4o",
      title: "openai/gpt-4o-mini",
      // ...
    },
    // ...
  },
};

Authentication

Available Models

Models are fetched at runtime from your LiteLLM proxy’s /v1/models endpoint and cached for 1 hour. The available models depend entirely on what you have configured in your proxy’s config.yaml.

Example proxy configuration

model_list:
  - model_name: openai/gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
  - model_name: anthropic/claude-sonnet-4-20250514
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

Image Generation

Image generation support depends on the models available through your LiteLLM proxy. If your proxy is configured with an image generation model (e.g., dall-e-3), enable it in your config:

const config: ConfigInput = {
  ai: {
    gateway: "litellm",
    tools: {
      image: { enabled: true, default: "dall-e-3" },
    },
  },
};

Using the LiteLLM Python SDK

You can also interact with your LiteLLM proxy using the Python SDK:

from litellm import completion

response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    api_base="http://localhost:4000",  # your proxy URL
    api_key="sk-...",                  # optional; your proxy key if required
)
print(response.choices[0].message.content)

Related

• Gateways Overview for the full gateway comparison
• LiteLLM Documentation for proxy setup and configuration
• Custom Gateway if you need more control than this gateway provides