OmniRoute Tutorial: Building a local AI API gateway and automatic switching of multiple models

OmniRoute is a locally running AI API gateway that puts different model suppliers, subscription accounts and free quotas behind a unified interface. Clients such as Codex, Claude Code, Cursor, Cline, and OpenCode only need to connect to an OpenAI compatible address, and then OmniRoute selects the model based on available quota, cost, latency, and health status.

It is suitable for developers who use multiple model services at the same time, often encounter current throttling, or want to uniformly view the call volume. It’s important to note that the gateway does not generate free credits out of thin air: account registration, API pricing, rate limits, and acceptable usage are still determined by each upstream provider.

Quick Answer

Install and start globally:

1
2

npm install -g omniroute
omniroute

Default address:

1
2

Dashboard: http://localhost:20128
API:       http://localhost:20128/v1

Go to the Providers page of the Dashboard to connect to at least one model provider, and then go to the Endpoints page to copy the local API Key. AI client uses:

1
2
3

Base URL: http://localhost:20128/v1
API Key:  Dashboard 中生成的 Key
Model:    auto

Verification model list:

1
2

curl http://localhost:20128/v1/models \
  -H "Authorization: Bearer YOUR_KEY"

If a connected model is returned, the gateway is basically available. Before officially connecting to the coding agent, test ordinary dialogue, streaming output, tool invocation, and long context respectively to avoid judging compatibility based on the model list alone.

How OmniRoute works

The client no longer directly connects to each model API, but sends the request to the local OmniRoute. The gateway reads the model name and routing rules, selects the currently available Provider, and converts the response into a protocol that the client can understand.

1
2
3
4
5
6
7
8
9

Codex / Claude Code / Cursor
              |
              v
 http://localhost:20128/v1
              |
              v
      OmniRoute 路由与回退
       /       |        \
   Provider A  B         C

This architecture brings three direct effects:

1. The client only maintains a Base URL and access key;
2. When a Provider is limited or fails, you can switch to the candidate model;
3. Call volume, cost, delay and errors are concentrated on the same control plane for observation.

The trade-off is that OmniRoute becomes a critical component in the request path. When it is down, misconfigured, or the data directory is corrupted, all clients forwarded by it will be affected, so production environments require persistence, backups, access control, and clear workarounds.

Environmental requirements and installation

Officials currently require Node.js 22 or 24 LTS, and recommend Node.js 24 LTS. Check the version first:

1
2

node --version
npm --version

Install using npm:

1

npm install -g omniroute

You can enter the boot process when running for the first time:

1

omniroute setup

Start the gateway and Dashboard:

1

omniroute

Interactive terminal chat:

1

omniroute chat

Run diagnostics when encountering provider, port or native dependency issues:

1

omniroute doctor

If the machine does not have an adapted better-sqlite3 precompiled file, the project will try to use other SQLite implementations. When the installation is abnormal, you should read the complete log first. Do not directly close all installation scripts or reinstall repeatedly with administrator rights.

Run using Docker

Officially provides multi-architecture Docker images:

1
2
3
4
5
6
7

docker run -d \
  --name omniroute \
  --restart unless-stopped \
  --stop-timeout 40 \
  -p 20128:20128 \
  -v omniroute-data:/app/data \
  diegosouzapw/omniroute:latest

Check status and logs:

1
2

docker ps --filter name=omniroute
docker logs -f omniroute

omniroute-data Save the configuration, database and running status, and do not delete them randomly when upgrading or rebuilding the container. Production environments should also replace latest with a validated, explicit version label and back up the volume before upgrading.

The -p 20128:20128 above may publish the port to all network interfaces of the host. When used only on this machine, it can be restricted to the loopback address:

1
2
3
4
5
6
7

docker run -d \
  --name omniroute \
  --restart unless-stopped \
  --stop-timeout 40 \
  -p 127.0.0.1:20128:20128 \
  -v omniroute-data:/app/data \
  diegosouzapw/omniroute:latest

When remote access is required, HTTPS, strong authentication, IP restrictions, or private networks should be used, and the Dashboard and API should not be exposed to the Internet.

Connect model supplier

Open after startup:

1

http://localhost:20128

Enter the Providers page and add a Provider based on the account or API Key you actually have. It is recommended to operate in the following order:

1. Connect to a low-risk test account first;
2. Confirm that the model directory and single conversation are available;
3. Set budget or quota limit;
4. Add a second Provider verification fallback;
5. Finally, connect to the daily coding client.

Do not expose OAuth Tokens, API Keys, or Dashboard access keys in screenshots, logs, or feedback. The warehouse description credentials will be encrypted and saved locally, but it may still pose a risk when the machine is compromised, the master key is leaked, or the reading process is maliciously extended.

Use auto automatic routing

The simplest configuration is to set the client model to:

1

auto

OmniRoute also provides automatic model names for different targets:

Automatic routing does not guarantee that different models will behave exactly the same. Tool call formats, context windows, reasoning capabilities, and output styles may vary. Key tasks should fix the model or limit the candidate set to avoid silently switching to models with obviously different capabilities in a long task.

Customize fallback chain and routing strategy

OmniRoute calls a set of model fallback targets a combo. Goals can be selected by priority, weight, cost, remaining balance, latency, or recent success status.

Common strategies include:

• priority: Use in a fixed order, go to the next one after failure;
• round-robin: Poll among targets;
• cost-optimized: Favor lower-priced available models;
• headroom: Favor connections with more remaining balance;
• context-optimized: Select model according to current context size;
• lkgp: Keep the path that has been successfully verified recently.

When configuring the fallback chain, don’t just compare model names. Also consider input and output prices, contextual constraints, tool calls, image capabilities, data regions, and vendor terms. For sessions where model consistency must be maintained, an appropriate stickiness strategy should be enabled or the provider should be pinned directly.

Access OpenAI compatible client

Tools that can customize the OpenAI Base URL can usually be used:

1

http://localhost:20128/v1

It is recommended to pass the access key through the request header:

1

Authorization: Bearer YOUR_KEY

Clients that cannot add custom headers can use compatible aliases with Token, but the URL will contain the key, making it easier to access browser history, agent logs, or screenshots. Use this method only when Header authentication is absolutely impossible, and rotate the key regularly.

When authenticating the chat interface, you can send a minimal request:

1
2
3
4

curl http://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Reply with OK"}]}'

The aliasing and quoting behavior of curl may differ in Windows PowerShell, it is recommended to use curl.exe or to construct the request according to the current PowerShell version.

MCP access and permission risks

OmniRoute not only forwards model requests, but also provides MCP to let Agent manage providers, routing, combo, caching, compression and other gateway functions.

stdio mode:

1

omniroute --mcp

HTTP MCP address:

1

http://localhost:20128/api/mcp/stream

Claude Code example:

1
2
3

claude mcp add-server omniroute \
  --type http \
  --url http://localhost:20128/api/mcp/stream

MCP permissions are more sensitive than normal model calls because the agent may modify routing or connection configurations. Minimum scope, independent access token, and audit logs should be used before accessing; operations such as deleting a provider, rotating keys, or adjusting team quotas should retain manual confirmation.

How should Token compression be evaluated?

The warehouse provides multi-stage compression and processing pipelines such as RTK and Caveman, and the officially demonstrated savings ratio ranges widely. The actual effect depends on tool output, duplicate content, contextual structure and compression level. The ratio in the README cannot be directly regarded as a guarantee for each project.

It is recommended to use fixed tasks for A/B testing:

1. Save the original prompt, tool output and final result;
2. Turn off compression and run it once;
3. Run again using standard or RTK configuration;
4. Compare input token, delay, cost and answer correctness;
5. Run the same set of tests or verification commands on the code modification.

Compression may delete information judged to be of low relevance. Security audits, long log diagnostics, and accurate code reviews shouldn’t just look at token savings, but also check miss rates and preserve paths to view raw output.

Free Credit and Supplier Terms

OmniRoute aggregates free tier, trial quota, and throttling information published by multiple vendors. The number will change with supplier policies, regions, account types and time, so the article does not fixedly quote a certain “total number of free tokens per month”. The current catalog of Dashboard and the upstream official price page should prevail.

There are also three types of resources to distinguish:

1. Long-term free tier;
2. One-time trial quota after registration;
3. Additional credits that require payment or subscription to be unlocked.

You must check the vendor’s terms of service before using subscription accounts, non-standard OAuth flows, or aggregation of multiple accounts. Being technically able to access does not mean that the provider allows individual subscriptions to be shared with teams, automated calls, or to circumvent quota limits.

Remote deployment considerations

When OmniRoute is deployed to a VPS, all client prompts, code context, and model responses go through that host. At least required:

• Use HTTPS to avoid clear text transmission of Token and Prompt;
• Restrict access sources to Dashboard, API and MCP;
• Issuing Keys with different scopes for different users or clients;
• Back up the data directory and encrypt the backup at the same time;
• Set the log retention period to avoid long-term storage of sensitive code;
• Monitor call costs, failure rates, latency and abnormal logins.

Do not simply replace http://localhost:20128 in the local example with the public IP and put it into use. It is recommended to verify through Tailscale, WireGuard or SSH Tunnel before deciding whether to configure a reverse proxy and public domain name.

FAQ

Dashboard cannot be opened

Run diagnostics and check ports:

1

omniroute doctor

Confirm that the process is still running and 20128 is not occupied by other applications. Docker users view container logs and port mappings.

/v1/models returns 401

Confirm that the request uses the local Key generated by Dashboard → Endpoints and contains:

1

Authorization: Bearer YOUR_KEY

Do not mistake the upstream Provider Key for the OmniRoute Endpoint Key.

auto An inappropriate model was selected.

Pin a verified model first to confirm client compatibility, then adjust combo candidates, strategies, budgets, and context requirements. Critical workflows can use auto/coding, but should still restrict models that do not meet tool call or context requirements.

Answer style changes suddenly after routing fallback

Different models have different system command compliance capabilities and tool formats. Enable session stickiness, close candidate model gaps, and preserve necessary task state across switches. Fixed models directly for tasks that require determinism.

Can OmniRoute reduce all AI costs?

Not guaranteed. It can route based on pricing and quota, and reduce some duplication of context, but the gateway itself cannot change the upstream billing rules. Fusion, pipeline, or multi-model reviews may also increase the total number of calls.

What scenarios is OmniRoute suitable for?

OmniRoute is suitable for individuals and teams who maintain multiple model accounts at the same time, need a unified OpenAI compatible entrance, want to automatically roll back when current is limited, or want to centrally observe costs and health status. For simple projects that only use one stable Provider, introducing a complete gateway may increase maintenance complexity.

Before adopting it in the team’s production environment, it is recommended to complete four verifications: client protocol compatibility, provider terms, key and permission isolation, and downgrade path in case of gateway failure.

Summary

OmniRoute uses local http://localhost:20128/v1 to connect multiple model providers to a unified API, and uses auto and combo to implement cost, speed, quota and health-driven routing. npm installation is suitable for local experience, and Docker is suitable for persistent operation; HTTPS, access control, backup and auditing must be completed during remote deployment. Free quota and compression ratio are dynamic indicators and should be evaluated in conjunction with upstream terms and your own benchmarks.

Project address: diegosouzapw/OmniRoute

Official website: omniroute.online