Reference
What is an LLM proxy?
An LLM proxy is a server that sits between your application and language model providers like OpenAI and Anthropic. Every API call passes through it, so it can route the request, track its cost, enforce a spending cap, cache responses, and fail over when a provider is down, all without changing your app code.
If you call more than one model, run agents that can loop and burn budget, or need to see exactly what each request costs, an LLM proxy is the layer that gives you routing, cost attribution, and control in one place. This page explains what it does, when you need one, how it differs from a gateway, and how to set one up in about a minute.
Run a local LLM proxy in under a minute
RelayPlane is a free, MIT-licensed, local-first LLM proxy. It installs with npm and runs on localhost:4100, so it adds no cloud latency and nothing leaves your machine.
- 1
Install the proxy
npm install -g @relayplane/proxy - 2
Start it on localhost:4100
relayplane start - 3
Point Claude Code (or any client) at it
export ANTHROPIC_BASE_URL=http://localhost:4100
What does an LLM proxy do?
A proxy takes over the plumbing between your code and the providers. The six jobs below are the ones that matter most in production.
Routing
Send each request to the right model or provider. Route by task, cost, latency, or a fallback order, without changing your application code.
Cost attribution
Price every request as it passes through and write it to a ledger, so you can see spend per model, per agent, or per run instead of one opaque monthly bill.
Spend caps and kill switch
Enforce a hard budget before a request ever leaves. When the cap is hit, block, downgrade, or warn. A kill switch can halt every routed call at once.
Caching
Return a stored response for an identical prompt instead of paying for it twice. Useful for repeated agent steps, evals, and retries.
Failover and retries
When a provider errors or rate-limits, retry or fall back to another model automatically so a single outage does not stall the whole app.
Observability
One place to see tokens, latency, errors, and cost across every provider, rather than stitching together three separate dashboards.
LLM proxy vs LLM gateway
The two terms are used interchangeably, and many tools are both. The useful distinction is scope: a proxy guards one app or one machine, while a gateway is a shared central service for many teams. Pick the framing that matches your problem.
| Dimension | Proxy (one app) | Gateway (many teams) |
|---|---|---|
| Primary job | Sit in front of one app or machine | Serve many teams and tenants |
| Where it runs | Often local (localhost) or a single service | Central shared service, usually cloud |
| Per-request cost ledger Both can track cost; a local proxy makes it trivial to attribute to one agent or run. | Sometimes | |
| Hard spend cap enforced pre-request | Sometimes | |
| Virtual keys and per-tenant budgets Multi-tenant key management is where a gateway earns its keep. | Not the focus | |
| Added network latency | ~0ms if local | 10 to 50ms round trip |
| Operational overhead | One command, no database | A service to deploy, scale, and secure |
For a deeper breakdown, see LLM proxy vs gateway and the full LLM gateway comparison.
Do you need an LLM proxy?
Yes, if
- You call more than one model or provider and want one interface
- You run autonomous agents that can loop and quietly burn budget
- You need cost broken down per agent, per run, or per feature
- You want a hard spending cap the proxy enforces before the call leaves
- You want failover so one provider outage does not stall everything
Probably not, if
- You make a handful of calls to a single model from one script
- You have no budget or routing concerns and never will
- Your provider SDK already covers everything you need
Using an LLM proxy with Claude (a “Claude proxy”)
A Claude proxy is just an LLM proxy placed in front of Anthropic’s API. It is the most common way to put cost tracking and a hard cap on Claude Code, which can otherwise run for hours and rack up spend with no per-run visibility.
Because Claude Code and the Anthropic SDK both respect the ANTHROPIC_BASE_URL environment variable, pointing them at a local proxy takes one line. Every Claude call then flows through the proxy, which prices it, logs it, and stops it if you have hit your cap.
export ANTHROPIC_BASE_URL=http://localhost:4100See the Claude proxy guide and the ANTHROPIC_BASE_URL walkthrough for the full setup.
How RelayPlane fits
There are good LLM proxies to choose from. LiteLLM and Portkey are strong team gateways. OpenRouter is a hosted routing layer. Cloudflare AI Gateway is a solid cloud option. RelayPlane is the one to reach for when the problem is cost and control on your own machine.
It is a free, MIT-licensed proxy that installs with npm, runs on localhost with near-zero added latency, prices every request into a local ledger, enforces a hard daily cap before a request leaves, and gives you a one-command kill switch. No database, no cloud account, nothing leaves your machine by default. That makes it a natural fit for agent teams and anyone who needs to answer “what did this run cost?” without standing up a service.
A gateway answers “who is allowed to spend?” A local proxy answers “what did this exact run cost, and how do I stop it before it goes too far?”
Questions people ask
- What is an LLM proxy?
- An LLM proxy is a small server that sits between your application and large language model providers such as OpenAI or Anthropic. Every API call passes through it, so it can route the request to the right model, record the cost, enforce a spend cap, cache responses, and fail over when a provider is down, all without changing your app code.
- What is the difference between an LLM proxy and an LLM gateway?
- The terms overlap. In practice a proxy tends to sit in front of one app or machine and focuses on routing, cost, and control, while a gateway is a shared central service built to serve many teams with virtual keys and per-tenant budgets. A local proxy adds near-zero latency; a cloud gateway adds a network round trip. Many gateways are also called proxies and vice versa.
- Do I need an LLM proxy?
- You likely want one if you call more than one model or provider, run autonomous agents that can loop and burn budget, need to see cost per agent or per run, or want a hard spending cap. If you make a handful of calls to a single model from one script, a proxy is optional.
- Can I use an LLM proxy with Claude or Claude Code?
- Yes. This is often called a Claude proxy. Point the Anthropic base URL at the proxy (for RelayPlane, set ANTHROPIC_BASE_URL=http://localhost:4100), and Claude Code routes every call through it. You then get per-request cost, a spend cap, and a kill switch on your Claude usage without touching the tool itself.
- Is an LLM proxy the same as caching?
- No. Caching is one feature a proxy can offer. A proxy also routes requests, tracks cost, enforces budgets, and handles failover. Caching alone stores and replays responses but does none of the routing or control work.
- Does an LLM proxy slow down my requests?
- A cloud proxy or gateway adds a network round trip, typically 10 to 50ms. A local-first proxy that runs on localhost adds effectively no latency, because the request never leaves your machine before it is forwarded to the provider.
Compare the options
Try the free LLM proxy. Nothing leaves your machine.
MIT licensed. No account. Runs on localhost:4100.
npm install -g @relayplane/proxy