What is an LLM gateway?
An LLM gateway is one endpoint in front of many model providers: it holds the keys, routes each request, falls back when a provider fails, and logs and caps what is spent. What to expect from one and where it stops.
An LLM gateway (also called an AI gateway) is a service that sits between the applications that make model requests and the providers that answer them. Applications talk to the gateway as if it were a provider; the gateway decides which real provider and model handles each request, forwards it with the right credential, and sends the answer back. Nothing about the application has to change when the provider does.
The pattern is older than language models. Every API gateway does the same job for ordinary web services. What makes the LLM variety its own category is what sits behind it: several providers with different request formats, prices that vary by a large multiple between models, rate limits that bite without warning, and outages that arrive in the middle of a working day.
What a gateway does
One endpoint, one key. Applications are configured once, usually with the request shape of the most widely imitated provider, the OpenAI-compatible API. The gateway translates to whatever each provider actually expects. A team's real provider keys live in the gateway, not in every application and laptop.
Routing. Which model answers can be a fixed choice, a rule on the request, or a decision made per request on its difficulty, cost or latency. Model routing is the part of a gateway that most changes what things cost.
Fallbacks and retries. When a provider times out, rate-limits or refuses, the gateway tries the next option in a list before returning an error. A well-built one distinguishes a broken key from a broken model, so one revoked credential does not take a provider out of rotation.
Logging. Every request is recorded: who made it, which model answered, how many tokens went in and out, how long it took. This is the raw material for everything else.
Budgets and keys. Per-user or per-team keys ("virtual keys") that can be revoked without rotating the provider's, and allowances that stop or redirect spend when a budget is reached.
Caching of identical requests, and sometimes of tool results, where that is safe.
What to look for
- Which wire formats it serves. Chat completions is table stakes; the Responses shape and the Anthropic Messages shape matter if you use the Codex CLI, Claude Code or the Anthropic SDKs.
- Whether routing can be set per team and per person as well as for the whole company, and whether a cheaper model can answer when a budget runs out rather than every request failing.
- What the logs contain. Tokens and models are useful; prompts stored by default are a privacy decision somebody should have made on purpose.
- Where it runs and who can read it. A gateway sees every prompt in the company.
- Whether you can still name a specific model when you need one, and whether an administrator can stop people doing so.
Where a gateway stops
A gateway moves requests. It knows nothing about your company, your customers or your systems, and it cannot make a model's answer better, only cheaper and more reliable. Memory, retrieval and tools are a separate layer, and most gateways leave them to the application. The extra hop also costs a little latency, and a gateway's own extensions, the fields it adds to requests and responses, are a dependency of their own.
Hosted marketplaces such as OpenRouter, open-source proxies such as LiteLLM and hosted gateways such as Portkey are the common shapes; the comparisons with OpenRouter, LiteLLM and Portkey go through the differences.
Where Memtro fits
Memtro is a gateway in this sense, with the OpenAI chat completions, Responses and Anthropic Messages shapes on one endpoint, routing by difficulty, fallbacks, keys per person or organisation, a request log and token allowances per team. The difference is what rides along with each request: the organisation's memory and its connector tools. The gateway page sets it out.