What is an AI Gateway?
An AI gateway is a proxy layer that sits between applications and LLM providers, centralising concerns such as rate limiting, API key management, cost controls, caching, and failover across model APIs.
WHY IT MATTERS
AI gateways grew out of API gateways. As teams adopted multiple model providers, they needed one place to manage keys, enforce spend limits, retry and route across providers, cache responses, and log usage. Products in this category — provider-agnostic proxies offered by API management vendors and LLM-ops platforms — operate on completion traffic: the prompts and responses flowing between an application and a model API.
An MCP-specific gateway is a different layer. It sits between AI clients and MCP servers and operates on tool-call traffic: tools/list and tools/call messages. At this layer the unit of control is not a prompt or a token budget but an action — which tool, with which arguments, by which person. See MCP gateway.
The two are complementary, not competing:
- An AI gateway answers: which models may this app call, at what cost, with whose keys?
- An MCP gateway answers: which tools may this agent invoke, against which servers, under what policy?
Teams running agents typically need both — model-traffic controls for spend and availability, and tool-call controls for what agents are allowed to do. Neither substitutes for the other: an AI gateway never sees tool calls made over MCP transports, and an MCP gateway never sees raw completion traffic.
HOW POLICYLAYER USES THIS
PolicyLayer is a gateway at the tool-call layer, not a model-traffic proxy. AI clients connect to their MCP servers through PolicyLayer, and every tools/call is evaluated against deterministic policy before it executes, with per-person scoped tokens and a full audit trail. It is designed to sit alongside whatever AI gateway a team uses for model traffic — the two operate on different protocols at different layers.