// GLOSSARY -- AGENTIC AI

What is LLM Router?

1 min read Updated

An LLM router is a system that intelligently directs AI requests to different models based on task complexity, cost, latency requirements, or domain — optimizing the quality-cost tradeoff across a portfolio of models.

WHY IT MATTERS

Not every request needs GPT-4. An LLM router classifies incoming requests and routes simple tasks to cheaper, faster models (Haiku, GPT-3.5) while sending complex tasks to frontier models (Opus, GPT-4). This can reduce costs by 50-80% with minimal quality impact.

Routing strategies include: classifier-based (a small model predicts difficulty), cascade (try cheap model first, escalate if quality is low), and rule-based (route by task type or keyword).

For agent systems, routing is particularly valuable. Tool calls, simple classifications, and formatting tasks don't need expensive models. Planning, complex reasoning, and financial decisions do.

FREQUENTLY ASKED QUESTIONS

How much can routing save?
Typically 50-80% cost reduction with <5% quality loss. The exact savings depend on your task distribution — more simple tasks means more savings from routing.
What's the best routing strategy?
Start with rule-based routing (route by task type). Graduate to classifier-based when you have enough data. Cascade routing works well when quality is easy to evaluate automatically.
Does routing add latency?
The routing decision itself adds milliseconds. For cascade routing, failed attempts add latency. For direct routing, the net effect is often negative (faster models for simple tasks).

FURTHER READING

Enforce policies on every tool call

Intercept is the open-source MCP proxy that enforces YAML policies on AI agent tool calls. No code changes needed.

npx -y @policylayer/intercept
github.com/policylayer/intercept →
// GET IN TOUCH

Have a question or want to learn more? Send us a message.

Message sent.

We'll get back to you soon.