Run an MetriLLM benchmark on a local model. Measures performance (tokens/s, TTFT, memory) and optionally quality (reasoning, math, coding, instruction following, structured output, multilingual). Returns a detailed fitness verdict. Warning: benchmarks take 30s to 5+ minutes depending on model size.
Part of the Metrillm MCP server. Enforce policies on this tool with Intercept, the open-source MCP proxy.
AI agents invoke run_benchmark to trigger processes or run actions in Metrillm. Execute operations can have side effects beyond the immediate call -- triggering builds, sending notifications, or starting workflows. Rate limits and argument validation are essential to prevent runaway execution.
run_benchmark can trigger processes with real-world consequences. An uncontrolled agent might start dozens of builds, send mass notifications, or kick off expensive compute jobs. Intercept enforces rate limits and validates arguments to keep execution within safe bounds.
Execute tools trigger processes. Rate-limit and validate arguments to prevent unintended side effects.
tools:
run_benchmark:
rules:
- action: allow
rate_limit:
max: 10
window: 60
validate:
required_args: trueSee the full Metrillm policy for all 4 tools.
Agents calling execute-class tools like run_benchmark have been implicated in these attack patterns. Read the full case and prevention policy for each:
Other tools in the Execute risk category across the catalogue. The same policy patterns (rate-limit, validate) apply to each.
run_benchmark is one of the high-risk operations in Metrillm. For the full severity-focused view — only the high-risk tools with their recommended policies — see the breakdown for this server, or browse all high-risk tools across every MCP server.
Run an MetriLLM benchmark on a local model. Measures performance (tokens/s, TTFT, memory) and optionally quality (reasoning, math, coding, instruction following, structured output, multilingual). Returns a detailed fitness verdict. Warning: benchmarks take 30s to 5+ minutes depending on model size.. It is categorised as a Execute tool in the Metrillm MCP Server, which means it can trigger actions or run processes. Use rate limits and argument validation.
Add a rule in your Intercept YAML policy under the tools section for run_benchmark. You can allow, deny, rate-limit, or validate arguments. Then run Intercept as a proxy in front of the Metrillm MCP server.
run_benchmark is a Execute tool with high risk. Execute tools should be rate-limited and have argument validation enabled.
Yes. Add a rate_limit block to the run_benchmark rule in your Intercept policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.
Set action: deny in the Intercept policy for run_benchmark. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.
run_benchmark is provided by the Metrillm MCP server (metrillm-mcp). Intercept sits as a proxy in front of this server to enforce policies before tool calls reach the server.
Open source. One binary. Zero dependencies.
npx -y @policylayer/intercept