Run an evaluator on a single log/span to verify it works. This is for quick verification of one record (e.g. confirm an evaluator scores as expected before running broader experiments). For scoring many records, create an experiment instead. Returns the actual score (boolean_value / numerical_val...
AI agents invoke run_evaluator to trigger actions in Respan MCP Server. What it does depends on the arguments the agent supplies, and its effects often reach beyond the immediate call — builds kicked off, notifications sent, workflows started.
This tool triggers execution of an evaluator against log/span data. While the immediate outcome is a score retrieval (which could be Read), the core action is executing an evaluator—a computational operation whose side effects and outputs are data-dependent. This falls under Execute rather than Read because it runs code/logic rather than passively retrieving pre-computed data.
From the tool's definition Tool name 'run_evaluator' combined with description 'Run an evaluator on a single log/span' and 'Returns the actual score'.
Attacks that exploit this kind of access
Run an evaluator on a single log/span to verify it works. This is for quick verification of one record (e.g. confirm an evaluator scores as expected before running broader experiments). For scoring many records, create an experiment instead. Returns the actual score (boolean_value / numerical_value / etc.) and cost. It is categorised as a Execute tool in the Respan MCP Server MCP Server, which means it can trigger actions or run processes. Use rate limits and argument validation.
Register the Respan MCP Server MCP server in PolicyLayer and add a rule for run_evaluator: allow, deny, rate-limit, or require approval. Point your MCP client at the PolicyLayer proxy URL and the rule is enforced on every call, before it reaches Respan MCP Server. Nothing to install.
run_evaluator is a Execute tool with high risk. Execute tools should be rate-limited and have argument validation enabled.
Yes. Add a rate_limit block to the run_evaluator rule in your PolicyLayer policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.
Set action: deny in the PolicyLayer policy for run_evaluator. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.
run_evaluator is provided by the Respan MCP Server MCP server (respanai/respan-mcp). PolicyLayer sits as a proxy in front of this server to enforce policies before tool calls reach the server.
Every MCP server has a record like this.
Type a name, get the same breakdown: verified identity, auth posture, risk grade, capabilities, recommended policy.
Teams ship this data inside their own products. See what a licence covers →