Runs a Dingo evaluation (rule-based or LLM-based) on a file. Infers data_format from input_path extension (.json, .jsonl, .txt) if not provided in kwargs. Defaults dataset to 'local' if input_path is provided and dataset is not in kwargs. If output_dir is not specified via kwargs or environment v...
AI agents invoke run_dingo_evaluation to trigger actions in Dingo MCP Server. What it does depends on the arguments the agent supplies, and its effects often reach beyond the immediate call — builds kicked off, notifications sent, workflows started.
This tool executes evaluation logic on user-provided files with outcomes that depend on the evaluation type and input data. While primarily analytical rather than destructive, it represents code/logic execution triggered by an AI agent's arguments. The ability to run LLM-based evaluations and generate output files places it in the Execute category rather than Read (which would be passive data retrieval).
From the tool's definition Tool description states it 'Runs a Dingo evaluation' on files, which triggers external operations. The evaluation can be 'rule-based or LLM-based', indicating execution of arbitrary evaluation logic.
Documented attack patterns abuse exactly the kind of access run_dingo_evaluation gives an agent:
PolicyLayer is an MCP gateway — it sits between your AI agents and Dingo MCP Server, and nothing reaches the server without passing your rules. This is the rule we recommend for run_dingo_evaluation:
{
"version": "1",
"default": "deny",
"tools": {
"run_dingo_evaluation": {
"limits": [
{
"counter": "run_dingo_evaluation_rate",
"window": "minute",
"max": 10,
"scope": "grant"
}
]
}
}
}run_dingo_evaluation stays usable, but rate-capped — a runaway agent can't fire it dozens of times a minute. Everything else on the server is denied unless you say otherwise.
Free to start. No card required.
Runs a Dingo evaluation (rule-based or LLM-based) on a file. Infers data_format from input_path extension (.json, .jsonl, .txt) if not provided in kwargs. Defaults dataset to 'local' if input_path is provided and dataset is not in kwargs. If output_dir is not specified via kwargs or environment variables, creates output relative to input_path. API keys for LLMs should be set via environment variables in mcp.json or system environment. Args: input_path: Path to the input file or directory. evaluation_type: Type of evaluation ('rule' or 'llm'), defaults to 'rule'. eval_group_name: The specific rule group or LLM model name. Defaults to empty, Dingo will use 'default' for rules or infer from custom_config for LLMs. (Optional when custom_config is provided for LLM evaluations via kwargs) kwargs: Dictionary containing additional arguments compatible with dingo.io.InputArgs. Use for: output_dir, task_name, save_data, save_correct, dataset, data_format, column_content, column_id, column_prompt, column_image, custom_config, max_workers, batch_size, etc. Returns: For Smithery deployment: The content of the result file (summary.json or first .jsonl) Otherwise: The absolute path to the primary output file (summary.json or first .jsonl). It is categorised as a Execute tool in the Dingo MCP Server MCP Server, which means it can trigger actions or run processes. Use rate limits and argument validation.
Register the Dingo MCP Server MCP server in PolicyLayer and add a rule for run_dingo_evaluation: allow, deny, rate-limit, or require approval. Point your MCP client at the PolicyLayer proxy URL and the rule is enforced on every call, before it reaches Dingo MCP Server. Nothing to install.
run_dingo_evaluation is a Execute tool with high risk. Execute tools should be rate-limited and have argument validation enabled.
Yes. Add a rate_limit block to the run_dingo_evaluation rule in your PolicyLayer policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.
Set action: deny in the PolicyLayer policy for run_dingo_evaluation. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.
run_dingo_evaluation is provided by the Dingo MCP Server MCP server (migoxlab/dingo). PolicyLayer sits as a proxy in front of this server to enforce policies before tool calls reach the server.
Start from Dingo MCP Server, add the rest of your stack, and see everything your agents can call. Then put policy on all of it.
Free to start. No card required.
6 Dingo MCP Server tools catalogued and risk-classified — across an index of 43,000+ MCP servers.