generate_robots_txt · Mcp Bot Crawler MCP: Risk & Policy

WHAT IT DOES

What generate_robots_txt does on Mcp Bot Crawler

AI agents use generate_robots_txt to create or update resources in Mcp Bot Crawler — usually the action step of a workflow, after the agent has gathered context. Every call changes real data in your Mcp Bot Crawler environment.

WHY IT NEEDS A POLICY

Why generate_robots_txt needs a policy

This tool creates or modifies the robots.txt file, which is a reversible write operation. While robots.txt changes affect site visibility and bot access policies, the modification itself is not destructive (can be reverted), not financial, and not code execution.

From the tool's definition Tool description states it will 'Generate a production-ready robots.txt' and 'declares sitemaps', indicating creation/modification of the robots.txt file on the website.

Attacks that exploit this kind of access

Recommended rule Ready

Verdict

generate_robots_txt Rate-limited

generate_robots_txt stays usable, but capped — an agent stuck in a loop can't make hundreds of changes a minute. Everything else on the server is denied unless you say otherwise.

View as policy code

policy.json

{
  "version": "1",
  "default": "deny",
  "tools": {
    "generate_robots_txt": {
      "limits": [
        {
          "counter": "generate_robots_txt_rate",
          "window": "minute",
          "max": 30,
          "scope": "grant"
        }
      ]
    }
  }
}

LIMIT THIS TOOL →

Instant setup, no code required.

PolicyLayer is an MCP gateway: every tool call is checked against your rule before it reaches Mcp Bot Crawler, so you can run agents against it safely.

FAQ

Questions about generate_robots_txt

What does the generate_robots_txt tool do? +

Generate a production-ready robots.txt that explicitly allows trusted bots, blocks chosen categories (AI, SEO, scrapers), and declares sitemaps. It is categorised as a Write tool in the Mcp Bot Crawler MCP Server, which means it can create or modify data. Consider rate limits to prevent runaway writes.

How do I enforce a policy on generate_robots_txt? +

Register the Mcp Bot Crawler MCP server in PolicyLayer and add a rule for generate_robots_txt: allow, deny, rate-limit, or require approval. Point your MCP client at the PolicyLayer proxy URL and the rule is enforced on every call, before it reaches Mcp Bot Crawler. Nothing to install.

What risk level is generate_robots_txt? +

generate_robots_txt is a Write tool with medium risk. Write tools should be rate-limited to prevent accidental bulk modifications.

Can I rate-limit generate_robots_txt? +

Yes. Add a rate_limit block to the generate_robots_txt rule in your PolicyLayer policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.

How do I block generate_robots_txt completely? +

Set action: deny in the PolicyLayer policy for generate_robots_txt. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.

What MCP server provides generate_robots_txt? +

generate_robots_txt is provided by the Mcp Bot Crawler MCP server (merulocal/hellogrowthcrmwebsite_mcp). PolicyLayer sits as a proxy in front of this server to enforce policies before tool calls reach the server.