Generate standardized LLMs.txt file for a given URL, which provides context about how LLMs should interact with the website.
Accepts URL/endpoint input (url); Single-target operation
Part of the Firecrawl Web Scraping Server MCP server. Enforce policies on this tool with Intercept, the open-source MCP proxy.
AI agents use firecrawl_generate_llmstxt to create or modify resources in Firecrawl Web Scraping Server. Write operations carry medium risk because an autonomous agent could trigger bulk unintended modifications. Rate limits prevent a single agent session from making hundreds of changes in rapid succession. Argument validation ensures the agent passes expected values.
Without a policy, an AI agent could call firecrawl_generate_llmstxt repeatedly, creating or modifying resources faster than any human could review. Intercept's rate limiting ensures write operations happen at a controlled pace, and argument validation catches malformed or unexpected inputs before they reach Firecrawl Web Scraping Server.
Write tools can modify data. A rate limit prevents runaway bulk operations from AI agents.
tools:
firecrawl_generate_llmstxt:
rules:
- action: allow
rate_limit:
max: 30
window: 60See the full Firecrawl Web Scraping Server policy for all 10 tools.
Agents calling write-class tools like firecrawl_generate_llmstxt have been implicated in these attack patterns. Read the full case and prevention policy for each:
Other tools in the Write risk category across the catalogue. The same policy patterns (rate-limit, validate) apply to each.
Generate standardized LLMs.txt file for a given URL, which provides context about how LLMs should interact with the website.. It is categorised as a Write tool in the Firecrawl Web Scraping Server MCP Server, which means it can create or modify data. Consider rate limits to prevent runaway writes.
Add a rule in your Intercept YAML policy under the tools section for firecrawl_generate_llmstxt. You can allow, deny, rate-limit, or validate arguments. Then run Intercept as a proxy in front of the Firecrawl Web Scraping Server MCP server.
firecrawl_generate_llmstxt is a Write tool with medium risk. Write tools should be rate-limited to prevent accidental bulk modifications.
Yes. Add a rate_limit block to the firecrawl_generate_llmstxt rule in your Intercept policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.
Set action: deny in the Intercept policy for firecrawl_generate_llmstxt. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.
firecrawl_generate_llmstxt is provided by the Firecrawl Web Scraping Server MCP server (Krieg2065/firecrawl-mcp-server). Intercept sits as a proxy in front of this server to enforce policies before tool calls reach the server.
Open source. One binary. Zero dependencies.
npx -y @policylayer/intercept