Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact with the site. **Best for:** Creating machine-readable permission guidelines for AI models. **Not recommended for:** General content extraction or...
Accepts URL/endpoint input (url); Single-target operation
Part of the Firecrawl Web Scraping Server MCP server. Enforce policies on this tool with Intercept, the open-source MCP proxy.
AI agents use firecrawl_generate_llmstxt to create or modify resources in Firecrawl Web Scraping Server. Write operations carry medium risk because an autonomous agent could trigger bulk unintended modifications. Rate limits prevent a single agent session from making hundreds of changes in rapid succession. Argument validation ensures the agent passes expected values.
Without a policy, an AI agent could call firecrawl_generate_llmstxt repeatedly, creating or modifying resources faster than any human could review. Intercept's rate limiting ensures write operations happen at a controlled pace, and argument validation catches malformed or unexpected inputs before they reach Firecrawl Web Scraping Server.
Write tools can modify data. A rate limit prevents runaway bulk operations from AI agents.
tools:
firecrawl_generate_llmstxt:
rules:
- action: allow
rate_limit:
max: 30
window: 60See the full Firecrawl Web Scraping Server policy for all 8 tools.
Agents calling write-class tools like firecrawl_generate_llmstxt have been implicated in these attack patterns. Read the full case and prevention policy for each:
Other tools in the Write risk category across the catalogue. The same policy patterns (rate-limit, validate) apply to each.
Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact with the site. **Best for:** Creating machine-readable permission guidelines for AI models. **Not recommended for:** General content extraction or research. **Arguments:** - url (string, required): The base URL of the website to analyze. - maxUrls (number, optional): Max number of URLs to include (default: 10). - showFullText (boolean, optional): Whether to include llms-full.txt contents in the response. **Prompt Example:** "Generate an LLMs.txt file for example.com." **Usage Example:** ```json { "name": "firecrawl_generate_llmstxt", "arguments": { "url": "https://example.com", "maxUrls": 20, "showFullText": true } } ``` **Returns:** LLMs.txt file contents (and optionally llms-full.txt). . It is categorised as a Write tool in the Firecrawl Web Scraping Server MCP Server, which means it can create or modify data. Consider rate limits to prevent runaway writes.
Add a rule in your Intercept YAML policy under the tools section for firecrawl_generate_llmstxt. You can allow, deny, rate-limit, or validate arguments. Then run Intercept as a proxy in front of the Firecrawl Web Scraping Server MCP server.
firecrawl_generate_llmstxt is a Write tool with medium risk. Write tools should be rate-limited to prevent accidental bulk modifications.
Yes. Add a rate_limit block to the firecrawl_generate_llmstxt rule in your Intercept policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.
Set action: deny in the Intercept policy for firecrawl_generate_llmstxt. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.
firecrawl_generate_llmstxt is provided by the Firecrawl Web Scraping Server MCP server (0xzapata/firecrawl-mcp-server). Intercept sits as a proxy in front of this server to enforce policies before tool calls reach the server.
Deterministic policy on every MCP tool call. Per-identity grants. Full audit log.