seo__check_site_crawlers
［搜尋引擎與 AI 爬蟲工具箱］檢查網站的 robots.txt、Sitemap、llms.txt，以及 Googlebot、bingbot、GPTBot、ClaudeBot、PerplexityBot 等是否被封鎖，附改善建議。
This record as markdown: /tools/com-penguindriver-hub/seo--check-site-crawlers.md
What seo__check_site_crawlers does on AI 代理人任務中樞
AI agents call seo__check_site_crawlers to retrieve information from AI 代理人任務中樞 without modifying anything. It is typically the context-gathering step in research, monitoring, and reporting workflows, before the agent takes action elsewhere.
| Parameter | Type | Required | Description |
|---|---|---|---|
url | string | Yes |
Parameters from the server's own tool schema.
Why seo__check_site_crawlers is rated Low
Even though seo__check_site_crawlers only reads data, uncontrolled read access leaks sensitive information and racks up API costs: an agent caught in a retry loop can make thousands of calls a minute without anyone noticing.
Risk signalsAccepts URL/endpoint input (url)
Attacks that exploit this kind of access
The rule that runs seo__check_site_crawlers safely
PolicyLayer is an MCP gateway: it sits between your AI agents and AI 代理人任務中樞, and checks every tool call against a rule you set before the call runs. Nothing changes on the server itself. For seo__check_site_crawlers, this is the rule to start with:
seo__check_site_crawlers is read-only, so it stays allowed. Everything else on the server is denied unless you say otherwise.
The button opens the PolicyLayer dashboard: create your workspace, connect AI 代理人任務中樞, apply this rule, and every seo__check_site_crawlers call is checked against it from then on.
Questions about seo__check_site_crawlers
［搜尋引擎與 AI 爬蟲工具箱］檢查網站的 robots.txt、Sitemap、llms.txt，以及 Googlebot、bingbot、GPTBot、ClaudeBot、PerplexityBot 等是否被封鎖，附改善建議。. It is categorised as a Read tool in the AI 代理人任務中樞 MCP Server, which means it retrieves data without modifying state.
seo__check_site_crawlers accepts 1 parameter: url. Required: url. The full parameter table on this page comes from the server's own tool schema.
Register the AI 代理人任務中樞 MCP server in PolicyLayer and add a rule for seo__check_site_crawlers: allow, deny, rate-limit, or require approval. Point your MCP client at the PolicyLayer proxy URL and the rule is enforced on every call, before it reaches AI 代理人任務中樞. Nothing to install.
seo__check_site_crawlers is a Read tool with low risk. Read-only tools are generally safe to allow by default.
Yes. Add a rate_limit block to the seo__check_site_crawlers rule in your PolicyLayer policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.
Set action: deny in the PolicyLayer policy for seo__check_site_crawlers. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.
seo__check_site_crawlers is provided by the AI 代理人任務中樞 MCP server (https://hub.penguindriver.com/mcp). PolicyLayer sits as a proxy in front of this server to enforce policies before tool calls reach the server.
More on AI 代理人任務中樞, and thousands of servers like it.
This server
Across the catalogue