# get_robots_txt_data

Retrieve robots.txt analysis data. Shows how websites configure crawler access rules, particularly for AI crawlers. Useful for understanding web crawler policies across domains.

Agent View of the PolicyLayer registry record for `get_robots_txt_data`. HTML page: https://policylayer.com/tools/cloudflare/get-robots-txt-data

## Facts

- Tool: `get_robots_txt_data`
- Server: Cloudflare (`https://builds.mcp.cloudflare.com/sse`) — https://policylayer.com/tools/cloudflare.md
- Homepage: https://github.com/cloudflare/mcp-server-cloudflare
- Risk category: Read (Low risk)
- Registry record: grade D, identity verified
- Server auth posture: gated
- Server rate-limited: no
- Parameters: 0
- Recommended policy verdict: Allowed

## Example call (MCP tools/call, JSON-RPC 2.0)

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "get_robots_txt_data",
    "arguments": {}
  }
}
```

## Why get_robots_txt_data is rated Low

This tool retrieves and analyzes existing robots.txt files, which are public HTTP resources. It performs no mutations, executions, or side effects—purely fetching and presenting data about web crawler policies. The read-only nature and public data source result in low severity and high confidence.

From the tool's own definition: "Tool name 'get_robots_txt_data' and description 'Retrieve robots.txt analysis data' indicate data retrieval with no modification. The description explicitly states it 'Shows' and is 'Useful for understanding', both passive observation verbs."

## Use case

AI agents call get_robots_txt_data to retrieve information from Cloudflare without modifying anything. It is typically the context-gathering step in research, monitoring, and reporting workflows, before the agent takes action elsewhere.

## Recommended policy (PolicyLayer)

Verdict: **Allowed**. Enforced by the PolicyLayer MCP gateway (https://policylayer.com/mcp-gateway) before a call reaches Cloudflare:

```json
{
  "version": "1",
  "default": "deny",
  "tools": {
    "get_robots_txt_data": {}
  }
}
```

## Other tools on Cloudflare (91)

- `container_file_delete` — Destructive — https://policylayer.com/tools/cloudflare/container-file-delete.md
- `container_exec` — Execute — https://policylayer.com/tools/cloudflare/container-exec.md
- `container_initialize` — Execute — https://policylayer.com/tools/cloudflare/container-initialize.md
- `dex_create_remote_pcap` — Execute — https://policylayer.com/tools/cloudflare/dex-create-remote-pcap.md
- `dex_create_remote_warp_diag` — Execute — https://policylayer.com/tools/cloudflare/dex-create-remote-warp-diag.md
- `container_file_read` — Read — https://policylayer.com/tools/cloudflare/container-file-read.md
- `container_files_list` — Read — https://policylayer.com/tools/cloudflare/container-files-list.md
- `container_ping` — Read — https://policylayer.com/tools/cloudflare/container-ping.md
- `dex_analyze_warp_diag` — Read — https://policylayer.com/tools/cloudflare/dex-analyze-warp-diag.md
- `dex_explore_remote_warp_diag_output` — Read — https://policylayer.com/tools/cloudflare/dex-explore-remote-warp-diag-output.md
- `dex_fleet_status_live` — Read — https://policylayer.com/tools/cloudflare/dex-fleet-status-live.md
- `dex_fleet_status_logs` — Read — https://policylayer.com/tools/cloudflare/dex-fleet-status-logs.md
- `dex_fleet_status_over_time` — Read — https://policylayer.com/tools/cloudflare/dex-fleet-status-over-time.md
- `dex_http_test_details` — Read — https://policylayer.com/tools/cloudflare/dex-http-test-details.md
- `dex_list_colos` — Read — https://policylayer.com/tools/cloudflare/dex-list-colos.md
- `dex_list_remote_capture_eligible_devices` — Read — https://policylayer.com/tools/cloudflare/dex-list-remote-capture-eligible-devices.md
- `dex_list_remote_captures` — Read — https://policylayer.com/tools/cloudflare/dex-list-remote-captures.md
- `dex_list_remote_warp_diag_contents` — Read — https://policylayer.com/tools/cloudflare/dex-list-remote-warp-diag-contents.md
- `dex_list_tests` — Read — https://policylayer.com/tools/cloudflare/dex-list-tests.md
- `dex_list_warp_change_events` — Read — https://policylayer.com/tools/cloudflare/dex-list-warp-change-events.md
- `dex_test_statistics` — Read — https://policylayer.com/tools/cloudflare/dex-test-statistics.md
- `dex_traceroute_test_details` — Read — https://policylayer.com/tools/cloudflare/dex-traceroute-test-details.md
- `dex_traceroute_test_network_path` — Read — https://policylayer.com/tools/cloudflare/dex-traceroute-test-network-path.md
- `dex_traceroute_test_result_network_path` — Read — https://policylayer.com/tools/cloudflare/dex-traceroute-test-result-network-path.md
- `dns_report` — Read — https://policylayer.com/tools/cloudflare/dns-report.md
- `get_ai_data` — Read — https://policylayer.com/tools/cloudflare/get-ai-data.md
- `get_annotations` — Read — https://policylayer.com/tools/cloudflare/get-annotations.md
- `get_as_details` — Read — https://policylayer.com/tools/cloudflare/get-as-details.md
- `get_as_relationships` — Read — https://policylayer.com/tools/cloudflare/get-as-relationships.md
- `get_as_set` — Read — https://policylayer.com/tools/cloudflare/get-as-set.md
- …and 61 more: https://policylayer.com/tools/cloudflare.md

## For agents

This record is a snapshot. Live verdicts and the full registry:

- Check every server in your MCP config at once: `npx -y policylayer stack`
- Vet a server before you add it: install the mcp-precheck skill — `npx skills add https://policylayer.com` (skill text: https://policylayer.com/skill.md)
- Query the registry over MCP: endpoint `https://api.policylayer.com/mcp` — tools `check_mcp_server`, `check_mcp_stack`, `check_tool`, `search_registry`, `get_change_events`

---

Source: the PolicyLayer MCP registry — one continuously verified record per MCP server. Full record: https://policylayer.com/registry?q=cloudflare · API: https://policylayer.com/registry/api · Policy library: https://policylayer.com/policies/cloudflare