web_scrape
Scrape a URL into markdown, HTML, links or metadata. Fetches a single public URL and returns clean content in the requested formats (markdown, html, raw_html, links, metadata). With render=auto the request starts as a fast HTTP fetch and escalates to a real browser when the page is blocked or ren...
This record as markdown: /tools/crawlora-mcp/web-scrape.md
What web_scrape does on Crawlora
AI agents call web_scrape to retrieve information from Crawlora without modifying anything. It is typically the context-gathering step in research, monitoring, and reporting workflows, before the agent takes action elsewhere.
| Parameter | Type | Required | Description |
|---|---|---|---|
scrapeOption | object | Yes | Scrape options |
Parameters from the server's own tool schema.
Why web_scrape is rated Low
This tool performs web scraping—fetching and parsing publicly available web content into various formats (markdown, HTML, links, metadata). While it retrieves data without side effects on the target server, the medium severity reflects the potential for misuse: unauthorized scraping can violate terms of service, robots.txt directives, copyright, rate limits, or privacy regulations.
From the tool's definition "Fetches a single public URL and returns clean content" - the tool retrieves data from web pages. The description explicitly states "only public pages are supported" and advises to "respect each site's terms of use and robots directives", indicating it…
Attacks that exploit this kind of access
The rule that runs web_scrape safely
PolicyLayer is an MCP gateway: it sits between your AI agents and Crawlora, and checks every tool call against a rule you set before the call runs. Nothing changes on the server itself. For web_scrape, this is the rule to start with:
web_scrape is read-only, so it stays allowed. Everything else on the server is denied unless you say otherwise.
The button opens the PolicyLayer dashboard: create your workspace, connect Crawlora, apply this rule, and every web_scrape call is checked against it from then on.
Questions about web_scrape
Scrape a URL into markdown, HTML, links or metadata. Fetches a single public URL and returns clean content in the requested formats (markdown, html, raw_html, links, metadata). With render=auto the request starts as a fast HTTP fetch and escalates to a real browser when the page is blocked or rendered with JavaScript. only_main_content (default true) strips navigation, headers, footers and other boilerplate before conversion. Only public pages are supported; respect each site's terms of use and robots directives. It is categorised as a Read tool in the Crawlora MCP Server, which means it retrieves data without modifying state.
web_scrape accepts 1 parameter: scrapeOption. Required: scrapeOption. The full parameter table on this page comes from the server's own tool schema.
Register the Crawlora MCP server in PolicyLayer and add a rule for web_scrape: allow, deny, rate-limit, or require approval. Point your MCP client at the PolicyLayer proxy URL and the rule is enforced on every call, before it reaches Crawlora. Nothing to install.
web_scrape is a Read tool with low risk. Read-only tools are generally safe to allow by default.
Yes. Add a rate_limit block to the web_scrape rule in your PolicyLayer policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.
Set action: deny in the PolicyLayer policy for web_scrape. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.
web_scrape is provided by the Crawlora MCP server (crawlora-mcp). PolicyLayer sits as a proxy in front of this server to enforce policies before tool calls reach the server.
More on Crawlora, and thousands of servers like it.
This server
Across the catalogue