Here’s how Cloudflare, which for many years was renowned for protecting websites from unwanted scrapers and bots, decided to surprise the public — by releasing their own parser. They announced a new endpoint /crawl for Browser Rendering, which allows fully scanning and extracting the content of any website with a single API request. This tool is aimed at creating RAG pipelines, training artificial intelligence, monitoring, and conducting various research.
It’s amusing that Cloudflare—one of the main defenders of websites against malicious crawlers and bots that collect data for AI training—now releases their own parser, creating a bit of irony in the situation.
As a justification, they note that their bot—unlike most similar entities—will behave properly: respecting website rules and robots.txt 😇. This statement seems especially amusing given their reputation for protecting resources from such automated scanners.
Created with n8n:
https://cutt.ly/n8n
Created with syllaby:
https://cutt.ly/syllaby