Cloudflare wants to help stomp out AI scraper bots for good

News
By
published

AI bot blocking is about to become more effective with new Cloudflare tool

a face with a blue filter and covered in words
Image Credit: Pixabay (Image credit: Geralt / Pixabay)

Cloudflare has announced a new feature that allows web hosting customers to block AI bots from scraping their website content without permission.

Companies responsible for AI tools, particularly those involved with training, have come under fire in recent months over their use of content published online.

Cloudflare says its new tool responds to widespread discontent among website owners and publishers about AI bots harvesting data and aims to protect content creators on the internet.

Cloudflare tool to stop AI bot scraping

In its announcement, Cloudflare wrote: “We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly. To help, we’ve added a brand new one-click to block all AI bots.”

The tool has become available for all customers, including those on Cloudflare’s free tier.

Previously, website owners have been able to use the robots.txt file to present bots from crawling their sites, but to varying degrees of success. The new tools offers a more robust defence against content scraping by using machine learning to identify and block bots.

Cloudflare’s system relies on digital fingerprinting to differentiate bots from legitimate users.

The company said that the enhanced protection comes as a response to significant AI bot activity across its network. Bytespider, GPTBot and ClaudeBot were among the most commonly observed, but despite previous blocking attempts, AI bots accessed more than one-third (39%) of the top one million sites served by Cloudflare in June.

Thankfully, the new feature is available to all Cloudflare customers, including those on the free tier. By navigating to the Security > Bots menu and blocking “AI Scrapers and Crawlers,” users can protect their website from unwanted AI bot activity. 

More from TechRadar Pro

Craig Hale
Craig Hale

With several years’ experience freelancing in tech and automotive circles, Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars and the decarbonisation of personal transportation. As an avid bargain-hunter, you can be sure that any deal Craig finds is top value!