The evidence is all around you. LLM systems hate your website but love your content. The second Mark Zuckerberg announced he was paying $15 billion for 49% of an AI scraping company called “Scale AI”, a company that would lose nearly 100% of its customers in the following days, an explosion of LLM systems around the world started lighting up the internet – more than they already had. Mark isn’t to blame here, the number of LLM bots has been increasing since ChatGPT first rolled out in November of 2022 as tech startup founders seek their fortunes and big tech smells huge opportunities.

Our tools started picking up heavier than usual load last week but lower or flat traffic across client sites, a few days later clients of ours using shared hosting plans began to crash. Our team dug in, worked 16-hour days and poured through log files discovering several unfriendly LLM systems either through fingerprinting or through their bot declarations. Now more and more websites are starting to feel the pain either from their servers crashing or from paying heavy overage fees to their hosting companies due to LLM-bot activity. We’re hearing the stories almost daily at an increasing pace.

All of this is happening because Google decided to follow in OpenAI’s footsteps early on and steal content from creators and websites instead of working with them. they determined to use existing consent frameworks designed for search to ensure Google’s AI could always steal your content instead of developing new ways to control this new way of using your content. Currently there are only 2 ways to stop an LLM bot from crawling and scraping your site, neither option is great:

1. Robots.txt block on the user-agent – If you block an LLM-bot from your site using their user-agent name they shouldn’t scrape any of your site’s content. Except several LLM systems are known to ignore this including OpenAI, Perplexity, Google, and Meta. Instead they only obey this directive for “pre-training” their LLM systems and allow user initiated actions to ignore the Robots.txt directive, meaning the bots or users of those systems can abuse your site and scrape at will and there’s nothing you can do about it.

2. Noindexing your page – The Robots meta tag and the noindex attribute emerged in the mid-00’s with collaboration from several large search engines at the time. The goal was to give website owners a more granular way of telling an engine to not crawl or index a page. Ultimately, these engines settled on two ways to do this: The X-Robots Noindex tag and the Meta Robots Noindex tag. The Meta Robots Noindex tag could also include an attribute for “Nofollow” to keep link-based engines from crawling through links on the page and discovering content. LLMs supposedly respect this tag as well and Google has stated you could use this to avoid your content being scraped by their LLM system.

If you’re keeping score at home that means your only options are to shut a page off from all search engines AND LLMs simultaneously with an X-Robots or Meta Robots Noindex tag OR try to block the LLM bot’s user-agent at the Robots.txt level knowing that most will just ignore this request.

The LLM Consent Framework: A proposal for Cooperative Behavior of LLM-based AI Systems

Today I want to propose a full Consent Framework for LLM systems due to the extreme harm they can cause the web ecosystem and businesses who build on the web. This framework is designed to allow creators and website owners to decide what content they give to LLM systems and what content they withhold from either pre-training or user scraping.

1. Respect Robots.txt Blocks – LLMs should immediately stop pretending that user initiated actions give them permission to ignore Robots.txt. We’ve seen LLM systems not just ignore Robots.txt directives but also then allow a user to take content from behind a subscriber gated publication, insert it into the context window, and rewrite or summarize it. Robots.txt is supposed to be the respected firewall for websites and though Google and other search engines downgraded its power years ago, when specifically declared, LLM bots should respect this boundary.

2. The NoLLM or NoScrape Robots Meta and X-Robots tag – Instead of just “Noindex” which rips content out of search engines that benefit a website, two new attributes should be included in both the X-Robots and Meta Robots HTML tags.

“NoLLM” = The content on this page is specially not to be used in any pre-training or post-training operation now matter what.

“NoScrape” = The content on this page specially is not to be given to a user in any way.

3.The NoLLM or NoScrape HTML Attribute – In 2005 there was a battle between search engines about how to noindex content. Yandex proposed a specific HTML tag so that a section of content could be wrapped in this tag and ignored by search engines, a Microformats draft had what is called the “Robots Exclusion Profile” which would allow a class of “robots-noindex” to be used and exclude a section of content. Yahoo! proposed a similar HTML calls “robots-nocontent” which would do the same thing.

Today, I recommend that all search engines and LLM systems respect the Microformats Robots Exclusion Profile class allowing websites to keep content out of both at a more granular level than the X-Robots or Meta Robots tag.

I also recommend that we introduce a new HTML class or microformat called “robots-nollm” and “robots-noscrape” or the “LLM Exclusion Profile” allowing websites to tell LLMs which specific sections of content are off limits to their training or post-training usage OR for a human user to scrape and use themselves.

4. Correcting Non-standard usage of content in search engines – Google does not respect blocking their Google-Extended user-agent for keeping your content out of their AI Overview system which has wrecked clickthrough traffic on the web over the past year.

They do respect the “data-nosnippet” tag though, which would also keep your content from being considered for a Featured Snippet.

I propose Google and other search engines instead adopt the above convention of “data-nollm” to allow websites a more granular way of keeping their content out of unintended AI usage.

5. Social Media and UGC Consent – It seems painfully obvious that websites/apps where content is uploaded by real human users are abusing this content to either sell for their own profits to LLM companies or to try and build their own LLM offerings (i.e. Reddit, X, Facebook, TikTok, YouTube, and Instagram). This means users are caught in what I refer to as “the trap of progress”.

They can either stop contributing to these platforms which make them a living OR they can allow this unintended use of their content (or accidentally agree to it via confusing terms of usage documents) ultimately helping the host company make millions or billions while suffering future losses from the abuse of their identity and creative works. Here I propose that social media / UGC websites gain EXCPLICIT user permission to allow their content for LLM pre-training and post-training usage.

Adversarial Needs and Next Moves

Considering the above framework will likely be 100% ignored by or a majority ignored by big tech and big AI systems – including small upstarts around the globe – we need a way to enforce good behavior. Thankfully there is some progress in this direction.

Current Adversarial Capabilities:

Cloudflare’s Easy No AI Button – Announced last year Cloudflare allows all accounts to block all known / suspected AI bots with one click. This might just save your business if LLM systems continue to abuse the web and don’t behave well. The system appears to block GoogleOther user-agent but no word on Google-Extended. It allows normal search bots though, but these bots abuse your content in a RAG system (especially Google’s bot which will use your content for AI Overviews unless you use data-nosnippet or noindex the page).

Cloudflare’s AI Labyrinth – Earlier this year Cloudflare announced a way for users to opt-in to a more aggressive approach to AI scraping – burn up their resources. In this system Cloudflare presents the AI bots disrespecting your content with a series of AI generated content that wastes their time and resources while protecting your content from scaping.

Our List of Good and Bad LLM Bots and Recommendations – We’ve been pouring through client access logs and finding bots that should be blocked. Most of this is easy for Cloudflare customers so you might consider switching over to Cloudflare, if you don’t want to you can block user-agents listed in your Robots.txt and ip addresses at the server level. We’ve discovered companies like Tata in India are likely preparing to build an AI bot system soon and are scraping the web at a high volume.

Adversarial Needs:

We need more ways to protect content online and keep traffic flowing.

Day of Action – AI systems need the internet more than the internet needs AI systems. Right now big tech – especially Google – is flexing their power and crushing traffic to websites all over the place. If big tech refuses to adopt most of or all of the above consent framework, then we should stand up for ourselves and take the web dark for a day. Noindexing every website that wishes to participate. Is this extreme? Yes, but the opposition doesn’t believe we’ll do it.

Distributed Early Alert, Detection, and Blocking System – A system for WordPress or other CMS-specific sites that acts as a WAP, uses machine learning to detect fake human users (which are almost always nefarious or LLM bots), blocks them, and quickly distributes that block list to the rest of the network. Companies like WordFence might already be working on this, if not they should get started.

Respectful LLM Systems – Decades ago there was a war over which browser would dominate the web. In this war websites sided with the one they chose by adding badges to their site like “best viewed on Netscape” to help promote the browser they wanted to win. It feels like it is time we do the same again for respectful LLM systems that follow the above consent framework and allow creators to be in control.

Host-level LLM Blocking – It is time for hosting companies to start enforcing good behavior for LLM systems. Any good webhost out there should be examining ways to build tools like Cloudflare did to protect their client’s websites and apps from LLM bots. From cloud systems like Digital Ocean and AWS to WordPress hosts like Kinsta and WP Engine to shared hosting companies like BlueHost and SiteGround. The web needs you to build tools to detect and block the surge of LLM scraping activity to protect their content.

Bot Detection and X-Robots / Meta Robots Replacement – One adversarial way we might be able to beat many LLM bots is to detect the user-agent or use fingerprinting to determine it is likely a non-human user / non-friendly bot and present that bot’s sessions specifically with either an X-Robots or Meta Robots Noindex tag. This would keep many bots who scrape but follow current guidelines from scraping content.

Next Steps

We are giving all LLM systems 30-days from the date this proposal was published to offer counter points, adopt the framework, or reject it. At the end of that 30-day timeframe we’ll be announcing next steps as our small agency works to protect the creative works, content, and livelihoods of millions of creators on the web.