X.com’s Current Robots.txt File

Pulled: September 7, 2026

Below is a copy of the Robots.txt currently being served by X to users (Desktop + Chrome). This was accurate as of 7:00pm Central (Dallas) time on September 7th, 2026.

User-agent: Googlebot
# Google / Bing Search Engine Robots
# ==================================
# Shared group so Googlebot and Bingbot always get the same rules.
User-agent: Googlebot
User-agent: Bingbot

Allow: /*?s=
Allow: /*?t=
Allow: /*?ref_src=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Allow: /i/api/
# Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in
# favor of Allow, so /i/api/ fetches stay crawlable.
Allow: /i/api/*
Disallow: /*?lang=en-ss
Allow: /*?lang=
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*/analytics
Disallow: /*/followers
Disallow: /*/following
Disallow: /*/verified_followers

Disallow: /account/deactivated
Disallow: /settings/deactivated

# Only `*` and `$` are metacharacters here (RFC 9309); regex classes match
# literally. /photo is anchored so /status/*/photo/1 media pages stay open.
Disallow: /*/likes
Disallow: /*/likes?
Disallow: /*/retweets
Disallow: /*/retweets?
Disallow: /*/media
Disallow: /*/media?
Disallow: /*/photo$
Disallow: /*/photo?
Allow: /*?

User-agent: facebookexternalhit

Allow: /*?lang=
Allow: /*?s=
Allow: /*?t=
Allow: /*?ref_src=
Allow: /hashtag/*?src=
Allow: /search?*cashtagRestId=
Allow: /i/api/
Allow: /i/api/*
Disallow: /search?q=
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/followers
Disallow: /*/following
Disallow: /*/verified_followers

Disallow: /account/deactivated
Disallow: /settings/deactivated

Disallow: /*/likes
Disallow: /*/likes?
Disallow: /*/retweets
Disallow: /*/retweets?
Disallow: /*/media
Disallow: /*/media?
Disallow: /*/photo$
Disallow: /*/photo?

User-Agent: Google-Extended
Disallow: *

User-Agent: FacebookBot
Disallow: *

User-agent: Discordbot
Disallow: *

# Meta AI / product crawlers (product tokens; versions like /1.1 are ignored
# under RFC 9309 token matching).
User-agent: meta-webindexer
Disallow: *

User-agent: meta-externalagent
Disallow: *

User-agent: meta-externalads
Disallow: *

User-agent: meta-externalfetcher
Disallow: *

# Every bot that might possibly read and respect this file
# ========================================================
User-agent: *
Disallow: /


# WHAT-4882 - Keep notification-email links (/i/u) out of search results.
# Named crawlers stay un-blocked on purpose: they must crawl /i/u to see its
# X-Robots-Tag noindex (the robots.txt Noindex directive died in 2019).
Disallow: /i/u

# Wait 1 second between successive requests. See ONBOARD-2698 for details.
Crawl-delay: 1

# Independent of user agent. Links in the sitemap are full URLs using https://
# and need to match the protocol of the sitemap.
Sitemap: https://x.com/sitemap.xml

Which Search and AI Bots Can Crawl X?

Here is how the current file shakes out for most of the major search, AI search, training, and discovery crawlers we care about today.

*The Applebot situation is especially interesting and I discuss it farther down.

X Is Basically Running a Crawler Whitelist

The most important part of the file isn’t actually one of the individual crawler blocks.

It is this:

User-agent: *
Disallow: /

That flips the normal way many websites approach robots.txt.

Instead of allowing the web by default and blocking specific crawlers X doesn’t like, X is effectively saying:

Nobody gets in unless we give you a specific path around the block.

That is why the lack of an OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot, or PerplexityBot section matters.

X doesn’t need to individually block all of them.

They’re already blocked.

Google and Bing Are Clearly the Preferred Search Engines

The first group in the file is shared by Googlebot and Bingbot:

User-agent: Googlebot
User-agent: Bingbot

That is about as explicit as X could make its intentions.

Both search engines receive the same crawling permissions and restrictions.

X allows them into most of the public-facing content we actually care about from a search perspective while restricting URLs such as:

• /*/analytics
• /*/followers
• /*/following
• /*/verified_followers
• /*/likes
• /*/retweets
• /*/media
• Various internal search URLs
• Deactivated-account URLs

This is mostly sensible crawl management.

A public post or profile can have substantial search value. There isn’t much reason to let Google crawl millions upon millions of follower, following, likes, retweet, and internal search combinations.

The strategy appears to be:

Index the content. Don’t crawl the social graph and endless internal URL permutations.

X Explicitly Blocks Google-Extended

Then we get this:

User-Agent: Google-Extended
Disallow: *

This is where the distinction between traditional SEO and AI visibility starts becoming important.

Google-Extended is not another version of Googlebot that goes around crawling the web. It is a robots.txt product token Google uses to let publishers control certain uses of content Google has crawled.

Google currently says Google-Extended controls whether crawled website content can be used for things including training future generations of Gemini models and certain Gemini grounding uses.

Google also says this setting does not affect inclusion or rankings in Google Search.

So X is effectively saying:

Google Search: Yes.

The Gemini uses controlled by Google-Extended: No.

This is probably one of the clearest examples you’ll find of a massive website deliberately separating traditional search crawling from certain generative AI uses.

And Blocking Google-Extended Does Not Block AI Overviews or AI Mode

This distinction is worth making because it is incredibly easy to incorrectly summarize this as “X blocks Google’s AI.”

It doesn’t.

Google’s AI features inside Google Search are still part of the Google Search ecosystem. Google has repeatedly stated that Google-Extended does not affect Google Search.

Since X continues allowing Googlebot, its public content can still be crawled for Google Search and can potentially surface through Google’s search experiences, including AI-driven search features.

What X is blocking is the separate set of Gemini-related uses controlled by Google-Extended.

That is a much more precise description of what is happening.

ChatGPT Search Is Blocked

OpenAI’s search crawler is OAI-SearchBot.

OpenAI currently tells publishers that allowing OAI-SearchBot is important if they want their site content to be discovered, summarized, and cited through ChatGPT Search.

X has no OAI-SearchBot group.

That puts it here:

User-agent: *
Disallow: /

Meaning X is telling OAI-SearchBot not to crawl X.com.

The same thing happens to GPTBot, OpenAI’s crawler associated with content that may be used for model training.

No special rule.

No access.

This creates an interesting divide between Google and ChatGPT:

X isn’t simply opting out of AI training.

In OpenAI’s case it appears to be opting out of the normal search crawler too.

Claude Gets the Same Treatment

Anthropic currently separates several crawler use cases with:

• ClaudeBot for model development
• Claude-SearchBot for Claude search
• Claude-User for user-initiated retrieval

None are named in X’s robots.txt file.

Anthropic says these bots honor robots.txt, so they run straight into X’s wildcard block.

Again, X is not merely trying to stop AI training here.

It is also blocking the search-oriented crawler.

PerplexityBot Is Blocked Too

The same applies to PerplexityBot.

Perplexity describes PerplexityBot as the crawler it uses to discover and index content for Perplexity search rather than a model-training crawler.

X doesn’t distinguish between the two concepts.

PerplexityBot is not specifically allowed, so:

User-agent: *
Disallow: /

gets it.

One caveat here is that Perplexity also has a user-triggered Perplexity-User fetcher and says that user-requested fetches operate differently from normal crawling. That is another good example of why robots.txt alone no longer gives us a perfect map of every way an AI product can possibly obtain information from a URL.

Meta Gets Explicitly Blocked Over and Over Again

X doesn’t rely only on the wildcard for Meta.

It goes out of its way to name multiple Meta crawlers:

User-Agent: FacebookBot
Disallow: *

User-agent: meta-webindexer
Disallow: *

User-agent: meta-externalagent
Disallow: *

User-agent: meta-externalads
Disallow: *

User-agent: meta-externalfetcher
Disallow: *

There isn’t much ambiguity here.

X does not want Meta’s general web, AI, advertising-related, or product crawlers crawling X.

Considering X and Meta compete directly for users, attention, advertising dollars, creators, and increasingly AI usage, I can’t imagine anyone is surprised by this.

But X Still Lets Facebook Make Link Previews

One Meta crawler does get through:

User-agent: facebookexternalhit

This is the crawler historically used to fetch URLs being shared through Facebook and generate previews.

X gives it a fairly restrictive set of rules, but it is not completely blocked.

This makes sense.

If somebody shares an X post to Facebook, X still wants the link to look useful enough for somebody to click it.

Allowing a link-preview fetcher does not require giving Meta broad crawling access to the platform.

So Meta essentially gets:

Make the link preview work: Yes.

Crawl the platform: No.

The Applebot Loophole Is Probably the Most Interesting Part

Applebot isn’t mentioned anywhere in X’s robots.txt file.

At first glance you would assume that means it gets caught by:

User-agent: *
Disallow: /

But Apple has a very unusual documented behavior for Applebot.

Apple says that when a robots.txt file does not contain instructions for Applebot but does contain instructions for Googlebot, Applebot will follow the Googlebot rules.

X has an extensive Googlebot ruleset and no Applebot ruleset.

That means, according to Apple’s own documentation, Applebot should effectively inherit the permissions X gives Googlebot.

So despite X’s giant wildcard block, Apple Search, Siri, Spotlight, and other Apple experiences powered by Applebot appear to still have crawl access to much of X.

Whether X deliberately intended this or simply knows and accepts Apple’s behavior is impossible to determine from the file alone.

But from a technical SEO standpoint it is a fascinating exception.

Apple’s AI Training Is a Different Question

Apple now also has Applebot-Extended, which works somewhat like Google’s Google-Extended.

Applebot-Extended itself does not crawl webpages. It is a control Apple uses to determine whether content collected by Applebot can be used to train Apple’s general-purpose foundation models.

X does not give Applebot-Extended its own group.

Under normal robots matching, that means the wildcard Disallow: / should apply to Applebot-Extended even though Apple’s special Googlebot fallback gives regular Applebot crawling access.

So the apparent outcome is:

Apple search crawling: Yes.

Apple foundation-model training controlled by Applebot-Extended: No.

That ends up looking remarkably similar to X’s explicit strategy with Google.

There Is Another Interesting Detail Buried in X’s Comments

X left comments in the actual robots.txt file explaining why certain named crawlers are intentionally allowed to crawl its /i/u URLs.

The reason is technically correct and worth highlighting.

X apparently sends an X-Robots-Tag: noindex response for those URLs.

If X blocked Googlebot from crawling the URL in robots.txt, Google wouldn’t be able to fetch the URL and see the noindex instruction.

This is why treating robots.txt as an indexing-control system continues to cause problems.

If you want a crawler to discover a noindex instruction, the crawler needs to actually be able to crawl the URL.

X’s engineers clearly understand that distinction.

The Crawl-Delay Rule Is Basically Irrelevant

Near the bottom X also has:

Crawl-delay: 1

But it sits inside the wildcard group.

The crawlers matching that group have already been told:

Disallow: /

So compliant bots in that group shouldn’t be crawling anything anyway.

Googlebot and Bingbot have their own more-specific user-agent group and don’t inherit the wildcard group. Google also doesn’t support the non-standard crawl-delay directive.

In practical terms the crawl-delay directive isn’t doing much here.

The Sitemap Doesn’t Override Any of This

X ends the file by publishing its sitemap location.

That can help crawlers discover URLs, but finding a URL in a sitemap does not magically override a robots.txt disallow.

A blocked crawler can know an X URL exists without being permitted to crawl its content.

This distinction matters more now because AI/search systems can discover URLs through third-party indexes, links, sitemaps, user prompts, APIs, licensed datasets, and other sources even when direct crawling is restricted.

What X’s Robots.txt Really Says

If we strip away all of the syntax, X’s current crawler policy looks something like this:

Google Search: Come in.

Bing: Come in.

Apple Search: Apparently come in too.

Gemini training / grounding uses controlled by Google-Extended: No.

ChatGPT Search crawler: No.

OpenAI training crawler: No.

Claude Search: No.

Anthropic training crawler: No.

Perplexity search crawler: No.

Meta AI: Definitely no.

Facebook needs to make a preview when somebody shares an X link: Fine.

Everyone else: No.

What SEOs Should Take Away From This

I think the biggest lesson here has almost nothing to do with X.

For a very long time robots.txt decisions were primarily about search engines and crawl efficiency.

Now a website can have completely different policies for:

• Traditional search crawling
• AI search crawling
• AI model training
• Real-time AI retrieval
• Social link previews
• SEO crawlers
• Web archives
• Research crawlers
• Third-party datasets

X’s file is a great example of how complicated this is getting.

The company clearly values being discoverable in Google and Bing enough to specifically allow their crawlers. At the same time it is preventing most of the emerging AI search ecosystem from building its own crawl-based view of X.

Google gets especially interesting because X separates Google’s traditional search crawler from Google-Extended.

OpenAI gets no such separation. Both search crawling and training crawling are blocked.

Anthropic gets no separation either.

Meta gets explicitly thrown out.

And Apple appears to sneak through the front door because Apple’s crawler intentionally follows Googlebot rules when it isn’t given rules of its own.

For SEOs this is where robots.txt is heading.

We’re no longer just deciding whether Google can crawl /wp-admin/ or whether faceted URLs should be blocked.

Robots.txt is increasingly becoming one of the places where publishers decide which search engines and AI ecosystems they actually want to participate in.

X.com’s current file makes that decision about as clearly as any major website I’ve seen.

Note: This teardown reflects the X.com robots.txt file retrieved on September 7, 2026. Robots.txt files can change at any time, so the current live file should always be checked before relying on these rules.

GET A FREE SEO REVIEW