Robots.txt

robots.txt
noun

Definition:
A plain text file placed at the root of a website that instructs search engine crawlers which pages or directories should or should not be accessed or indexed. It uses directives like Disallow, Allow, and User-agent.

Usage:
“The robots.txt file blocked crawlers from accessing the admin dashboard.”

Compare:
meta robots tag, sitemap.xml

Joe Youngblood

view all posts

Joe Youngblood is a top Dallas SEO, Digital Marketer, and Marketing Theorist. When he's not working with clients or writing about marketing he spends time supporting local non-profits and taking his dogs to various parks.

COMMENTS Join the Conversation →