Robots.txt
robots.txt noun Definition: A plain text file placed at the root of a website that instructs search engine crawlers which pages or directories should or should not be accessed or indexed. It uses directives like Disallow, Allow, and User-agent. Usage: “The robots.txt file blocked crawlers from accessing the...
