Which file lists the directories and files that the website owner wants to hide from web crawlers?

Which file lists the directories and files that the website owner wants to hide from web crawlers?

• A. robots.txt
• B. exclude.html
• C. crawlignore.txt
• D. sites.txt

Answer: (A)

Telling web crawlers which parts of a site to ignore is done with a robots.txt file placed at the site’s root. This plain text file lists directories and files you don’t want search engines to crawl or index, using directives like User-agent and Disallow. For example, you can specify that all crawlers should avoid /admin/ by adding a line that Disallow: /admin/. This is the standard robots exclusion protocol and is widely honored by major crawlers. Remember, robots.txt is publicly accessible and not a security barrier—some agents may ignore it or access disallowed paths directly. Other proposed options aren’t the established mechanism for controlling crawling, so they don’t serve the same purpose.

Telling web crawlers which parts of a site to ignore is done with a robots.txt file placed at the site’s root. This plain text file lists directories and files you don’t want search engines to crawl or index, using directives like User-agent and Disallow. For example, you can specify that all crawlers should avoid /admin/ by adding a line that Disallow: /admin/. This is the standard robots exclusion protocol and is widely honored by major crawlers. Remember, robots.txt is publicly accessible and not a security barrier—some agents may ignore it or access disallowed paths directly. Other proposed options aren’t the established mechanism for controlling crawling, so they don’t serve the same purpose.