X-Robots-Tag HTTP Headers

Because HTML meta tags only work in HTML pages, there needed to be a way for other files to send a message to bots that blocks them from being indexed.

That method is an HTTP header called the X-Robots-Tag.

How to implement this header requires programming knowledge so we won’t go into the details yet, but when a Web server includes a tag with the name X-Robots-Tag (case-insensitive), search engines are supposed to respect the values.

Here’s an example of an X-Robots-Tag telling search engines not to include the page in the search results:

x-robots-tag: noindex

To learn more about the possible values of the header, see this page.

Takeaways

• It’s possible to tell search engines not to index files on the Web by including an X-Robots-Tag in the HTTP response headers.