The X-Robots-Tag is an HTTP response header used to control how search engines index a webpage or file. It works similarly to the meta robots tag in HTML, but it’s applied at the server level and works for any type of file, including PDFs, images, and other non-HTML documents.
What It Can Do
- noindex: Prevent the page from appearing in search results
- nofollow: Tell crawlers not to follow links on the page
- noarchive: Prevent Google from showing a cached copy
- nosnippet: Stop Google from showing a text snippet or thumbnail in results
When to Use X-Robots-Tag Instead of Meta Robots
The meta robots tag only works on HTML pages. If you want to control indexing of a PDF, image file, or other non-HTML resource, the X-Robots-Tag is the only option. You configure it at the server level (Apache or Nginx) or via PHP headers.
Setting It Up on WordPress
For standard pages and posts, the meta robots tag via your SEO plugin is simpler. Use X-Robots-Tag when you have PDFs, downloads, or files you want to exclude from search. Check your headers using Chrome DevTools or Google’s Rich Results Test to confirm the tag is applied correctly.
Related: Robots.txt
