X-Robots-Tag is an HTTP response header that carries the same indexing directives as the meta robots tag (noindex, nofollow, noarchive, etc.), but at the protocol level instead of inside HTML. Because it's a header, it can control indexing for any file type — PDFs, images, videos, and other non-HTML resources that have no place to put a meta tag.

Use it when you need to control indexing for non-HTML files (like keeping PDFs out of search), when you want to apply directives at scale via server config without editing each page, or when you can't modify the HTML. For ordinary HTML pages, the meta robots tag is usually simpler and easier to audit.

Because it's invisible in the page source. A page can have perfect HTML with no meta robots tag, yet still be deindexed by an X-Robots-Tag: noindex header set globally by a CDN, reverse proxy, security plugin, or leftover staging config. Many site owners spend hours auditing HTML while the real culprit is a header they never see.

Yes. You can prefix directives with a user-agent, e.g. 'X-Robots-Tag: googlebot: noindex' applies only to Googlebot, while 'X-Robots-Tag: noindex' (no prefix) applies to all crawlers. You can also send multiple X-Robots-Tag headers for different bots. This tool lists every directive and any user-agent prefix it detects.

It's configured at the server level. On Apache, use Header set X-Robots-Tag in .htaccess or a vhost; on Nginx, use add_header X-Robots-Tag; on a CDN like Cloudflare you can add it via a response header rule or worker. To target file types (e.g. all PDFs), wrap the rule in a FilesMatch or location block.

They don't override — search engines combine all robots directives they find from every source and apply the most restrictive interpretation. If the header says noindex and the meta tag says index, the page will be noindexed. This tool checks the header in isolation; pair it with the Meta Robots Tag Checker to see the full picture.

The most common reason is that robots.txt blocks the URL, so Google never makes the request and never receives the header. Other causes: the header is set on the wrong response (e.g. only on a redirect, not the final page), a typo in the directive, or caching serving an old response. Verify the header appears on the final 200 response for the exact URL.