What the meta robots checker looks at
The meta robots tag is a small line in the head of a page that tells search engines how to treat that page: whether they may show it in results, whether they may follow its links, and a handful of finer behaviours. It is invisible to visitors but enormously powerful for search, because a single word in it can decide whether a page ever appears in Google at all. This tool fetches a page, reads the meta robots tag from its head, and also reads the X-Robots-Tag that can be sent in the HTTP response, then decodes what those directives actually mean in plain terms. Instead of squinting at page source, you get a clear readout of whether the page is set to be indexed and its links followed, or quietly told to disappear.
Checking both places, the page head and the HTTP header, matters because the same instruction can come from either, and they combine. A page can look perfectly indexable in its HTML while a header sent by the server overrides it with a noindex you never see in the source. This checker surfaces both so you get the real, combined picture a search engine would act on, not just the half you happened to look at.
Noindex and nofollow: what each word does
The two directives that matter most are noindex and nofollow, and they do completely different jobs that people constantly mix up. Noindex tells a search engine not to include this page in its results. The crawler may still fetch and read the page, but it will keep it out of the index, so it never shows up when someone searches. This is the correct tool for pages you want crawled but never surfaced, such as internal search results, thank-you pages after a form, login screens, or thin archive pages that add no value in search.
Nofollow, when placed in the page-level meta robots tag, tells the search engine not to follow any of the links on the page, meaning it will not pass ranking signals through them or use them to discover new URLs. The opposite defaults, index and follow, are what a normal page should be, and they are also the assumed behaviour when no meta robots tag exists at all. That last point is crucial: a page with no meta robots tag is treated as indexable and followable by default, so the absence of the tag is usually fine. The danger is almost always an unwanted noindex that is present, not a missing tag.
The directives beyond noindex and nofollow
The meta robots tag carries more than the famous two. Noarchive tells search engines not to store a cached copy of the page. Nosnippet tells them not to show any text snippet or video preview in results, which can hurt click-through because your listing loses its descriptive blurb. There are also directives that cap how long a text snippet may be, limit image preview sizes, and limit video preview lengths, giving you fine control over how your listing looks. Another, noimageindex, asks that the images on the page not be indexed in image search.
These finer directives are less commonly needed, but when they appear unexpectedly they can quietly degrade how your page performs. A nosnippet you did not intend strips the description from your search listing. An overly tight snippet limit cuts your message short. The checker decodes any of these it finds so you understand not just whether a page is indexed, but exactly how it has been told to present itself, which is the kind of detail that is easy to set once and forget about for years.
Reading the verdict on your page
The most important thing to read first is the indexing verdict. If the checker reports the page as indexable, with index and follow either explicitly set or simply implied by the absence of any restriction, then nothing in the meta robots layer is stopping the page from ranking. If it reports noindex, treat that as a red alert on any page you actually want in search, because that one word is almost certainly the reason the page is not showing up. The whole value of the tool is catching an unintended noindex before it costs you weeks of lost visibility.
Next, read whether the directive came from the HTML tag, the HTTP header, or both, because that tells you where to go to fix it. A noindex in the HTML lives in your template or page settings, often controlled by a CMS toggle or an SEO plugin. A noindex in the header is set at the server or application level and is invisible in page source, so people hunt fruitlessly through their HTML while the real cause sits in a server config. Knowing the source turns a frustrating mystery into a targeted, one-place fix.
Targeting specific bots with the tag
A subtlety many people miss is that the meta robots tag can be aimed at all crawlers or at one named crawler. The generic form applies to every robot, while a named form applies only to the bot it names, such as a directive aimed specifically at Google's crawler. This means a page can carry a general index instruction and a more specific instruction for one bot at the same time, with the named one taking precedence for that bot. It is an easy way to accidentally create a page that behaves one way for most crawlers and another way for a particular search engine.
When the checker reports the directives, note whether any of them are bot-specific, because a targeted noindex aimed at a single search engine will block that engine while leaving the page visible elsewhere, which can be baffling if you are only watching one set of rankings. The same mechanism is increasingly relevant for AI-oriented crawlers, where you might want different treatment for traditional search versus AI systems. Seeing exactly which bot each directive names lets you confirm the page behaves the way you intend across every engine that matters to you.
Common mistakes that hide pages by accident
By far the most common and most painful mistake is a site-wide noindex left over from development. Many platforms have a setting that discourages search engines while you build, and it works by adding a noindex to every page. Flip the site live without unchecking that box and the entire site tells search engines to stay out, so it never ranks no matter how good the content is. Running a few key pages through the checker right after launch catches this immediately, and it is the first thing to suspect whenever a freshly launched site fails to appear in search.
Other recurring errors include leaving noindex on pages you later decided you want indexed, such as a blog category that has grown valuable; setting noindex but also blocking the page in robots.txt, so the crawler never even reads the noindex and the page can stick around in results anyway; and contradicting yourself by setting an indexable directive in one place and a noindex in another, where the more restrictive instruction generally wins. The checker exposes each of these by showing the true combined directive, so you can resolve the conflict instead of wondering why a page behaves the opposite of how you expected.
How indexing control fits modern and AI search
Controlling what gets indexed is more strategic now than it used to be. Search engines reward sites that present a clean, intentional set of pages rather than a sprawl of thin, duplicate, or low-value URLs. Using noindex deliberately to keep the junk out, the endless filter combinations, the empty tag pages, the printer-friendly duplicates, while leaving your genuinely useful pages fully indexable, helps concentrate your site's standing on the pages that deserve it. A page that should never have been in the index can dilute the perception of your whole site.
The same logic extends to AI-driven search and answer engines, which increasingly draw on the open, indexable web. A page set to noindex is telling those systems it does not want to be part of search visibility, which can keep it out of the pool of content AI tools draw from. So the meta robots tag has quietly become part of your AI-search strategy too: the pages you want cited and surfaced by AI need to be indexable, and the checker is how you confirm they are, rather than assuming and being surprised later.
What to do after you run the check
Start by sorting the outcome into intended and unintended. If a page you want in search reads as noindex, find the source the checker identified, the HTML tag in your CMS or the header in your server config, remove the restriction, and re-check the page to confirm it now reads as indexable. If a page you wanted hidden reads as indexable, add the noindex in the right place and verify it took effect. Either way, re-running the check after the change is what proves the fix actually landed rather than just assuming it did.
Make a few principles routine. Audit your most important pages after any launch, redesign, or platform migration, since those are exactly the moments stray noindex tags slip in. Keep noindex for pages crawlers should read but never show, and never pair it with a robots.txt crawl block on the same page, because the two cancel each other out. And remember that no tag at all means indexable, so you only ever need to add a meta robots tag when you want to change a page's default behaviour. A page whose indexing status you have actually verified is one less silent failure waiting to drain your traffic.