Crawlers use the longest-matching rule. If both 'Allow: /blog/post' and 'Disallow: /blog/' match a URL, the more specific (longer) Allow wins. When two rules are equally specific, Google's crawler defaults to allowing the URL. This tool replicates that exact precedence logic.

A crawler obeys the most specific User-agent group that names it. If your robots.txt has a 'User-agent: Googlebot' block and a 'User-agent: *' block, Googlebot follows only the Googlebot block and ignores the wildcard entirely. This is a common source of confusion.

No — this is a critical misconception. Disallow only stops crawling, not indexing. A blocked URL can still appear in search results (without a description) if other sites link to it. To remove a page from the index, use a noindex meta tag or X-Robots-Tag header — and don't block it in robots.txt, or Google can't see the noindex.

The asterisk (*) matches any sequence of characters, and the dollar sign ($) anchors the end of a URL. So 'Disallow: /*.pdf$' blocks all URLs ending in .pdf. Wildcard support is standard in Google and Bing but not guaranteed in every crawler. This tool simulates Google's interpretation.

Usually because a Disallow rule is more specific (longer). Crawlers weigh rules by path length, not order. If 'Disallow: /private/secret' is longer than 'Allow: /private/', the Disallow wins for that URL. Lengthen your Allow rule to make it more specific than the competing Disallow.

Yes. 'Disallow: /Admin' will not block '/admin' — they're treated as different paths. URLs and robots.txt paths are case sensitive (except the domain). Always match the exact casing your site uses. This tool performs case-sensitive matching to mirror real crawler behavior.

This tool tests one user-agent against one URL at a time so you can see exactly which rule matched and why. To audit how 20+ search and AI bots are treated across your live robots.txt, use our Robots.txt Tester, which fetches your file and runs all major crawlers automatically.