Indexability Checker: can Google crawl and index this URL?
Enter a URL and see exactly what would stop Google from indexing it: the HTTP status, the robots.txt rule that matches, meta robots and X-Robots-Tag directives, canonical tags, snippet controls, and whether the URL appears in a sitemap. Every finding shows the evidence and how to fix it.
We fetch the page once as HelloSEOToolsBot, plus its robots.txt and up to five sitemap locations, reading at most 1.5 MB. Public hostnames only. Results are cached for 5 minutes; nothing is stored beyond that. Indexable means crawlable and not blocked, not that the URL is in Google's index.
What each finding means
| Verdict | Indexable: the final URL returns 200 HTML, robots.txt allows Googlebot, and no noindex was found. Blocked: at least one hard stop was found (robots.txt disallow, noindex, an error status, a redirect that never completes, or a non-HTML response). Uncertain: nothing blocks it outright, but a cross-canonical, an unreadable robots.txt, or a fetch failure means we cannot say. None of these verdicts tells you whether the URL is currently in Google's index. |
|---|---|
| Fetch outcome | The initial and final HTTP status and the redirect chain. A DNS failure, timeout, or 5xx is reported as "could not fetch" and kept separate from SEO settings, because it may be temporary or specific to our crawler. |
| robots.txt | Fetched from the origin of the final URL. We evaluate the path for Googlebot and for the generic * group using longest-match rules, the same way Google does, and show the exact group and rule that matched. A missing robots.txt (4xx) means everything is allowed. A 5xx means Google treats the file as temporarily unavailable and may stop crawling the site. |
| Meta robots and googlebot | All <meta name="robots"> and <meta name="googlebot"> tags. noindex and none block indexing; nofollow stops link discovery. When both tags exist Google applies the most restrictive combination. |
| X-Robots-Tag | The HTTP header equivalent of meta robots, checked on the final response and shown for every redirect hop. It is the only way to noindex non-HTML files and a common source of accidental blocks left over from staging. |
| Canonical | How many canonical tags exist, whether the href is absolute, whether it points to this URL (self) or another one (cross), whether it uses http on an https page, and whether a Link: rel=canonical header agrees with it. A cross-canonical usually means Google will index the other URL. |
| Snippet controls | nosnippet, max-snippet, max-image-preview, max-video-preview, noarchive, notranslate, and unavailable_after. They do not block indexing but change how the result looks. A past unavailable_after date acts like noindex. |
| Sitemaps | Sitemap lines from robots.txt plus /sitemap.xml and /sitemap_index.xml. We report which exist and, for sitemaps under 200 KB, whether the final URL is listed. Listing helps discovery; it never forces indexing. |
| Conflicts | Combinations that cancel each other out: robots.txt blocking a page that carries noindex (Google cannot see the noindex), a noindex page listed in a sitemap, a blocked page listed in a sitemap, and a canonical that disagrees with the header. |
Limitations and data sources
- We fetch the page as HelloSEOToolsBot from Cloudflare's network, not as Googlebot. Sites that serve different content to Googlebot, require JavaScript to insert meta tags, or block unknown crawlers can give a different answer than Google sees. Use URL Inspection in Search Console for Google's own view.
- Only HTML is evaluated. PDFs, images, and other files are reported as not evaluated, even though Google can index some of them; use X-Robots-Tag headers and robots.txt to control those.
- robots.txt evaluation covers User-agent, Allow, Disallow, wildcards, and end anchors. Crawl-delay and other non-standard lines are ignored, as Google ignores them.
- Sitemap listing is a plain text match of the final URL. Sitemap indexes are not expanded, compressed sitemaps are not read, and files over 200 KB are reported as not checked.
- Reads are capped at 1.5 MB for the page, 200 KB for robots.txt and each sitemap, and 20 seconds in total. Private, loopback, and raw IP destinations are refused before any request is made.
- We never claim a URL is indexed. That information exists only in Google Search Console.
Frequently asked questions
- What is the difference between crawlable, indexable, and indexed?
- Crawlable means Googlebot is allowed to fetch the URL: robots.txt does not block it and the server answers. Indexable means nothing on the fetched page tells Google to keep it out: it returns 200, is HTML, and has no noindex in a meta tag or X-Robots-Tag header. Indexed means Google has actually stored the page and can show it in results. This tool can only verify the first two. A page can be perfectly indexable and still not indexed because Google has not crawled it yet, considers it a duplicate, or judged it low value.
- Why does a robots.txt block hide a noindex tag?
- A noindex directive lives inside the page, in a meta tag or a response header. If robots.txt disallows the URL, Googlebot never fetches the page, so it never reads the noindex. Google can still index the URL from links pointing to it, usually with no title or description. To remove a page from search, allow it in robots.txt and use noindex, or remove it with a 404 or 410.
- What happens when a page has more than one canonical tag?
- If the tags point to different URLs, Google ignores all of them and picks a canonical on its own, which may not be the URL you want. The usual cause is a theme, an SEO plugin, and a page builder each emitting one. Emit exactly one canonical with an absolute https URL, and make sure any Link: rel=canonical HTTP header agrees with it.
- Does a 200 OK response mean the page is indexed?
- No. A 200 response only means the server delivered the page. Google may still skip it because of a noindex tag, a canonical pointing elsewhere, a robots.txt block, duplicate content, or simply because it has not been crawled. This tool reports indexable when nothing it can see blocks indexing; it never claims a page is in the index.
- How do I confirm whether a URL is actually in Google's index?
- Use the URL Inspection tool in Google Search Console. It shows whether the URL is on Google, when it was last crawled, which canonical Google selected, and any indexing problem it found. A site: search in Google is only a rough hint and can be incomplete. Third-party tools, including this one, do not have access to Google's index and cannot confirm index status.
API
The same result is available as JSON, free, no key required, cached for 5 minutes (add &fresh=1 to bypass the cache):
GET https://helloseotools.com/api/indexability?url=https://example.com/
Response: { ok, input, finalUrl, checks[], summary: { verdict, reasons, meaning }, evidence: { hops, status, headers, robotsUrl, robotsStatus, robotsGroup, robotsRule, robotsExcerpt, metaRobots, metaGooglebot, xRobotsTag, canonical, canonicalTarget, sitemapUrls, sitemaps, snippetControls } }. Each check has id, status (pass, fail, warn, info, skipped), severity, observed, why, fix, reference, confidence, and source. Errors return { ok: false, error }.
Please keep automated use under a few requests per second. Heavy or abusive use will be rate limited.
Related tools
- WHOIS Lookup: registrar, expiry, status codes, and DNS for the domain.
- Redirect Checker: the full hop-by-hop redirect chain with status codes.
- SERP Preview: how the title and description will look in results.
- Schema Generator: build valid JSON-LD structured data.
- SEO Audit: on-page checks for a single URL in one report.
Related reading
- Google: introduction to robots.txt
- Google: how robots.txt HTTP status codes are handled
- Google: robots meta tag and X-Robots-Tag specifications
- Google: how to specify a canonical URL
- Google: build and submit a sitemap
- Google Search Console: URL Inspection tool
- Google: how Search works (crawling, indexing, serving)
- Schema.org: WebApplication