Technical SEO · 7 min read
Robots.txt says allowed. Why is the page not indexed?
An allowed robots rule only answers whether the pasted rule blocks a crawler. It does not establish that the live page can be fetched, is indexable, or will be selected for search.
Check the layers in order
Start with the exact URL, including its scheme, hostname and path. A robots file applies to its own origin. A rule pasted from www.example.com does not establish the rule for example.com. Keep the crawler token and test time in your notes.
Use the live robots file and URL Inspection for Google-specific evidence. A text checker cannot observe server errors, redirects, firewall challenges or a different file being served to the crawler.
A diagnosis table, not an indexing promise
| Evidence | What it answers | Next check |
|---|---|---|
| Pasted robots allows the URL | The selected rules permit crawling | Confirm the live origin serves that file |
| HTTP error or login challenge | The content may not be retrievable | Inspect response, redirect destination and access controls |
| noindex in HTML or HTTP header | The page requests exclusion from indexing | Remove only if the page is intended to be public |
| Canonical points elsewhere | The page signals a preferred duplicate URL | Compare content and the selected canonical in URL Inspection |
| Accessible, indexable public page | Basic technical conditions may be satisfied | Review quality, duplication and Search Console evidence |
Keep private pages private
Robots.txt is public and is not an access-control mechanism. Do not publish account paths or confidential filenames as a security strategy. Authentication and permissions must protect private pages and data.
A crawler blocked by robots.txt may be unable to see a page’s noindex directive. Decide whether the page should be accessible and excluded, or private behind authentication, before changing crawl rules. Do not open private account pages merely to make an indexing tool return a positive result.
A canonical is a signal, not a command
A rel=canonical link helps indicate your preferred URL among duplicates. Google may select a different canonical. Compare both URLs and their redirects, internal links, content and sitemap entries before assuming the tag has solved duplication.
A submitted sitemap helps discovery. It does not guarantee indexing. An allowed robots result and a submitted sitemap are separate pieces of evidence, not a combined pass certificate.
Use the checker for the question it can answer
- Paste the actual robots text from the tested origin.
- Enter the full URL and crawler product token.
- Read the selected group and winning line, then note unsupported lines.
- Check live delivery, noindex and canonical separately.
- Record the indexing evidence and the next review date without promising a deadline.
Sources and editorial method
Written and maintained by Mapseen. Guidance is linked to primary documentation. Examples are fictional; they explain a calculation or decision rather than report customer outcomes. Provider behaviour and report availability can change.
Found a mistake? Open an email draft to Mapseen with the guide URL and the detail to review.