Robots.txt and noindex solve different problems
Separate crawl permission from indexing instructions before changing either.
Two controls, two questions
Robots.txt tells supporting crawlers which paths they may request. A noindex directive tells supporting search engines that a page should not be indexed. One controls access to a fetch; the other communicates a selection instruction. Understanding that difference prevents a common repair mistake: adding a robots block when the real goal is to remove an already known page from search.
A blocked URL can still be known through links or other discovery sources. Blocking a crawl does not reliably erase the URL from search. Conversely, a crawler usually needs to access a page to observe its noindex instruction. If you block that access immediately, the crawler may not see the very change you want it to process.
Choose the intended outcome
For private information, use proper authentication and access control. Neither a public noindex directive nor robots.txt is a security boundary. If an account page contains personal records, preventing unauthorised access comes first. Search instructions should be a supplementary decision, not the mechanism protecting the records.
For a public page that should remain accessible but not appear in search, review a noindex implementation. For a large set of low-value crawlable URL combinations, robots rules may help manage requests, but think through whether search engines already know the URLs and what discovery signals remain. The right decision depends on the page’s role, not on a universal rule to block everything unimportant.
Inspect every delivery layer
An HTML meta tag is only one place to look. X-Robots-Tag response headers can also carry indexing directives, including for documents that have no HTML head. Hosting configuration, application middleware and publishing templates can each add rules. A staging noindex header copied to production can affect many otherwise healthy pages.
Check the deployed response rather than relying on the editor setting. Read the exact URL without a session and inspect both headers and returned HTML. If the site is rendered with JavaScript, compare initial output and rendered output where needed. A simple HTML audit is useful evidence but not a complete browser-rendering analysis.
Repair with a before-and-after record
First decide which URLs should be eligible for search. Record the current directive and where it originates. Make the narrowest template or configuration change, redeploy and fetch again. For a large site, inspect multiple templates so you do not remove an intentional exclusion from private or duplicate areas.
Noor Indexer checks sampled HTML meta directives and X-Robots-Tag headers. Its robots interpretation is conservative and limited to wildcard rules, so engine-specific groups still require review. Treat a warning as a prompt to inspect intent. A page excluded on purpose is not broken, and a page with no visible exclusion is not necessarily indexed.
Technical references
Primary documentation checked for this launch. Examples are illustrative unless explicitly described otherwise.
Continue reading
04A canonical tag is a decision about which page matters
Trace unexpected canonicals back to templates, duplicates and real publishing intent.
The indexing handover a freelance website client can use
An illustrative handover that explains ownership, evidence and the next check.