In this post
To keep a page out of Google, a robots.txt file is not the right tool. According to Google's documentation, robots.txt is mainly for managing crawler traffic, not for keeping a page out of search results. Use a noindex rule or password protection instead.
What robots.txt actually does
A robots.txt file sits at the root of your site and tells crawlers which areas they should not request. Google's introduction to robots.txt says it is used mainly to avoid overloading a site with requests, and that it is not a mechanism for keeping a web page out of Google.
The reason is simple. If a page is blocked in robots.txt, Google will not crawl it. But Google may still find and index the bare URL if other sites link to it. You can end up with a listing that has no description, which is the opposite of what you wanted.
What noindex does
A noindex rule tells Google not to show a page in search results. You add it as a meta tag in the page's HTML, or as an HTTP response header. Google explains the details in its guidance on blocking search indexing with noindex.
There is one catch that catches many site owners out. Google can only see a noindex rule if it is allowed to crawl the page. If you block the page in robots.txt and add noindex, Google never reads the noindex rule, because it never fetches the page.
How to keep a page out of Google properly
Pick the method by what you are trying to achieve. If the page is for the public but not for search, such as a thank-you page, use noindex and leave it crawlable. If the page is private, such as a client area or a draft, put it behind a login so nobody can open it without permission. If the goal is only to reduce crawler load on large, low-value sections of a site, robots.txt is the better fit. Mixing the goals is where most mistakes come from.
Which one should you use?
| What you want | Use |
|---|---|
| Keep a page out of Google results | noindex, or password protection |
| Stop crawlers wasting time on low-value URLs | robots.txt |
| Keep private content truly private | Password protection or login |
| Remove a page that no longer exists | Delete it and return a proper "not found" or "gone" response |
Do not put noindex in robots.txt
Some old guides suggest adding a noindex line to robots.txt. Google has said it does not support this. A 2019 post on the Google Search Central blog about unsupported rules in robots.txt explains that Google retired that behaviour. Use the meta tag or header instead.
Common mistakes on small business sites
Blocking the whole site by accident
A developer may block everything while a site is being built. If that file goes live with the new site, search engines are told to stay away. Always check robots.txt after a launch by visiting yourdomain.com/robots.txt.
Leaving "discourage search engines" ticked
WordPress has a setting under Settings, then Reading, that asks search engines not to index the site. It is handy on a staging copy. It should be off on your live site.
Using robots.txt to hide private pages
Robots.txt is a public file. Anyone can read it, so listing a private folder there points people towards it. Private pages need a login.
How to check a page is set up correctly
- Open the page and view its source, then search for "noindex". If you find it in a robots meta tag, the page is asking not to be indexed.
- Check that the page is not blocked in robots.txt, so Google can read the tag.
- Use the URL Inspection tool in Google Search Console to see how Google treats the page.
Changes are not instant. Google needs to crawl the page again before a noindex rule takes effect, and that can take some time.
What about pages already in Google?
If a page is already showing in results and you want it gone, do not block it in robots.txt first. Add noindex, leave the page crawlable, and give Google time to see the change. Once it has dropped out, you can decide whether any other step is needed. If the page was published by mistake and holds sensitive information, remove it from the site or protect it with a login straight away, then ask your developer to help.
Which pages are worth keeping out of search?
Think about thin or duplicate pages, such as thank-you pages, internal search results and login pages. Your key service and product pages should always be open to search engines.
Guidance can change, so check Google's pages linked above before making big changes. If you would rather have someone review your site, our SEO optimisation team can audit it, or you can send us a message.
Common questions
Can I use both robots.txt and noindex on the same page?
Not for the same goal. If robots.txt blocks the page, Google cannot read the noindex rule, so the page may still appear.
How long does noindex take to work?
Google has to crawl the page again before it sees the rule, so it is not instant.