Sometimes, your web site might contain pages that you do not want to see in Google’s search results. Some people block Google without knowing it and others don’t know how to exclude individual pages. This week’s article shows you how to prevent Google from indexing pages of your website.
1. Block pages and directories with the robots.txt file
The disallow directive of the robots.txt file is an easy way to exclude single files or whole directories from indexing. To exclude individual files, add this to your robots.txt file:
User-agent: *
Disallow: /directory/name-of-file.html
To exclude whole directories, use this:
User-agent: *
Disallow: /first-directory/
Disallow: /second-directory/
Note that your website visitors still can see the pages that you exclude in the robots.txt file.
2. Block individual pages with the meta robots noindex tag
The meta robots noindex tag enables you to tell search engine robots that a particular page should not be indexed. To exclude a web page from the search results, add the following code in the <head> section of a web page:
In this case, search engines won’t index the page and they also won’t follow the links on the page. If you want search engines to follow the links on the page, use this tag:
The meta robots noindex tag only influences search engine robots. Regular visitors of your website still can see the pages.
3. Block pages with the correct server header status
The server header status code enables you to send real website visitors and search engine robots to different places on your website. A web page usually has a “200 OK” status code. For example, you can use these server status codes:
- 301 moved permanently: this request and all future requests should be sent to a new URL.
- 403 forbidden: the server refuses to respond to the request.
4. Password protect your web pages
If you password protect your pages, only visitors who know the password will be able to view the content.
Search engine robots won’t be able to access the pages. Password protected pages can have a negative influence on the user experience so you should thoroughly test this. Details on how to password protect a page can be found here.
5. Use cookies and JavaScript to present your content
Cookies and JavaScript can also help you to keep search engine robots away from your door. For example, you can hide content by making it only accessible to user agents that accept cookies.
You can also use very complex
JavaScripts to execute your content. Most search engine robots do not
execute complex JavaScript code.
Blocking Google can be helpful for some pages. In general, you want Google to index page pages. The tools in SEOprofiler help you to make sure that Google and other search engines can index your web pages correctly.
If you haven’t done it yet, create your SEOprofiler account now:
Please tell your friends and colleagues about SEOprofiler and click one of the following buttons: