Project menu

Official Google answers to common SEO questions

Enter your question: Please wait...Please wait...

Question:

If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using Meta Robots noindex,nofollow a better alternative?

Google's Matt Cutts answers:

Official Google Answer

Summary of the answer:

The summary of the answer is available to paid members. Become a member now. It's fast, easy and risk-free.

Answered: 10 March 2010. Views: 1,021.

Related questions:

Question Views
The Sitemap.xml file states there are 10,000 URLs but only 1500 have been indexed. After numerous crawls it does not appear Google is going to index these additional detail pages. What can I do to get Google to index my unique and current detail pages. 1,086
Should a "Sale Page" be in a robots.txt file to avoid duplicate content? 1,046
Can I use robots.txt to optimize Googlebot's crawl? 869
Uncrawled URLs in search results 1,335
Does Googlebot use inference when spidering - having crawled site.com/article/page1.htm and /page2.htm, can it guess at the existence of a /page3.htm and crawl it? 846
If I externalize all CSS style definitions and JavaScript scripts and disallow all user agents from accessing these external files (via robots.txt), would this cause problems for Googlebot? 900
What is the nofollow equivalent for JavaScript links/redirections (now that you follow those too)? 820
Now that Google can crawl JavaScript links, what is going to happen with all those paid links that were behind JavaScript code? 1,016
How can Googlebot crawl and index pages that don't have any links to them on my website? 1,012
We still have old content in the index. We block them via robots.txt, use 404 and delete via Webmaster Tools, but Google still keeps it. What can we do to quickly delete content from the index? 984