Official Google answer: How can Googlebot crawl and index pages that don't have any links to them on my website?

Question:

How can Googlebot crawl and index pages that don't have any links to them on my website? I find each day two or three pages in the index that don't have any links to them on my site. The pages are generated by the search field of my website.

Google's Matt Cutts answers:

Summary of the answer:

The summary of the answer is available to paid members. Become a member now. It's fast, easy and risk-free.

Question	Views
The Sitemap.xml file states there are 10,000 URLs but only 1500 have been indexed. After numerous crawls it does not appear Google is going to index these additional detail pages. What can I do to get Google to index my unique and current detail pages.	917
If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using Meta Robots noindex,nofollow a better alternative?	874
Can I use robots.txt to optimize Googlebot's crawl?	704
What are Google's plans for indexing the deep web?	788
Does Googlebot use inference when spidering - having crawled site.com/article/page1.htm and /page2.htm, can it guess at the existence of a /page3.htm and crawl it?	694
Is there a limit to the number of pages that Google will index from one site?	907
HTML sitemap vs. XML sitemap. Which one is yummy for Google search engine spider?	949
Now that Google can crawl JavaScript links, what is going to happen with all those paid links that were behind JavaScript code?	855
Last year, one of my client's web servers when down for over a day. Would this have affected the site's PageRank at all?	748
Is it a good thing to put 'nofollow' in links to a disclaimer, privacy statement and other pages like that with the internal PageRank in mind?	891

Question

Views

The Sitemap.xml file states there are 10,000 URLs but only 1500 have been indexed. After numerous crawls it does not appear Google is going to index these additional detail pages. What can I do to get Google to index my unique and current detail pages.

917

If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using Meta Robots noindex,nofollow a better alternative?

874

Can I use robots.txt to optimize Googlebot's crawl?

704

What are Google's plans for indexing the deep web?

788

Does Googlebot use inference when spidering - having crawled site.com/article/page1.htm and /page2.htm, can it guess at the existence of a /page3.htm and crawl it?

694