On Twitter, Google’s John Mueller talked about the importance of crawl budgets. Do you have to optimize the crawl budget of your website, or can you ignore this topic?
What is the crawl budget?
Your crawl budget is the number of pages that Google will crawl on your site. Your crawl budget is determined by the size of your website, the number of links that point to your website, and the number of errors that Google encounters on your site.
What is crawl budget optimization?
If your website has more pages than the crawl budget allows then some pages of your website might not be indexed by Google. For that reason, some webmasters try to optimize the crawl budgets by blocking less important pages of their website in the robots.txt file of their site.
By blocking the less important files, Google can index more of the important pages on a website.
What Google says about crawl budget optimization
Google’s John Mueller said on Twitter that crawl budget optimization is an interesting topic. However, it is only relevant if you have a website with billions of URLs. Crawl budget optimization is not important for the average website owner:
IMO crawl-budget is over-rated. Most sites never need to worry about this. It's an interesting topic, and if you're crawling the web or running a multi-billion-URL site, it's important, but for the average site owner less so.
— John ☆.o(≧▽≦)o.☆ (@JohnMu) 30 May 2018
Removing unimportant pages from your website is still a good idea. Everything that helps you website visitors to find relevant content more easily is recommended:
I think it's worth being clear about it all though. Removing 25 useless pages is great for a leaner site, and can help users from getting lost there, but it's not a crawl-budget question. Would people only do it for a SEO bonus? How can we help you to help them?
— John ☆.o(≧▽≦)o.☆ (@JohnMu) 30 May 2018
How to check the pages that search engines can find on your website
Check your website with the website audit tool in SEOprofiler to find out which pages search engines can find on your website. The website audit tool will show you errors that prevent search engines from indexing your pages. You will also find out how many clicks people need to find particular pages on your website, you will find out if your server is secure, and much more. You can create your SEOprofiler account here:
