One of the most underrated concepts in SEO is that of a crawl budget. This article will look at what it is and why it is such an important subject, especially for very large websites with tens of thousands of pages.
In a nutshell, the crawl budget is the number of webpages Google’s spiders crawl on your website, on any given day. It depends on the size of a website, the number of errors Google encounters on a website and the number of links to the website.
Google's bots are usually busy and occupied with trying to access millions of web pages. In fact, the entire SEO domain is on its toes to crave attention from crawlers. SEO specialists want bots to crawl as many of their web pages as possible, to ensure more and more pages are indexed and ranked.
- We've built a list of the best WordPress SEO plugins out there
- Here's our list of the best SEO keyword research tools right now
- Check out our list of the best onpage SEO tools available
Julia Nesterets is founder of SEO crawler Jetoctopus
But the web is a vast and massive universe of pages and other online assets like JavaScript and CSS files and mobile page variants etc. Hence, it’s next to impossible for search engine bots to crawl and index everything. At the same time, search engines need to keep their indexes updated to include all the important content.
Search engines do not enjoy unlimited resources; hence, they need to prioritize their crawling efforts. They need to determine:
- How to prioritize web pages over the other
- What content to crawl (and what to ignore)
- Whether to recrawl certain pages often or never go back to them
These factors define the way search engines access and index online content. That’s where crawl budget and its optimization come into play.
Crawl budget is the number of pages the bots crawl and index within a given time frame. If search engines fail to crawl your page, it won’t get ranked in the SERPs. Which is to say, if the number of web pages exceeds your crawl budget, you’ll have more pages that aren’t crawled and indexed.
Assigning a crawl budget helps search bots crawl your website efficiently and therefore, boosting your SEO efforts. It’s the search engine’s way to divide attention among the millions of pages available on the web.
Thus, crawl budget optimization can ensure that the most critical content on your site is crawled and indexed.
Google explains that most websites don’t have to worry about the crawl budget. However, if a website is rather big, spiders need to prioritize what to crawl and when. Plus, they have to determine how many resources the server hosting the website can allocate for crawling.
Several factors like low-value URLs, broken or redirected links, duplicate content, wrong indexation management issues, broken pages, site speed issues, hreflang tag issues, and overuse of AMP pages, among others, can affect your crawl budget. Managing these factors will help users and crawlers access your most critical content easily and save the crawl budget from going to waste.
Besides, it’s critical to monitor how the crawlers visit your site and access content on it. Google Search Console can offer you useful information on your site’s stance in the index and the search performance. You will also find a Crawl Stats report in the Legacy tools section that shows the bot’s activity on your site over the past 90 days.
Also, server log file analysis can tell you exactly when the crawlers visit your site and the pages they visit often. Automated SEO crawlers and log analyzers can comb through your log files to find broken links and errors that bots have encountered when crawling your site. Further, the tool can audit your redirects and optimize your crawl budget to ensure that the bots crawl and index as many important pages as possible.
Wasting or failing to optimize your crawl budget equates to hurting your SEO performance. Pay special attention to crawl budget if:
- You own a huge website (especially an ecommerce site with 10K+ pages)
- You just added new content or web pages
- Your site has many redirects and redirect chains (as they eat the crawl budget)
- Your web hosting is slow
What about SEO pruning?
Google algorithms are trained to prioritize quality over quantity. Hence, it’s wise to prune or cut off the underperforming webpages, thus optimizing the crawl budget and improving your domain quality score and UX.
The process of removing obsolete and low-performing web pages or content from Google’s indexing is called SEO pruning. However, it doesn’t necessarily involve deleting these pages from a website (though at times, it may seem to be the best option!).
- Here's our list of the best rank checking tools right now