Search engine optimisation is the process of getting your content ranked on search engines like Google, Bing, and Yahoo. To someone who doesn’t understand search engine optimisation, the process of getting your website to appear to your customers is confusing and filled with complicated jargon.

A commonly misunderstood term is crawl budget. Crawl budget is how a search engine looks through a website in the most efficient way possible. Each search engine has their own processes to “crawl” websites, thought process on choosing how to explore a website.

The search engines that are used the most are Google, Bing, and Yahoo, but regional search engines also exist in China (Baidu) and Russia (Yandex) that also make up parts of the search engine market. When we think about SEO, we think about optimising for Google, as over 90% of web searches are done through their search engine (StatCounter).Each search engine has their own crawler, which finds all the links on the website and adds them to a queue. When considering your crawl budget, the goal is to allow this crawler easy access to as many pages on your website as quickly and efficiently as possible.In this article, we will cover:

How do crawl budgets work?
Issues affecting crawl budget
Why do crawl budgets matter?
How to optimise crawl budget
Finding help with your crawl budget

How Do Crawl Budgets Work?

Crawl budgets were designed to save the resources of a search company. Although Google is worth $1 Trillion USD, the company must index over 4 billion web pages and track them for changes over time. Once your web page is indexed, it has the ability show up in Google’s search results.

There are two main methods Google uses to assign your website a crawl budget.

Crawl limit, or host load, is how much crawling a website has capacity for. This is affected by the web host you use, as well as the preferences of the website owner. Some websites decide they do not want to be found on Google; in which case they add a file to their website (named robots.txt) that contains aNoIndex directive. This means that your website will never rank.

Crawl demand, or crawl scheduling, finds the best pages on a website and decides that they will update the ranking of those pages more. This is affected by the quality of the content on a web page, as well as how popular the page is. If a page is updated, this can trigger another crawl also.

Although SEO relies on well written content to bring traffic onto a website, if Google can’t reach the content due to misconfigurations on the website or the hosting server, then your website won’t be accessible via Google.

Issues Affecting Crawl Budget

There are several crawl limit issues that impact Google’s ability to index your web pages:

Server overcapacity – If your website is receiving too many visitors, it can overload your hosting server and timeout the site visitor.
Shared webhosting – If you host your website through shared cloud hosting, many websites that you don’t have control over will use the same crawl limit. A small website is less impacted by this, but as more pages are added to a website the limit will be beneath its required level.
Manual limiting – You can limit the amount of crawling Google can do by visiting Google Search Console. Sometimes the webmaster, or the website owner, will lower the limit due to the use of bandwidth that crawling can use. You should leave Google to automaticallyallocate your crawl limit, as optimisation for bandwidth use is done as part of their system.
Google’s servers – Google runs their own server centres, but like any web server they can be taken offline or be overloaded. In this case, every website will have a temporary drop in crawl limit to avoid the remaining servers being overloaded.

Crawl demand is how Google keeps the best pages on the internet up to date in its search engine. Pages which have popular articles, which rank in Google for many different searches, are crawled much more than other pages. Likewise, if a page isn’t updated frequently, such as a privacy policy page, the likelihood Google would re-crawl it is low.

Why Do Crawl Budgets Matter?

Your company website has been created with the customer in mind. They can access all the pages on your website just by clicking around and finding articles that appeal to them.

Google can’t do this. They can’t see how you have organised your website, they can only find the content and links on the page. This means that your website might be easy to navigate for its viewers, but if the linking on a site is inefficient the crawl budget decreases.

When crawl budgets decrease, less pages can be viewed on your website. This means that they could miss blog posts that are less well linked to around your site.

This affect can mean posts go stale quickly without crawl budget making sure that they receive their fair share of visitors. It becomes much more of a problem once a website hits thousands of pages, as the likelihood many pages are difficult to access increases.

How To Optimise Your Crawl Budget

There are many ways to stop wasting crawl budget. Common fixes include:

Improving URL taxonomy – URLs with few categories become easier to find directly, minimising the number of steps a crawler must take to visit the URL.
Improve the hosting server – if the website isn’t loading or is loading slowly, migrating to a dedicated server can be a worthwhile investment.
Avoid content cannibalism – If a page has similar or duplicate content, Google will consider both pages the same and rank neither.
Create high quality content – SEO relies on well written content that visitors want to read.
Choose to index pages – Making sure that most pages on a website are indexed means that the crawler is not slowed trying to find alternate URL routes around them.
Broken links – If an URL is no longer in use, it should be redirected to a working URL covering the same topic.
Improve the internal link structure – No page should be more than 5 clicks away from the homepage so adding category pages can make a website easier to navigate for crawlers.

There are many ways in which a crawl budget can be negatively impacted, but these common solutions are usually what is causing crawl limits to be reached on a website that is still relatively small. By finding these solutions early, it can solve major problems down the line should the website grow larger.

Finding Help With Your Crawl Budget

With so many individual factors affecting crawl rate, it’s important to find a solution that fixes any issues quickly. Our team of SEO experts at Maxweb Solutions can identify the specific issues that are causing you to waste crawl budget and provide a comprehensive fix for the problem.

Working early to fix crawl budget issues allows your website to grow quicker, as your website will not hit the same issues as the website grows. This means your customers will be satisfied being able to find the information they are looking for.

To find out more about crawl budgets, call us at 0151 652 4777 or send us an email at info@maxwebsolutions.co.uk.

Posted on Thursday, March 19th, 2020 in SEO.