Google "polished and clarified" their guide to optimizing your crawl budget, mostly to improve clarity. Barry Schwartz combed through the changes and highlighted some good ones:

> Here are some new lines added:

> • "Every site starts with the same default, conservative crawl capacity limit. If there is demand to crawl more and the site remains healthy, Google's systems will automatically adjust this limit over time."

> • "For Googlebot, demand varies based on a site's size, update frequency, page quality, and relevance, compared to other sites. The primary factors you can influence are."

> • "While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others."

> • "Improve loading speed: Optimize your server response times and resources to make pages load faster."

> • "Use HTTP caching: Support 304 (Not Modified) HTTP status codes. If a page hasn't changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources."

That second bullet point is worth reading the whole section:

• *Perceived inventory:*Without guidance from you, Google tries to crawl all or most of the URLs that it knows about on your site. If many of these URLs are duplicates, or you don't want them crawled for some other reason (removed, unimportant, and so on), this wastes a lot of Google crawling time on your site. This is the factor that you can positively control the most.
• *Popularity:*URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.
• *Staleness:*Our systems want to recrawl documents frequently enough to pick up any changes.

Additionally, site-wide events like site moves may trigger an increase in crawl demand in order to reprocess the content under the new URLs.