Google Set to Say Goodbye to Its Crawl Rate Tool in 2024

Google’s crawl rate limiter tool is set to become obsolete on January 8, 2024. The search engine will be removing the tool in increments from the Google Search Console. Google has improved its crawling logic to such a degree the tool isn’t useful anymore. There are also other tools publishers can use.

The crawl rate limiter lets the user communicate to Google that it should reduce its crawl rate. The tool is within the Google Search Console’s legacy version. Google was against limiting the crawl rate and has made this recommendation known. The system does limit the crawl rate. But only if there are server load problems caused by Googlebot hitting the server.

Google analyst Gary Illyes hinted that the tool deprecating was inevitable. He noted that the improvements made to Google’s crawling logic have rendered it obsolete. He also said there are other tools open to publishers.

Illyes explained that Googlebot reacts to how the site responds to HTTP requests. For instance, too many HTTP 500 status code returns will see Googlebot slowing down its crawl. Googlebot also slows down if request response times get longer.

The rate limiter tool within the legacy Search Console is slower. It takes more than a day for the new crawling limits to take effect. It’s also rare for site owners to use the tool. Those who did use it often set it to the smallest crawling speed.

Google is set to lower the crawling speed. This is a reaction to the tool’s deprecation. It allows the Google search updates to honor the settings site owners have set. It does this without the crawlers wasting the site’s bandwidth.