Google slow to crawl website (Search Console)

[phenomlab]

There are times when Google can be remarkably slow in crawling your site - particularly if you had issues previously, and you need to tell them you’ve fixed it.

You might see something like this when requesting validation

These have all been fixed, but to “speed things up”, you can

1. Remove and re-submit the sitemap.xml file
2. Send a ping to Google to ask them to crawl again (see below)

http://www.google.com/webmasters/sitemaps/ping?sitemap=http://example.com/sitemap.xml

Some more useful tips can be found here

https://superdevresources.com/ping-sitemap/

Obviously, you’d substitute the sitemap= part to match your own site.

[phenomlab]

It just goes to show that despite being the behemoth that Google is, it’s crawl rate is ridiculously slow. Bing and Yandex have adopted “IndexNow” - see below article, which is well worth the read in itself

https://www.searchenginejournal.com/indexnow-what-we-know/439679/#close

DuckDuckGo are reportedly still considering adopting IndexNow, but given they pull sources from over 400 other entities (of which Bing and Yandex are amongst), their content also updates much quicker (provided you are using Bing and Yandex Webmaster tools, which I’d recommend) - see below example

https://duckduckgo.com/?hps=1&q=site%3Asudonix.org&ia=web

And so, Google, my point is this. You’re the largest player on this planet, yet also the slowest at crawling sites? Come on!

Evidently, Google have been testing since 2021, but are yet to adopt

https://www.searchenginejournal.com/google-will-be-testing-indexnow/426602/

[phenomlab]

It seems that in order for Google to crawl your site, you must include a forward slash at the end of the domain name when you add it (if you are adding it as the site itself, and not at domain level) - who knew? That isn’t documented anywhere!

[DownPW]

I don’t know

[phenomlab]

@DownPW It’s a completely new one on me also! You don’t have to do this with Bing or Yandex

[phenomlab]

Seems like Google is finally crawling this site. And, “crawling” in the sense that it’s still extremely slow …