Am I deferred from the Google search hub?

[DownPW]

Hello @phenomlab

It would seem that we are no longer referenced at Google whereas we were before.

How can I remedy this?

Things to do in NodeBB? Cloudflare? nginx?

I admit I don’t understand much about referencing except that search engines come to index a specific file on the server: sitemap.xml, robot.txt

How does NodeBB handle this?

Knowing that I did nothing specific to be referenced with them before.

Other search engines like Duck Duck go, bing, yahoo seem to work.

[phenomlab]

@DownPW Thanks. The missing Google Search Console account alone is your #1 issue here and you should add the site as soon as possible. There is nothing wrong with the robots.txt or sitemap.xml so once you’ve completed the step above, things should start to move.

However, Google can take ages to index a site (as I know first hand when moving from .com to .org here). Bing has a great feature called IndexNow which works with Cloudflare, and can parse a site as soon as it changes to crawl it. Google sadly has not yet adopted this feature despite promising they would as it’s now an open standard.

https://www.indexnow.org/

I’m not sure that Google acknowledge manual pings these days.

[phenomlab]

@DownPW Do you have a Google Search Console account for your domain? If not, and your site is not listed, then this will be because Google changed their algorithm some time ago so that web content isn’t automatically crawled unless it has been actively requested.

Similarly, if your forum were private (which I know it isn’t) then Google would not crawl this either. Can you provide a copy of your site’s robots.txt file?

[DownPW]

@phenomlab said in Am I deferred from the Google search hub?:

Do you have a Google Search Console account for your domain? If not, and your site is not listed, then this will be because Google changed their algorithm some time ago so that web content isn’t automatically crawled unless it has been actively requested.

Nope, we have to create one. At the time everything was done alone, google had automatically indexed the site without it being nothing to do.

@phenomlab said in Am I deferred from the Google search hub?:

Similarly, if your forum were private (which I know it isn’t) then Google would not crawl this either. Can you provide a copy of your site’s robots.txt file?

Exactly not private it’s a public website.

here the robots.txt available in https://XXXXXXX-XXXXX.net/robots.txt

User-agent: *
Disallow: /admin/
Disallow: /reset/
Disallow: /compose
Sitemap: https://XXXXXXX-XXXXX.net/sitemap.xml

Its’ a robot.txt generated by nodebb and we have not modified it

Here, the sitemaps.xml available in https://XXXXXXX-XXXXX.net/sitemap.xml

I have disable Bot Fight Mode in CloudFlare like you said in DM :

and I have submit a ping to googlebot like this :

https://www.google.com/webmasters/sitemaps/ping?sitemap=http://XXXXX-XXXXX.net/sitemap.xml

[phenomlab]

@DownPW Thanks. The missing Google Search Console account alone is your #1 issue here and you should add the site as soon as possible. There is nothing wrong with the robots.txt or sitemap.xml so once you’ve completed the step above, things should start to move.

However, Google can take ages to index a site (as I know first hand when moving from .com to .org here). Bing has a great feature called IndexNow which works with Cloudflare, and can parse a site as soon as it changes to crawl it. Google sadly has not yet adopted this feature despite promising they would as it’s now an open standard.

https://www.indexnow.org/

I’m not sure that Google acknowledge manual pings these days.

[DownPW]

@phenomlab said in Am I deferred from the Google search hub?:

I’m not sure that Google acknowledge manual pings these days.

Ok, so we just asked to crawl the website with the Google Search Console, it’s the only option.

[phenomlab]

@DownPW Yes, exactly.