Implemented IndexNow - Instantly index content in Search Engines

[ivp]

It would be nice to implement new IndexNow protocol that websites can call to notify Bing whenever website contents on any URL is updated or created allowing instant crawling, and discovery of the URL.

Home | IndexNow.org

IndexNow enables webmasters, website owners to inform search engines about the latest content changes on their website. Helps search engines to maintain latest data about a website and redirect searchers to the fresh content. Webmasters/Website owners should use IndexNow to overcome crawl delays...

www.indexnow.org

IndexNow - new initiative by Microsoft and Yandex to push content to search engines

Google seems to currently not be participating in this initiative.

searchengineland.com

 

[nodle]

I would like to see this implemented as well. I know the competitor software (don't want to link it here) just released a big writeup of all their SEO changes coming with their next update. They have this feature implemented as well along with other improvements.

 

[Chromaniac]

Cloudflare has this feature I believe. Currently with a waiting period. I got it few days ago and I have noticed that Bing has indexed recent content which it usually never does. Not sure if this feature is the reason or I just do not check Bing enough to see changes over a longer period of time.

Fact remains, search engines already have access to rss feed and sitemaps to monitor changes on websites. I am not sure why this feature would not become irrelevant once it is adopted by a large number of sites in coming years.

Native support for it would be nice to have of course.

 

[Paul B]

Implemented in 2.3.

XF 2.3 - Miscellaneous changes for XenForo 2.3

Due to time constraints and family matters, this is a slightly different HYS to what was planned, but it is still a bumper feast of new features coming in XenForo 2.3. This week we will be mostly focusing on a bunch of smaller new changes and improvements we're no less excited to show you in the...

xenforo.com

 

[masmo]

Implemented in 2.3.

I recommend checking the robots.txt file to prevent XenForo IndexNow links from being blocked:

M

XF 2.2 Post in thread 'Does XF has a default robots.txt file?'

Jul 24, 2024

You just create one to suit your site and needs.

This is the one from here: https://xenforo.com/robots.txt

Just to say that yesterday Bing reported the indexing problem to me because I have used the robots.txt recommended on this site and used by many others users.

XenForo IndexNow is using links with the format https://sitename.com/posts/123456/ and in the robots.txt file there is Disallow: /posts/
So be careful because this robots.txt blocks all IndexNow links!

I hope I was helpful...

• masmo

 

[nodle]

I recommend checking the robots.txt file to prevent XenForo IndexNow links from being blocked:

M

XF 2.2 Post in thread 'Does XF has a default robots.txt file?'

Jul 24, 2024

You just create one to suit your site and needs.

This is the one from here: https://xenforo.com/robots.txt

Just to say that yesterday Bing reported the indexing problem to me because I have used the robots.txt recommended on this site and used by many others users.

XenForo IndexNow is using links with the format https://sitename.com/posts/123456/ and in the robots.txt file there is Disallow: /posts/
So be careful because this robots.txt blocks all IndexNow links!

I hope I was helpful...

• masmo

Can you confirm if we should do this @Chris D ?

 

[masmo]

@nodle

As I said, all links sent with IndexNow contain /posts/ and are blocked by robots.txt (if Disallow: /posts/ is included as in this and your site).
Bing Webmaster Tools reported it to me, check your links yourself.

 

[nodle]

@nodle
View attachment 306967
As I said, all links sent with IndexNow contain /posts/ and are blocked by robots.txt (if Disallow: /posts/ is included as in this and your site).
Bing Webmaster Tools reported it to me, check your links yourself.

I am not doubting you, I just wanted to see what staff thinks before implementing it since they still have it in their own robots.txt still.

 

[Old Nick]

But this is intentional, you will lose crawling budget if you tell the robots to index all your posts, especially since most of them will be useless...
The Google Search Console also informs that the URLs /posts/ are blocked via the robots.txt file but again this is not a problem knowing that we want it.

 

[Old Nick]

Even in the sitemap there is no option to send posts to search engines. There is a reason...

 

[masmo]

But this is intentional

No, this was poorly implemented as most XenForo installations have "Disallow: /posts/" in the robots.txt
IndexNow links must be different otherwise it is useless.

Other add-ons use for example a link to the discussion/page (not to the single post) and they have always worked correctly.

if you tell the robots to index all your posts

In XenForo there are canonical URL's to prevent this.

 

[Old Nick]

I see, I will still wait for more information because I am not convinced by the interest of indexing posts in terms of SEO. Or maybe I didn't understand the purpose of IndexNow which ultimately sends alerts to search engines when content is modified, deleted or created?

 

[Old Nick]

Other add-ons use for example a link to the discussion-page for example and not to the post and they have always worked correctly.

I used it in my last forum.

 

[masmo]

I am not convinced by the interest of indexing posts in terms of SEO

In theory you are not asking to index the single post:
https:/mysite.com/posts/12345 redirect to https:/mysite.com/threads/discussion-title/#12345 that contain a canonical URL to the discussion and not to a single post.

I think it was designed to work in that way but if https:/mysite.com/posts/12345 is blocked by robots.txt (in most installations) it doesn't make sense anymore.

If the developers can't change the IndexNow URLs maybe it's better to add a note in the admin panel...

I used it in my last forum.

Me too

 

[Rusty Snippets]

Me three! AndyB is the best!