XF 2.3 Curious issues with google indexing of 'Whatsnew Posts' page

[ISPr_Winter]

I noticed in our logs that google appears to be heavily and repeatedly trying to index a lot of XenForo URLs that lead to the exact same page. I can see where this is coming from, because when I manually navigate to click on the 'New posts' link, which is expressed by Xenforo on mouse-over as 'https://www.mydomain.com/forum/whats-new/posts/', it then loads the page with a different url like this:

https://www.mydomain.com/forum/whats-new/posts/2534695/

^ that's if I'm logged in as a guest, but if I'm a member then it returns this URL ID on the end:

/2534702/

… and if I try a different browser, it returns this:

/2534695/

For some reason (cache?) the site seems to be generating lots of different URL IDs for the same page and causing Google bot to have an indexing frenzy (using masses of different IDs that must have been generated over time for the same page). What's the best solution?

 

[ForumDevs]

I noticed in our logs that google appears to be heavily and repeatedly trying to index a lot of XenForo URLs that lead to the exact same page. I can see where this is coming from, because when I manually navigate to click on the 'New posts' link, which is expressed by Xenforo on mouse-over as 'https://www.mydomain.com/forum/whats-new/posts/', it then loads the page with a different url like this:

https://www.mydomain.com/forum/whats-new/posts/2534695/

^ that's if I'm logged in as a guest, but if I'm a member then it returns this URL ID on the end:

/2534702/

… and if I try a different browser, it returns this:

/2534695/

For some reason (cache?) the site seems to be generating lots of different URL IDs for the same page and causing Google bot to have an indexing frenzy (using masses of different IDs that must have been generated over time for the same page). What's the best solution?

You should disallow search engines to index the whats-new pages through robots.txt:

User-agent: *
Disallow: /forum/whats-new/