I'm trying to figure out why Xenforo.com decided to prevent google to crawl /posts/, which are probably the most prominent links on a XF default install.
By doing so, you actually kill each link to your threads from the homepage.
ALso, /posts/ are actually 301 redirects to /threads/
However, I observe that Google, when indexing /posts/ , decides to keep the /posts/ URL in the SERP ( it seems it does not care at all about the URL beeing a 301 redirect). The expected behaviour would be for google to use the /threads/ destination in the SERP, but I've lokoed at the results for various XF installs, and it does not
So your content is crawled, but with an "unfriendly url" in the result.
eg this example I took with someone posting his forum in the "showcase" forum
I also noticed some very large and respectable installs, like digitalpoint, decided (like I did) to let google crawl /posts/ otherwise you really make it impossible for google to quickly discover content (it has to make 2 or 3 "click" to find any URL starting with /threads." ...
I noticed that DP.com urls in google results were mostly /threads/. I looked at their http header and they simply 301 redirect /posts/, which is the default behaviour of XF. They do nothing special.
What do you all think about indexing /posts/ ?
If @digitalpoint reads me, wdyt ?
Why is google indexing /posts/ in certain installs (and it should NOT it's a 301 redirect!), and /threads/ in other ?