We've had great success by checkingAllowfirst, and then checkingDisallow, the idea being thatAllowwas intended to be more specific thanDisallow. That's because, by default (i.e. in the absence of instructions to the contrary), all access is allowed. But then we run across something like this: User-agent: * Disallow: /norobots/ Allow: / The intent here is obvious, but thatAllow: /will cause a bot that checksAllowfirst to think it can crawl anything on the site.
Answer is it depends on the bot. Googlebot respects it following the Disallow entries above it but other bots might not and interpret it as allowing everything, essentially ignoring your explicit Disallow entries.
I used to use it but if xenForo is using a correct 301 and it is still being indexed then something else needs to be figured out here. Google isn't perfect and Matt Cutts is available to talk to here. Something isn't right. These urls are always 301 redirected correct? There is no circumstance where these urls exist where there is not a 301 tagged to them? Therefore, they should not be indexed at all and the correct url, with the canonical tag on it, should be indexed.
No, there is no circumstance where a /posts link is not given a proper 301. If Google ignores the redirect and indexes it improperly, there's not much XenForo can do except redirect when it is hit. I also believe any /posts/ link is nofollow'd by default.