XF 1.4 Question about friendly URLs

[SimonV]

I just migrated to XenForo just over a week ago and as such I've been checking 404's and redirections from my old content to new content.

While I was checking around I noticed I was getting hits for threads and forums with index.php added at the end of the friendly URLs urls such as:

http://www.certforums.com/threads/loving-the-new-cf-site-platform.50527/index.php
http://www.certforums.com/forums/lounge-off-topic/index.php

Where as following the links via the forums the end URL is:

http://www.certforums.com/threads/loving-the-new-cf-site-platform.50527/
http://www.certforums.com/forums/lounge-off-topic/

Where are these links being generated?

This looks like it also happens here at xenforo.com so I know its not just my site as I do have extra rules in my .htaccess to 301 old content from vB:

https://xenforo.com/community/threads/miscellaneous-improvements.78446/index.php
https://xenforo.com/community/forums/have-you-seen/index.php

You can even manipulate the .php and the URL is resolved:

https://xenforo.com/community/threads/miscellaneous-improvements.78446/index.wtf
https://xenforo.com/community/forums/have-you-seen/index.poopypoo

Is this classed as duplicate content? If so what would be the most appropriate way to fix this?

 

[Mike]

The URLs aren't generated anywhere. It's possibly just down to clients (generally robots/scrapers) that don't properly understand the <base> tag in the HTML.

 

[SimonV]

Its not just a few pages or a few hits its a lot of traffic. It looks like nearly all URLs since I converted, very strange.

I think I plugged it with this rule:

    RewriteCond %{THE_REQUEST} ^[A-Z]{3,9}\ /(.*)index\.php($|\ |\?)
    RewriteRule ^ /%1 [R=301,L]