XF 1.5 problem with sitemap and excluded urls

[boeing747]

Hello fellows,

Since two months age I have faced with a strange problem that I couldn't find the exact solution. it drops amount of visitors significantly.

As you know google has introduced its new search console since months ago, it the performance section of this console it shows amount of excluded URL which in my case is more than 950K and the most of them are "Page with redirect". There are tons of URLs like this http://forum.mydomain/posts/4395811/


The other problem is sitemap. in new search console the status of sitemap is "success", but it doesn't show any discovered URL and the number of URLs which has discovered is 0!

When i'm checking google webmasters dashboard in section crawl/sitemap the status is pending with more than 80K warnings.

it says:
Url blocked by robots.txt.
Sitemap contains urls which are blocked by robots.txt.

but I dont block anything.

my robots.txt

User-agent: *
Disallow: /admin.php



Allow: /

I would be thankful if you share your knowledge.

 

[boeing747]

any suggestion or solution?

 

[Ozzy47]

Can't say for certain, but I would adapt this to your site, https://xenforo.com/robots.txt

 

[Optic]

When i'm checking google webmasters dashboard in section crawl/sitemap the status is pending with more than 80K warnings.

Any example URLs for these warnings?

 

[boeing747]

Any example URLs for these warnings?

Sitemap: forum.mydomain.com/sitemap.php?c=3
Value: http: //forum.mydomain.com/data/avatars/l/32/32192.jpg?1458038631

look at the below image



11 soft 404 (its new)

 

[Optic]

Have you recently changed to HTTPS ?

It is very strange that none of your URLs are recognised in your sitemap. It is almost as if Google Search Console thinks it is a different site as it is redirecting all your URLs.

If you changed to HTTPS you need to readd your site as https:// in Google Search Console. Google sees HTTPS and HTTP as different website properties.

 

[boeing747]

Have you recently changed to HTTPS ?

It is very strange that none of your URLs are recognised in your sitemap. It is almost as if Google Search Console thinks it is a different site as it is redirecting all your URLs.

If you changed to HTTPS you need to readd your site as https:// in Google Search Console. Google sees HTTPS and HTTP as different website properties.

No I haven't.

what do you think about htaccess?
I deleted the parameter which was related to redirect www to none www, but nothing happened again.

 

[Optic]

Hi @boeing747

I looked at your forum, it is still running 1.5.9. You should update to 1.5.17 (latest in 1.5 branch)

Google made a change a little while ago when indexing URLs with query strings.

XenForo fixed this in version 1.5.16a :

• Use an alternative approach for sitemap XML URLs due to changes in Google that prevented the old query string approach from being indexed as expected.

https://xenforo.com/community/threads/xenforo-1-5-16a-released.137929/

I deleted the parameter which was related to redirect www to none www, but nothing happened again.

Undo this, you should make sure it redirects to www or no-www. Also you should put a redirect to redirect all http:// to https:// - right now both are accessible.

 

[boeing747]

I looked at your forum, it is still running 1.5.9. You should update to 1.5.17 (latest in 1.5 branch)

I will update it tonight and let you know what happened

Undo this, you should make sure it redirects to www or no-www. Also you should put a redirect to redirect all http:// to https:// - right now both are accessible.

even without SSL certification?
cause I dont have it.

Thanks for your attention.

 

[Optic]

even without SSL certification?

I manually typed "https://" behind your forum URL and it worked OK. So I thought you already setup HTTPS.

For now fix the sitemap issue first. Later you should implement HTTPS when you have time, because web browsers are now marking all non-HTTPS sites as "Not secure".

 

[boeing747]

I manually typed "https://" behind your forum URL and it worked OK. So I thought you already setup HTTPS.

For now fix the sitemap issue first. Later you should implement HTTPS when you have time, because web browsers are now marking all non-HTTPS sites as "Not secure".

Hi,

Fortunately the site map problem has been solved and it shows the number of URLs which submitted and indexed respectively. the amount of indexed URLs are few according to submitted URLs, but it's increasing.

Excluded Urls and crawl errors condition are same as before.

 

[boeing747]

@Optic

do you have any suggestion about excluded urls?

 

[Optic]

Good to hear.

Excluded Urls and crawl errors condition are same as before.

Crawl errors are logged with a “date found” for historical reasons so you should mark them all as “Fixed”. If there are any new errors they will appear in future.

 

[boeing747]

@Optic

Hello,
It really kills me!
Now I got a warning "Indexed, though blocked by robots.txt "
it shows lots of urls like this
forum.mydomain.com/posts/1141376/

I've changed the robots.txt according to Xenforo community.
https://xenforo.com/robots.txt

It disallowed posts, so do I.

the interesting thing is, if I change the /posts/ from disallowed to allow on robots.txt google will exclude posts as redirected urls.

Excluded urls reached to 1.1M :|

 

[trapped_soul]

Hello,
It really kills me!
Now I got a warning "Indexed, though blocked by robots.txt "
it shows lots of urls like this
forum.mydomain.com/posts/1141376/

I'm getting a huge amount of these too...

 

[Optic]

Now I got a warning "Indexed, though blocked by robots.txt "
it shows lots of urls like this
forum.mydomain.com/posts/1141376/

I've changed the robots.txt according to Xenforo community.
https://xenforo.com/robots.txt

It disallowed posts, so do I.

the interesting thing is, if I change the /posts/ from disallowed to allow on robots.txt google will exclude posts as redirected urls.

Hi,

Unfortunately, you cannot fix this issue in XF 1.5. But it's not a harmful issue, because when Google follows the link it redirects correctly via 301 header.

Those links are from the front page (latest post for a topic) and also for thread/topic list inside a forum. Example:


In XF 1.5.x the link is "forum.mydomain.com/posts/1141376/". You click on it and it will 301 redirect to post permalink, e.g. https://xenforo.com/community/threads/problem-with-sitemap-and-excluded-urls.146077/#post-1249074

In XF 2.0 they changed this. They included full topic URL and appended /latest/ at end. e.g. https://xenforo.com/community/threads/problem-with-sitemap-and-excluded-urls.146077/latest , then when you click on it, it redirects to post permalink: https://xenforo.com/community/threads/problem-with-sitemap-and-excluded-urls.146077/#post-1249074

Also, in XF 2.0 they added "rel=nofollow" for these links:

<a href="/community/threads/problem-with-sitemap-and-excluded-urls.146077/latest" rel="nofollow">Today at 7:27 AM</a>

You have to understand, the new Google Search Console is very sensitive and tries to be helpful by reporting these warnings. It should be OK to ignore, but yes it will be annoying..

I think in XF 2.0 this should not be an issue now, but I am not using XF at the moment to 100% confirm.

 

[boeing747]

@Optic Thanks for your help and those information which you have shared with me in this thread.

If someone is using XF2, please inform me about this situation.


Thanks