XF 1.1 Significant drop in traffic since vB switch - hasn't recovered

[mcadx]

It's been about 9-10 months since we made the switch from vB4 to XF. We expected a significant drop in traffic since the search engines would have to re-index everything but it has not recovered. In fact, guest traffic has continued to drop. In the final days before the cutover, we had some 45,000 pages indexed with Google. Today we have less than 30,000. The site hasn't shrunk - we have over 40,000 threads and the number of posts continues to grow on a regular basis as member activity has been about the same if not a bit higher but search traffic is a mere fraction of what it used to be. We have a sitemap generator installed and there are no errors to speak of in Google Webmaster Tools and our PageRank has not dropped.

Mods/add-ons are minimal at less than 10 so I have a feeling that we may have some settings that aren't correct. What things could we look at to make sure that everything is being indexed as it should?

Thanks

 

[craigiri]

If your site is easily crawled by a human, then google should be able to find all the pages.

As a test - are you saying that you can grab some text from one of the 15,000 pages (not indexed) and do a google search and not find it? It may be that google has changed some of their crawling and indexing methods - and/or you have set the amount of threads per page differently (which would result in fewer pages)....

I assume you did the basic redirects after you moved?

My site is fairly vanilla and we lost virtually no search mojo in our move about 9 months ago - and, in fact, are gaining in visits and visitors.

Assuming that guests can view most everything in your forums, I don't think there are too many settings on your end to fuss with.

 

[mcadx]

Yes, the redirects are in place and guests can easily navigate the site. One thing I didn't mention in the OP is that at the time of upgrade we changed the structure. The forums used to be in the root whereas Wordpress now lives in the root and the forum in /community. Either way, we placed the redirect script into the root and updated it to reflect the new location of the forums.

As you can see, it's indexing new content that has been added but that drop is significant.

 

[hellreturn]

Do you have higher 404 or 403 in your google webmaster ?

 

[mcadx]

Only have 154 404 errors. I don't see any 403. We do have some errors in the last few weeks due to my host getting hacked so I imagine those will clear up in due time.

 

[mcadx]

Just kidding. There are a lot of 403s but that's due to guests being prohibited from viewing full size images/attachments. That's no different than before with vB, however.

EDIT: Scrolled through about 1,000 403 entries and only 2 were links to content that should've been visible to the bots. Marked them as fixed. The others were all attachment pages that guests don't have access to.

 

[psTubble27]

Are the redirects (espescially from root to /community) permanent or temporary?

 

[mcadx]

Ok, as I'm digging deeper I found a bunch of 404s that seem to indicate that perhaps my redirect script isn't working properly and seems to have spiked significantly about 3 weeks ago. Looking at the URLs, they're vB URLs and not Xenforo. Though if I do a Google site search for one of the threads listed under the 404 list, it comes up with the updated page. Could there be conflicting caches of my site in Google?

 

[mcadx]

Are the redirects (espescially from root to /community) permanent or temporary?

The redirect is Kier's 301 redirect script so it should be permanent. The file has been there since the day of the cutover.

 

[mcadx]

Ok, as I'm digging deeper I found a bunch of 404s that seem to indicate that perhaps my redirect script isn't working properly and seems to have spiked significantly about 3 weeks ago. Looking at the URLs, they're vB URLs and not Xenforo. Though if I do a Google site search for one of the threads listed under the 404 list, it comes up with the updated page. Could there be conflicting caches of my site in Google?

In doing some more digging, it seems that right around this time is when Apache was upgraded to 2.2.23 and screwed up my .htaccess rewrite rules that were created based on this tool: http://tools.geekpoint.net/xfseo/

Here's what my .htaccess rewrite rules look like currently

<IfModule mod_rewrite.c>
RewriteEngine On

RewriteRule [^/community]+/([\d]+)-.+-([\d]+).html showthread.php?t=$1&page=$2 [NC,L]
RewriteRule [^/community]+/([\d]+)-.+.html showthread.php?t=$1 [NC,L]

RewriteBase /community
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>


# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /community
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

Safe to say that the problem is in here somewhere? Any guesses?

 

[CTXMedia]

My CycleChat site shows a similar downward total of indexed pages but we're busier than ever. I suspect because we no longer allow user profile pages or individual post URLs to be indexed (and our separate .co.uk domain is now used as an internal CDN URL for static content) so there's less "volume" (down from 685,000 indexed pages to 190,000 or so) but more quality as the content that is indexed (mainly threads/first posts) is keyword and topic relative to our niche - cycling.

What has been the trend for your bounce rate and visitor/return ratio over this period?

Also, from within your XF ACP - what does your post count stats graph look like for the past 2 years?

Cheers,
Shaun

 

[Jarod]

Yes, the redirects are in place and guests can easily navigate the site. One thing I didn't mention in the OP is that at the time of upgrade we changed the structure. The forums used to be in the root whereas Wordpress now lives in the root and the forum in /community. Either way, we placed the redirect script into the root and updated it to reflect the new location of the forums.

As you can see, it's indexing new content that has been added but that drop is significant.

View attachment 36997

Hello,

Have you a same graph(Google analytics) with visits or page views since Vb4 switch ? .

Thanks

 

[mcadx]

My CycleChat site shows a similar downward total of indexed pages but we're busier than ever. I suspect because we no longer allow user profile pages or individual post URLs to be indexed (and our separate .co.uk domain is now used as an internal CDN URL for static content) so there's less "volume" (down from 685,000 indexed pages to 190,000 or so) but more quality as the content that is indexed (mainly threads/first posts) is keyword and topic relative to our niche - cycling.

What has been the trend for your bounce rate and visitor/return ratio over this period?

Also, from within your XF ACP - what does your post count stats graph look like for the past 2 years?

Cheers,
Shaun

Bounce rate seems to be relatively steady in the same time period. Lower after the upgrade for about 2 months(I'm guessing because users were scoping out the new setup), settling off for a while and then trending higher since the above said Apache upgrade last month.

Roughly 40% are new visitors since the upgrade, YTD.

Post count graph seems to be slowly trending upwards, as it should.

 

[mcadx]

Hello,

Have you a same graph(Google analytics) with visits or page views since Vb4 switch ? .

Thanks

 

[CTXMedia]

What are page views for the same period?

 

[mcadx]

Used to be around 100k / month. Post migration it's about half that.

 

[Jake Bunce]

Page views always go down. XenForo is more efficient with page loads. For example, AJAX replies remove one page view. This is not a problem.

 

[mcadx]

That's good info, thanks.

Any suggestions for the htaccess rewrite rules to reduce those errors?

 

[Jake Bunce]

Which errors for which pages specifically? If they are old vB links then they would either be redirected or turned to 404s.