vB 3.8 > xF 2.2 migration : 8 months later

benFF

Well-known member
Been running my forum on vB since 2005 (was on Snitz before that) - but like a lot of people never progressed further than vB 3.8. However after getting stuck in quarantine last November and with nothing else to do, I figured I should use the time productively to upgrade! :D

So upgrade I did - using the standard xF convertor and a load of custom scripts I was able to import all the data over with everything nicely 301'd:
  • threads (450k)
  • posts (4.5m)
  • attachments (200GB)
  • reactions from Dragonbyte Thanks/Like
  • photopost (150GB)
  • photopost classifieds
  • dragonbyte downloads
  • gars
  • vbGarage
To get it done as fast as possible, after doing 2 test imports on my laptop - I then spun up a super powerful compute VPS at OVH with 24 cores and 48GB of RAM to do the actual live import. The posts and whatnot were done very fast, about 40 minutes, the attachments took another few hours, but the longest was the gallery import as this doesn't support multithreaded. All in all, the conversion took about 7 hours.

Then another few hours to copy all the data back over onto the original server, plus one copy onto Wasabi storage just incase it all mucked up.

As I had already converted my vb3 template to responsive, it was easy enough to emulate the same design with xF, just moderning it up a bit - so it wouldn't be such a big shock to the users.

So great, all done and ready to go live - and everything was great for about 3 days until Google decided to basically throw a wobbly.

Traffic dropped by about 33% and revenue just hit the floor (although the revenue part was kinda expected as AdSense would need to spider all the pages) so I just put that down to "needs must" - so let's just wait....

8 months later and where are we then?

Sadly, still down 25% - the traffic never recovered :(

1660071376019.png
(That's Jan > July of 2022 vs 2021)

1660071000082.png

As you can see 2020 and 2021 were running about the same until the end of November when the upgrade took place and it just got worse as time went on (this is sessions, not pageviews - they can be seen below)

1660071153462.png

That basically correlates to the sessions.

Now interestingly if we look at Google search console for the period May 1 > July 31 2022 vs 2021 (That's as far back as the comparison goes) we can see that impressions are up (or steady)

1660071633488.png

But clicks are down 20%

1660071679134.png
Which seems to be related to search position (down 30%) - although that doesn't explain the bit at the end.

1660071736292.png

Like I said, everything was 301'd correctly from the off - the ONLY thing that was different is that xF defaults to 20 items per page, whereas my vB install used 25 items per page. This means that I guess the old and new results weren't exactly "like for like" - would that have made a big difference, perhaps Google then didn't see it as moved content but totally new? I don't know, but I hope it wasn't that as if it's that important, the importer should have copied the setting over...

Would it be worth changing that setting back - or has too much time passed now to make a difference?

I did have the xF AND vB sitemaps submitted at the same time, so I was able to watch as one gained indexing and the other lost it. It took about 4 months until the flippening (more indexed results in the new one than the old). Also I amended my robots.txt to only index the important stuff - so tons of pages were dropped from that too. Currently out of the 500k submitted pages, 203k are indexed and 308k excluded.

Code:
User-agent: *
Disallow: /account/
Disallow: /attachments/
Disallow: /classifieds/authors/
Disallow: /donate/
Disallow: /goto/
Disallow: /login/
Disallow: /logout/
Disallow: /lost-password/
Disallow: /media/comments/
Disallow: /members/
Disallow: /online/
Disallow: /posts/
Disallow: /register/
Disallow: /search/
Disallow: /tags/
Disallow: /whats-new/
Disallow: /admin.php
Disallow: /index.php?sam-item/
Allow: /

Anyway - what is the take away from all this, was it all worth it...?

Ignoring the revenue side of things, definitely - the site as a whole is much nicer to use now and working on it is much much easier. So from a UX perspective, top marks!

From a business perspective though - it may have been the wrong decision. A 30% hit in revenue is massive, so you need to consider this carefully....
 
Last edited:

nicodak

Well-known member
the ONLY thing that was different is that xF defaults to 20 items per page, whereas my vB install used 25 items per page
why not have configured the number of pages in the same way on XF?
/admin.php?options/groups/messageOptions/
 

benFF

Well-known member
why not have configured the number of pages in the same way on XF?
/admin.php?options/groups/messageOptions/

I only realised this many months after the migration, so figured by then it was too late....
 

briansol

Well-known member
Without changing anything on my site, I'm seeing a similar decline over the last year, -15%

So, while there is correlation, you're also missing market share and global user trends towards social media instead of forums that are hard to measure.

I would leave the pages at default. Don't trigger yet another re-index if you're still in mid- re-index.

the numbers of indexed vs crawled is pretty normal. I get about 20-30% indexed these days.
google has stopped indexing a lot of fluff. classifieds, chit chat, etc.

Are you using responsive ads? or auto ads?
served behind a network?


A drop in page views is expected, as xf uses more ajax and less confirmation pages/redirects.
sessions though, should stay flat -ish.
 

benFF

Well-known member
A drop in page views is expected, as xf uses more ajax and less confirmation pages/redirects.
sessions though, should stay flat -ish.
Yes, I had expected the pageviews drop, but as you can see, sessions took a right bash at the same time - it wasn't gradual either, within a week it was down.
Are you using responsive ads? or auto ads?
Back then it was AdSense - but I switched to AdThrive in February as they promised a better return (which I badly needed). I didn't like the idea of sticky footers and those bottom right corner videos, but sadly it had to be done.
So, while there is correlation, you're also missing market share and global user trends towards social media instead of forums that are hard to measure.

That would make sense if there was a gradual decline, but there was a big drop after conversion - then flat line ever since.
 

benFF

Well-known member
A drop in page views is expected, as xf uses more ajax and less confirmation pages/redirects.
I've just been pondering over this - and I'm not sure that is correct. I didn't have any confirmation pages or redirects on vb (except the login page, but that didn't account for much) plus those types of pages don't have analytics installed anyway.

90% of visitors just look at the home page, forum listing and thread view - there was nothing AJAX'd there. Infact, showing less items per page should have in theory meant XF pageviews increased.
 

VersoBit

Well-known member
Those are some interesting drops in your views/interactions.... would you mind sharing your domain? I'd be interested in taking a closer look as our numbers have stayed relatively stable for multiple years now on XenForo (even through our moves through two other forum softwares)
 

benFF

Well-known member
Yikes mate, that's even worse - sorry to see that :(

You were having a nice amount of growth just before the switch too - did you notice much of a change in your Google results?
 

benFF

Well-known member
That made me think actually, I forgot that Analytics can filter by Source/Medium.

So just to add to the data points so far, here is the traffic filtered by Google search source.

1662637051418.png
 

Adam Bloch

Member
Hi thanks for getting back to me. I seem to have a problem with too many redirects.... about 141,000 of the. Each page is being redirected up to 5-6 times. Yes I have a few old VB references still in there as well. There are a few http:// broken images and a host of other things that are really not helping. I'm just running an audit on the site at the moment and at 30% of the site being crawled I have 6000 mixed content problems. I have managed to get rid of all the 5xx errors which is a start.

Now working on all the 404s

I not sure how to work out how I have so many redirects.

 

Adam Bloch

Member
just out of interest this is the .htaccess we are using for the VB any thoughts or comments?

I'm thinking of removing the wild card at the end...

RewriteEngine On

#vbSEO Threads (basic & paginated)
RewriteRule ^[^/]+/([0-9]+)-.+-([0-9]+)\.html$ /xf/index.php?threads/$1/page-$2 [R=301,L]
RewriteRule ^[^/]+/([0-9]+)-.+\.html$ /xf/index.php?threads/$1/ [R=301,L]

#vbSEO Forums
RewriteRule ^([0-9]+)-[A-Za-z0-9-/]+$ /xf/forums/$1/ [R=301,L]

#vbSEO Profiles
RewriteRule ^members/([0-9]+)/.+\.html$ /xf/members/$1 [R=301,L]

#non-SEO Profiles
RewriteCond %{QUERY_STRING} ^u?=?([0-9]+).*$
RewriteRule ^member\.php$ /xf/members/%1/ [R=301,L]

#non-SEO Threads
RewriteCond %{QUERY_STRING} ^t?=?([0-9]+).*$
RewriteRule ^showthread\.php$ /xf/index.php?threads/%1/ [R=301,L]

#non-SEO Forums
RewriteCond %{QUERY_STRING} ^f?=?([0-9]+).*$
RewriteRule ^forumdisplay\.php$ /xf/forums/%1/ [R=301,L]

#bounce everything else to index.
RewriteRule ^.*$ /xf? [L]
 

IMCD

Member
We saw a similar drop after moving from vb3.8 to vb4 and then from vb4 to xf another drop.

What's interesting though is that we have multiple forums and one that was moved to xf in 2019 had a less pronounced drop in 2020 and 2021 but now in 2022 has dropped big.

We do have one that was moved in November 2021 so you and I could compare directly in this regard. YOY is around -22%.

Without a doubt, xf pages are less favorable in Google results than vb pages but I've never seen any explanation as to exactly what makes this so.
 

woody

Well-known member
Back then it was AdSense - but I switched to AdThrive in February as they promised a better return (which I badly needed). I didn't like the idea of sticky footers and those bottom right corner videos, but sadly it had to be done.
I switched to AdThrive in 2020. Good first year, amazing revenues in 2021, and 2022 is down 15% or so.

I ONLY serve AdThrive to guests. They see the full load of ads. Registered users still get my own AdX/APS/direct ad setup, allows me to better manage how many ads those users see as well as keep the layouts very consistent. Effectively, a reward for being logged into the forum.

85%ish of my traffic is from Google searches, which is why AdThrive performs well.
 

MySiteGuy

Well-known member
Like I said, everything was 301'd correctly from the off - the ONLY thing that was different is that xF defaults to 20 items per page, whereas my vB install used 25 items per page. This means that I guess the old and new results weren't exactly "like for like" - would that have made a big difference, perhaps Google then didn't see it as moved content but totally new? I don't know, but I hope it wasn't that as if it's that important, the importer should have copied the setting over...
This can actually be a significant change to content.
Google also recommends keeping redirect chains to 3 or less if you want to maintain full page rank.
 

MySiteGuy

Well-known member
That's a shame - I guess 10 months later is too late to change it back now :(

I can't answer that, but I can say I've seen Google do things like recrawl a URL that's been missing for 10+ years, find the content again (recovered from archive.org), and give the page decent rank. They have a long memory!
 

benFF

Well-known member
I can't answer that, but I can say I've seen Google do things like recrawl a URL that's been missing for 10+ years, find the content again (recovered from archive.org), and give the page decent rank. They have a long memory!

Hmm... that's a very interesting observation.

I did a quick mysql call and see that 10% of my threads are over 20 posts so would have been affected.

I guess I'm slightly worried that doing a big change again now could backfire and reset everything again :s

Worth a shot?
 

MySiteGuy

Well-known member
Worth a try but only if you take care of any outstanding issues search console might have listed. It wouldn't hurt to get a Semrush trial account to see if they can point out issues.
 
Top