Traffic Down Since VB to XF Migration

[Alfuzzy]

Sorry...I'm not exactly where to find all these:

* Settings -> Crawl Stats...can find this no problem. What part of the Crawl Stats would be helpful? We've been talking "By Response" area earlier...not sure what other part to screenshot.

* Search Results (not sure where to find this)...and what to screenshot.

* Coverage...I can find this no problem...I go here a bunch. Not sure which part to screenshot.

Thanks very much.

 

[arn]

Sorry...I'm not exactly where to find all these:

* Settings -> Crawl Stats...can find this no problem. What part of the Crawl Stats would be helpful? We've been talking "By Response" area earlier...not sure what other part to screenshot.

* Search Results (not sure where to find this)...and what to screenshot.

* Coverage...I can find this no problem...I go here a bunch. Not sure which part to screenshot.

Thanks very much.

Start with the graphs on top on these pages.

 

[Masetrix]

Just to give folks an idea of the kind of magnitude we're talking here in terms traffic lost when the site was migrated from vB4.2.5 to XF 2.1 (July, 2020).

You're making the same mistake as I made it, so a lot of your content (postings, members ...) is excluded from the search (because of Disallow in robots.txt!) That's why the traffic(~) is lost.

In addition, in contrast to vB, XF does not consider postings in its sitemap. The result is that you have a lot of content that was still considered and rated by Google in Vb, is now missing in the Google rating in XF.

Here is a graphic that shows how much Google then excludes: (Grey== excluded Content/Postings Members....)

That's why I'm still looking for a way to use XF to send all postings in the sitemap to Google.

 

[Alfuzzy]

Sorry Arn...my Google Search Console doesn't look like that. In the left vertical margin area where all the different GSC section are to click on...I don't have "Search Results, Discover, or Google News".

It appears from your screenshot that those areas are within the "Performance" section. I do have "Performance" in the left margin area to click on....but it's not a section like it is with your GSC (with additional areas within "Performance").

Here's a screenshot of most of what I have (there is more...but it's mostly "Security" and "Legacy Tools" items):



Thanks

 

[Alfuzzy]

You're making the same mistake as I made it, so a lot of your content (postings, members ...) is excluded from the search (because of Disallow in robots.txt!) That's why the traffic(~) is lost.

Thanks very much for that info. In your opinion...what should I remove from my robots.txt?

Thanks

 

[briansol]

my robots file looks like this:

User-agent: *

Disallow: /account/
Disallow: /conversations/
Disallow: /find-threads/
Disallow: /goto/
Disallow: /help/
Disallow: /job.php
Disallow: /login/
Disallow: /lost-password/
Disallow: /misc/
Disallow: /online/
Disallow: /posts/
Disallow: /profile-posts/
Disallow: /register/
Disallow: */unread$
Disallow: */latest$

Allow: /search/$
Disallow: /search/*

Allow: */threads/post-*
Disallow: */post-*

Sitemap: https://mydomain.com/sitemap.php

 

[Masetrix]

Thanks very much for that info. In your opinion...what should I remove from my robots.txt?

Thanks

Everything that concerns members, login and postings.

@arn
There is a new view of the GSC, which is also used by the TE

 

[arn]

Sorry Arn...my Google Search Console doesn't look like that. In the left vertical margin area where all the different GSC section are to click on...I don't have "Search Results, Discover, or Google News".

It appears from your screenshot that those areas are within the "Performance" section. I do have "Performance" in the left margin area to click on....but it's not a section like it is with your GSC (with additional areas within "Performance").

Here's a screenshot of most of what I have (there is more...but it's mostly "Security" and "Legacy Tools" items):

View attachment 244818

Thanks

Just performance screenshot then would work.

 

[Masetrix]

@briansol
These entries are not a good idea. Users should set their privacy themselves in their settings and excluding postings (goto) is also not good.

Disallow: /account/
Disallow: /goto/
Disallow: /help/
Disallow: /login/
Disallow: /online/

Disallow: /post-

Since I've removed these things from robots.txt, the number of hits has increased again, Google indexes more again and my "excluded" content in the GSC is less.

Have a look here:

Traffic Down Since VB to XF Migration

Sorry...I'm not exactly where to find all these: * Settings -> Crawl Stats...can find this no problem. What part of the Crawl Stats would be helpful? We've been talking "By Response" area earlier...not sure what other part to screenshot. * Search Results (not sure where to find this)...and...

xenforo.com

Unfortunately, I only found out all of this by working with the AMP add-on from @mazzly ...

 

[briansol]

I have no reason for google to index my help pages, and anything that 301's. That's why i have those things in there.

I have messed with the robots file from time to time and honestly, short of blocking something stupid like 'threads', i've not seem much of a difference in index quality or bloat pages polluting the index.

 

[Alfuzzy]

Just performance screenshot then would work.

Here are two screenshots...hopefully they are helpful.

1st screenshot is from Google Search Console "Performance" (May 1, 2020 thru Sept 30, 2020).

• Red circle is when migration to XF 2.1 took place.
• Just for fun...red arrow is when the big Google algorithm change happened May 2020.

2nd screenshot is from Google Analytics (Pageviews)...May 1, 2020 thru Sept 30, 2020. Red circle is when migration to XF 2.1 took place.



Hope this helps. Thanks

 

[Alfuzzy]

For you guys helping with suggestions for robots.txt. Here is mine...what would you guys add or remove?:



Thanks

 

[arn]

Here are two screenshots...hopefully they are helpful.

1st screenshot is from Google Search Console "Performance" (May 1, 2020 thru Sept 30, 2020).

• Red circle is when migration to XF 2.1 took place.
• Just for fun...red arrow is when the big Google algorithm change happened May 2020.

View attachment 244825

2nd screenshot is from Google Analytics (Pageviews)...May 1, 2020 thru Sept 30, 2020. Red circle is when migration to XF 2.1 took place.

View attachment 244826

Hope this helps. Thanks

hmm.

• Big drop was from Google core update
• There's a definite but smaller drop after XF migration.
• What's with the massive uptick in the early part of the Google analytics graph?
• Has the number of daily posts dropped too around that time? You can graph that in Xenforo https://url/admin.php?stats/

 

[arn]

Google Analytics:

Behavior -> Site Content -> All Pages



Click on All Users at the top.
Uncheck All Users in the options
Check off Direct Traffic and Organic Traffic
Click Apply

Then it should look like this.



and post that graph during the same time period.

 

[Alfuzzy]

• Big drop was from Google core update

Yes...not easy to miss that one.

• There's a definite but smaller drop after XF migration.

The Pageview drop in screenshot #2 is greater than 50%. Drop in site revenue confirms this as well.

• What's with the massive uptick in the early part of the Google analytics graph?

I seem to remember this being a "site glitch" due to someone with access to the Admin CP changing a setting & causing chaos!

• Has the number of daily posts dropped too around that time? You can graph that in Xenforo https://url/admin.php?stats/

Stats grabbed from XF (Jan, 2020 thru today)...Posts & Post Reactions. Red circle is when migration to XF took place. Spike in activity at that time was probably due to the migration to XF (lots of reactions/questions).



I'm not too surprised by this graph (not much change before or after migration). All of this activity would have been from registered users. Much of site traffic is from guest visitors (guest visitors make up way more of the site traffic than registered users) This is good for page views & ad impressions/revenue.

I feel like after the migration to XF...less site content is showing up in Google search results. Which has resulted in fewer guest visits...thus less site traffic...thus less page views...thus less site revenue (ad impressions).

So the question could be...what is causing so much less site content not showing up in Google searches before & after the migration from vB to XF?

• Automatic site maps are regularly submitted to Google.
• Of course Google crawls the site whenever Google crawls the site.
• Google core update May 2020 & Google algorithm update Sept 2020. Site migration/drop in site traffic was between June & August 2020 (July 2020)...thus those two Google updates were before & after the drop in traffic & shouldn't be responsible for the drop in traffic in July.

I'm of course confused how to solve this.

Thanks

 

[Alfuzzy]

...and post that graph during the same time period.

Hopefully I followed your instructions correctly. Here's the Google Analytics graph for Pageviews for the same time period as previous graphs (May1, 2020 thru Sept 30, 2020)...Direct Traffic & Organic Traffic. Red circle is when migration to XF took place (1st week of July 2020):







Hope this helps,

Thanks

 

[arn]

Hopefully I followed your instructions correctly. Here's the Google Analytics graph for Pageviews for the same time period as previous graphs (May1, 2020 thru Sept 30, 2020)...Direct Traffic & Organic Traffic. Red circle is when migration to XF took place (1st week of July 2020):

View attachment 244849



View attachment 244848

Hope this helps,

Thanks

Don't have any deep insights.

Best guess is that traffic drop is all Google traffic. The direct views appear similar to the direct views back in May according to that graph. I still don't know if the increase from June->July for direct views is real or not -- guessing not real.

So that leaves just a drop in organic traffic that is reflected in Google's Search Console. Which is what you thought to begin with.

In Settings -> Crawl Stats. What is the number of crawl requests in the past 90 days? And how many Valid pages are in your Coverage tab?

 

[Alfuzzy]

In Settings -> Crawl Stats. What is the number of crawl requests in the past 90 days? And how many Valid pages are in your Coverage tab?

• Number of Crawl Requests past 90 days = 1.94M
• Number of Valid Pages from Coverage tab = 152K

Valid page number used to be much higher. On the Valid Page Graph...on October 30th, 2020 (oldest date available on the graph)...it was 368K...and before this it was probably much higher still. Maybe this is due to Google algorithm changes.

Thanks

 

[sbj]

Isnt it that in vB Google crawled everything, which means any page of a thread and in XF only first page is to be found?

I remember somewhere something like that and people using SEO addons to get everything crawled and to be found. But I might be wrong. That might be the reason why you lost so many visits and your coverage went down.

 

[arn]

• Number of Crawl Requests past 90 days = 1.94M
• Number of Valid Pages from Coverage tab = 152K

Valid page number used to be much higher. On the Valid Page Graph...on October 30th, 2020 (oldest date available on the graph)...it was 368K...and before this it was probably much higher still. Maybe this is due to Google algorithm changes.

Thanks

Ok. Then the 4XX responses is your biggest problem.

If that's 46% of your 1.94M crawl requests, that's a ton. It wasn't originally clear if Google just wasn't crawling your site, but with 1.94M crawl requests, that seems like a solid amount.

For comparison, here's my stats:



My 4XX is 2% of my crawl. Others here: https://xenforo.com/community/threads/crawl-budget-stats-in-google-search-console.188395/

If you have access to your server logs, you might be able to look there. But it's a little more technical to pull the data from that. You just need to figure out why Google is hitting 4XX errors when visiting half your site. Your options are:

• Permission set wrong
• Google is being directed to pages it's not supposed to be. Sitemap? Bad linking somewhere?
• other less common ones: https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_client_errors

arn