Traffic Down Since VB to XF Migration

[Alfuzzy]

The topic is extensive and, depending on the situation, difficult.

Thanks for the extra info.

In post #18 you included a link...and mentioned "Here you should scan the individual pages and see which 4xx codes are in the game."

My question was...What tool do I use from the link to scan my site?:



Thank you

 

[Masetrix]

The site above is for explaining "HTTP Status Codes" only.
The Link in #18 is the Tool you need and what you looking for.

Or, see this...

URL Inspection Tool - Search Console Help

About the URL Inspection report and test The URL Inspection tool provides information about Google's indexed version of a specific page, and also allows you to test whether a URL might be indexable.

support.google.com

 

[Alfuzzy]

If I go to the website mentioned in post #18:

Free Tools from WebSitePulse

Free tools to check your website or server by WebSitePulse. Detect any malfunctions and fix them immediately.

www.websitepulse.com

And use the "Web Page Test" tool:

Web Page Test by WebSitePulse

Verify the URL availability, download the HTML content, the images, and all the other internal web page objects and measures their download speed.

www.websitepulse.com

• If I go to the Google Search Console (Settings -> Crawl Stats)...in the "By Response" section -> Other client error (4xx).
• If I then enter some of the individual URL's from the Other client error (4xx) list into the "Web Page Test" tool.
• The "Web Page Test" tool returns with the Status = "Down"

If I enter the same URL into my browser...it goes to the correct page just fine. Thus I'm really sure what the "Web Page Test" tool (or Google Search Console) is seeing that I'm not seeing (I'm not really sure what to do to correct the situation).

Thanks

 

[arn]

By Response:

• Other Client Error (4xx) = 46%
• Ok (200) = 31%
• Moved Permanently (301) = 20%
• Not Found (404) = 2%
• Moved Temporarily (302) = 1%

Click on "Other Client Error (4xx)" and see what the list of urls looks like.

When it shows you the urls, visit the URL in a non-logged-in browser window and see what happens.

arn

 

[Alfuzzy]

Click on "Other Client Error (4xx)" and see what the list of urls looks like.

When it shows you the urls, visit the URL in a non-logged-in browser window and see what happens.

arn

Hello Arn,

I did as you advised:

• Logged out of the site.
• Clicked on "Other Client Error (4xx)"...then Google Search Console displays a list of URL's.
• When I hover over any of the URL's...to the right of the URL 3 icon's appear (copy URL to clipboard...open in new tab...and inspect URL).

If I click on the "open in new tab" icon...I get the message from Xenforo "You must be logged-in to do that." I did this for a bunch of the URL's in the list for "Other Client Error (4xx)"...and they all came back with the same message.

As a double check...I then logged into the site...did the same thing with the "Other Client Error (4xx)" URL list...and every one of the URL's loaded fine.

I know site content should be setup to be visible to non-registered users (they can see & read site content)...but they need to be registered to create new threads or post replies to existing threads.

Maybe some content is visible to non-registered visitors...and maybe some content isn't...since the "Other Client Error (4xx)" percent = 46%.

If this sounds like the case (some site content is not visible to the non-registered internet users)...what should I explore to make sure all site content is visible to everyone (except private areas...like site Staff Only areas)?

Thanks very much.

p.s. By the way...I should also mention in case it helps. Google Search Console -> Crawl Stats -> By Response ->"Other Client Error (4xx)" = 46%. The number of URL's in the list is eleven (11) over the past 90 days (it seems Google only lists the last 90 days). I don't know if this is a lot...or if there are many more past 90 days. Also...I'm sure in that same 90 day period...many many new threads were created on the site.

 

[arn]

If this sounds like the case (some site content is not visible to the non-registered internet users)...what should I explore to make sure all site content is visible to everyone (except private areas...like site Staff Only areas)?

Where the URLs you tested supposed to be visible to guests or not?

11 URLs doesn’t seem like much at all. maybe that’s why it’s such a larger percentage.

What is your robots.txt?

you should also look at Google Search Console -> Coverage and see if that looks funny.

 

[Masetrix]

Everything you need is already there in the GSC. You should make the effort to use the Google help for the GSC to learn how to use it to troubleshoot and then cause Google to re-crawl the faulty things. Then the Google values of your site will get better again

You have a lot to do, so get started ...

 

[Anatoliy]

If this sounds like the case (some site content is not visible to the non-registered internet users)...what should I explore to make sure all site content is visible to everyone (except private areas...like site Staff Only areas)?

if those urls you are talking about are threads, then check the permissions of your nodes in ACP. It looks like some of your forums have "can't view" for guests.

 

[mazzly]

Sounds to me like there are some redirects missing after migration.. So other sites linking to your site etc are now going to a 404 (Not found) or 403 (Forbidden) pages.. This is not optimal, as you will then be "less relevant" for those keywords from where the links are coming. (which in turn leads to lower rank in search engine results)

For VB3 there was some custom script needed to handle redirects to new "url format", not sure if those are correctly handled by XF when it is migrated from VB4 ?

If you have access to Ahrefs, it might make sense to see if there are a lot of backlinks going to "broken pages" (403/404) on your domain
- I have access, if you PM your forum domain, I can export a list later today

Otherwise, I would install this 404 addon which will find pages where visitors are trying to "come in" (from other sites etc) but are being blocked.. See if you can find some "pattern" over the broken pages.
- If there are many similar ones, I doubt it makes sense to redirect them one by one, but a "programatical solution" would be the way to go

I bought a forum in 2019 that had previously been VB3 -> XF1.5 migrated, and first thing I did was to find old backlinks that went to "broken pages" in Ahrefs, and I fixed the VB3-redirection script to also handle those missing cases.

 

[Alfuzzy]

Everything you need is already there in the GSC. You should make the effort to use the Google help for the GSC to learn how to use it to troubleshoot and then cause Google to re-crawl the faulty things. Then the Google values of your site will get better again

You have a lot to do, so get started ...

To be honest I'm not 100% sure what you're suggesting I focus on in the Google Search Console. I'm sure you know a lot more than me about this...thus maybe I missed something you mentioned previously.

Could you please remind me what you think I should focus on in the Google Search Console?

Thanks

 

[Alfuzzy]

Where the URLs you tested supposed to be visible to guests or not?

11 URLs doesn’t seem like much at all. maybe that’s why it’s such a larger percentage.

After further investigation...turns out all 11 of the URL's classified as "Other Client Error (4xx)"...were either closed/deleted threads...or threads in the private forum Staff area.

What is your robots.txt?

Do you mean "What is in your robots.txt?". If so...here it is. Not really blocking anything major (I don't think). But I would be happy to add or delete anything if it will help.

you should also look at Google Search Console -> Coverage and see if that looks funny.

I do check this pretty often...but since I'm not an expert on this sort of stuff...I'm not always 100% sure how to interpret things.

One section of the Google Search Console (Coverage) I check often is the "Error" section. Right now it's indicating 110 errors...which is wayyyy lower than it was when running vBulletin 4.2.5. Of the 110 errors...largest category is "Blocked by robots.txt" (80)...2nd largest is "Submitted URL seems to be a Soft 404" (22).

Anything else I should look at?

Thanks

 

[Alfuzzy]

As a reminder to everyone reading/helping. I know thread discussions can sometimes branch out into different areas. My main concern at the moment is a possible large traffic drop after migrating from vB4.2.5 to XF 2.1 months ago.

If something can be spotted that's responsible for this...I would love to spot it...and take care of it!

Thanks

 

[rdn]

I would install this 404 addon which will find pages where visitors are trying to "come in" (from other sites etc) but are being blocked.. See if you can find some "pattern" over the broken pages.
- If there are many similar ones, I doubt it makes sense to redirect them one by one, but a "programatical solution" would be the way to go

Install this, and after few days you can sort what is the top 404 pages.

 

[Alfuzzy]

Install this, and after few days you can sort what is the top 404 pages.

Working on that...will do soon. Thanks.

 

[Masetrix]

To be honest I'm not 100% sure what you're suggesting I focus on in the Google Search Console. I'm sure you know a lot more than me about this...thus maybe I missed something you mentioned previously.

Could you please remind me what you think I should focus on in the Google Search Console?

Thanks

My problem is that I cannot help you in the necessary detail because English is difficult for me.
Nevertheless, I realized that you are not yet familiar with the GSC and that I would first have to impart fundamental knowledge to you in order to help you. However, my knowledge of English is not sufficient and the Google translator is still too flawed when translating complicated content.

Therefore, I recommend that you deal with the GSC help. Only when you no longer have to be told where to click to identify the causes of errors in the GSC can I give you help in eliminating the errors.

Btw. you shouldn't forbid /goto/ in the robots.txt to prevent access to postings. The same goes for /members ... /member.php and so on. XF users can set which data can be found by search engines themselves. Doing this via robots.txt costs you a lot of "sympathies" from google.

 

[Alfuzzy]

My problem is that I cannot help you in the necessary detail because English is difficult for me.
Nevertheless, I realized that you are not yet familiar with the GSC and that I would first have to impart fundamental knowledge to you in order to help you. However, my knowledge of English is not sufficient and the Google translator is still too flawed when translating complicated content.

Therefore, I recommend that you deal with the GSC help. Only when you no longer have to be told where to click to identify the causes of errors in the GSC can I give you help in eliminating the errors.

Thanks for trying to help...totally understand.

I'm not totally unfamiliar with GSC...but I'm definitely no expert either. Thus I appreciate it when I can be told where to click & what to do...so I can be 100% sure I'm doing things correctly when someone is helping.

Thanks

 

[Alfuzzy]

Just to give folks an idea of the kind of magnitude we're talking here in terms traffic lost when the site was migrated from vB4.2.5 to XF 2.1 (July, 2020).

Using Google Analytics "Pageview" statistics...comparing monthly Pageview numbers from May & June (site on vB4.2.5)...to monthly Pageviews from August & September, 2020 (site on XF 2.1)...monthly site Pageviews were down more than 50% (and have remained this way).

In addition to this...monthly site revenue dropped significantly. Site revenue is primarily from Ad Views/Ad Impressions...which is directly related to site traffic/Pageviews.

If anyone has suggestions where to look or what to do...I'm all ears!

 

[arn]

you may not want to share your url, but it's hard to debug without knowing the site. Also would help to post or dm screenshots of major google search console views.

 

[Alfuzzy]

...would help to post or dm screenshots of major google search console views.

What GSC screenshots would be helpful?

Thanks

 

[arn]

I'd start with the Settings -> Crawl Stats, Search Results, and Coverage.