[XF 2.0] Google will only index about 30% of XenForo URLs

I don't even worry about it, to be honest. For my busiest forums, we come out near the top when searching for a handful of popular topics on Google, and that is what draws visitors to us. Like any busy forum, many of the threads turn out to be nonsensical in the grand scheme of things. The number of visitors we have certainly never has dropped.

... so just migrated to a new server and changed from Sucuri to CF to see if things settle down.
Sucuri killed the business at one of my client's sites! It gave so many errors to visitors and even Google's page ranking that it drove her into retirement. I had custom-written a catalog system that tied to a third-party cart that worked well and she had her best years on that system. As I got out of coding and all the fancy languages and stuff went way over my head, she moved to Magento (mistake #1) and since it was already being hacked (using Magento is like having a giant "Kick Me!" sign taped to your back), the developers she used pushed her into Sucuri (BIG mistake #2). That company would rarely ever respond to support tickets, and there were hours if not an entire day that something would not be working due to Sucuri, as she lost business from being offline. That and it was sniffing all over our server, going into directories and throwing up all kinds of errors for sites that weren't even online anymore.
 
Maybe this is to stay and is not a bug because i think only affect forum boards , i have wordpress, vbulletin, xenforo, etc... and only the platform boards are affected Vbulletin and xenforo.
 
Maybe this is to stay and is not a bug because i think only affect forum boards , i have wordpress, vbulletin, xenforo, etc... and only the platform boards are affected Vbulletin and xenforo.
Because the big G sees forums as non-authoritative. Having a blog makes it authoritative in G's eyes.
 
People are posting utter nonsense in this thread.

I've seen these boogie man theories for a couple of decades, every time someone's site is down, oh it's Google targeting them, their niche, their software, etc.

The number of pages indexed isn't a measure of how much traffic a site will get. What SERPs show, and where (first three accounts for about 70-90% of click-throughs) is what matters. If traffic from Google is down, and they aren't indexing pages - you have an issue with quality content, topic authority and page rank. And just because you think your site is quality, authoritative, etc. doesn't make it so.

Having a blog does not make a site authoritative in the least. More poppycock. Having quality content at any type of page can lend authority if people are engaged, other authoritative sites are sending it traffic, it's inbound link pattern tends to be authorities and in "good neighborhoods", people are finding answers, etc.

Do you want to engage people, keep them on the site, reduce bounce rates (especially in mobile)? If so, start looking at your site from the point of view of a user instead of a webmaster showing. Especially on mobile, since Google's index is now "mobile first". Let's take for example the CafeSaxophone site. (don't take this personally, this is constructive criticism based on 2+ decades development experience on the Internet)

  • Huge logo and page headers taking half the above the scroll screen space.
  • Then graphic navigation blocks under that.
  • Then an ad.
  • And finally.... you get to see actual content if you scroll down.

People visiting a site generally don't care about how much "wow" is put in front of them when they are looking for answers, interaction, etc. Put content front and center when they visit! You can't see any content without scrolling!

So in 2 minutes, even without taking an in-depth look at your site, I see serious issues which are going to ensure high bounce rates. Google uses that as a ranking signal.

Take a look at Google, Amazon, Facebook, Twitter, Reddit, and so on. They focus on having content in front of you without the design distracting. Their designs are functional.
 
Do you want to engage people, keep them on the site, reduce bounce rates (especially in mobile)? If so, start looking at your site from the point of view of a user instead of a webmaster showing. Especially on mobile, since Google's index is now "mobile first". Let's take for example the CafeSaxophone site. (don't take this personally, this is constructive criticism based on 2+ decades development experience on the Internet)

Huge logo and page headers taking half the above the scroll screen space.
I have done testing in the past, using a small logo/header and made no difference to engagement that I could tell from analytics. And of course if we are talking about mobile which I think is the important one as you say, the logo is small in the navigation bar (and no header image (as per xenforo default)
Then graphic navigation blocks under that.
If you mean the banners, then that is an interface to my (main) other site. Quite important.
Then an ad.
Actually, the two banners should be rotating in a single block, will look into that. Again I've tested with and without ads, no significant difference.

I have to say it's annoying that when responsive on mobile Google changes a horizontal ad into a square which does obviously take up more vertical space. Maybe there is a way to keep it "leaderboard" style.
People visiting a site generally don't care about how much "wow" is put in front of them when they are looking for answers, interaction, etc.

Maybe things changed since I last tested so I will do some more testing with reduced height before content and see what happens. It would be useful to have a different style per usergroup as the forum members do like the big header image, quite a few complained when I removed the picture and the coffee beans background.

But the big issue for me is that Google Console is reporting that many (thread) URLs are blocked by robots.txt when in fact they aren't.

If they are just not indexed due to thin or duplicate content I would be OK with that (although it would be nice if xenforo gave us better SEO tools to deal with those) but to just be given plain wrong info does rankle a bit (page rankle, get it? :) ) If they did fix the bug from earlier, it certainly doesn't seem like it
 
I have to say it's annoying that when responsive on mobile Google changes a horizontal ad into a square which does obviously take up more vertical space. Maybe there is a way to keep it "leaderboard" style.

I have now fixed this by enclosing in a fixed height div. I actually hadn't even realised the ad was expanding so much vertically on mobiles.

Seems too me there is a conflict within Google in that they want engagement from SEO (helped as you say by viewable content without scrolling) and yet the responsive ads by default can expand vertically on mobiles causing the user to scroll for content.
 
Having a blog does not make a site authoritative in the least.
There was a whole update with the author tag years ago. Authors can carry weight from site to site...for a while they said otherwise but looks like from 2011-2016 it carried weight and now reports at the end of 2018 are saying that they are weighing it again.
just to link a couple but there are plenty of results on this

Blogs allow you to utilize the author tag better than forums...in fact, I think you have to have a custom addon if you want to do anything with changing the author tag to "link" your account to "yourself" (as an author).

I've seen many sites where a blog gets better SERPs almost instantly compared to a forum that these people were running for years prior. So now they use their blog (separate site) to push traffic to their forums. For whatever reason, Google seems to like certain site software or interfaces such as blogs/article views.
 
I have done testing in the past, using a small logo/header and made no difference to engagement that I could tell from analytics. And of course if we are talking about mobile which I think is the important one as you say, the logo is small in the navigation bar (and no header image (as per xenforo default)

Did you do this as A/B testing, or a test where everyone sees the same change?

Plus, above the fold is a signal in Google's search algorithms - you can't measure visitors who don't come because Google's not giving you better presence in search results.

This is what Matt Cutts, head of Google Search at the time, had to say about layouts like yours:
"As we’ve mentioned previously, we’ve heard complaints from users that if they click on a result and it’s difficult to find the actual content, they aren’t happy with the experience. Rather than scrolling down the page past a slew of ads, users want to see content right away. So sites that don’t have much content “above-the-fold” can be affected by this change. If you click on a website and the part of the website you see first either doesn’t have a lot of visible content above-the-fold or dedicates a large fraction of the site’s initial screen real estate to ads, that’s not a very good user experience. Such sites may not rank as highly going forward."
 
Did you do this as A/B testing, or a test where everyone sees the same change?
Everyone. I wouldn't know how to do A/B testing.
Plus, above the fold is a signal in Google's search algorithms - you can't measure visitors who don't come because Google's not giving you better presence in search results.

I was just looking out for significant changes in visors and behaviour.

Anyway I'll try again now. I'd prefer to keep the ad at the top, Matt Cutts did mention one ad should not be a problem
 
There was a whole update with the author tag years ago. Authors can carry weight from site to site...for a while they said otherwise but looks like from 2011-2016 it carried weight and now reports at the end of 2018 are saying that they are weighing it again.
just to link a couple but there are plenty of results on this

I am well aware of that. But nowhere does it give any indication "Having a blog makes it authoritative in G's eyes. " Blogs do not build a reputation and authority. Being a reputable author does. If you have crappy content, it doesn't matter how many attributes tags you have on the site's pages to give Google signals. The tag attribute helps Google identify who that authority is and links their content together.

Blogs allow you to utilize the author tag better than forums...in fact, I think you have to have a custom addon if you want to do anything with changing the author tag to "link" your account to "yourself" (as an author).

The rel="author" is not the only way Google determines authors. They also use, among other things, itemscope and data-author attributes both of which are built into Xenforo.

I've seen many sites where a blog gets better SERPs almost instantly compared to a forum that these people were running for years prior. So now they use their blog (separate site) to push traffic to their forums. For whatever reason, Google seems to like certain site software or interfaces such as blogs/article views.

Can you give me an example?
 
Everyone. I wouldn't know how to do A/B testing.


I was just looking out for significant changes in visors and behaviour.

Anyway I'll try again now. I'd prefer to keep the ad at the top, Matt Cutts did mention one ad should not be a problem

Nothing wrong with ads, I run them on my own sites and some clients. If you want, I can PM you some example sites so you can see some of the tweaks I did such as very small header images. Spacing between common screen elements reduced over stock Xenforo values, in order to pull content up a little bit more. "Join" and other action buttons deliberately ugly enough to stand out (lol), without being too garish. Menu bar height reduced, again to pull up content. And more, but those are some of the biggest items. On my own sites over a 2 week period, ad RPM rates increased as Google's smart pricing algorithm kicked in. They've held steady with each month being between 25%-35% higher than the same period the previous.

Depending on the audience, if many of them are over 40, you might want to also consider increasing most font sizes on the screen by 1px. It is a subtle enough change not to impact layout, but makes a big difference in readability as near vision clarity tends to diminish after 40.

And again, please don't take it personally. :) Your design is good, and it can stay good while drawing the user's focus towards the content.
 
Do you still have same issues with Google?

I'm on vBulletin 3.8 and I will upgrade to xF2.1. I hope that I will not have same issues!
 
It doesn't matter what forum software you use. Most of your threads are not going to be indexed because they are of little interest to anyone but your forum members.
 
It doesn't matter what forum software you use. Most of your threads are not going to be indexed because they are of little interest to anyone but your forum members.
Yes but I hope I will not see huge decrease of visitors that comming from Google. Google is the biggest source of visitors for any web site.
 
It doesn't matter what forum software you use. Most of your threads are not going to be indexed because they are of little interest to anyone but your forum members.
Well its about having the same pages indexed as before. You dont want to switch software and found out that that results in 70% decrease. I would be nice to hear from users who recently switched software.
 
When looking at the ones not indexed, they are either thin content or OLD threads. Google wants to see content constantly updated so if a thread goes to the wayside, Google will eventually drop that from their index since that content is now "outdated" in their eyes. Which is sad, because a lot of my "outdated" information is actually still current and people are beginning to search for it again (og Xbox modding). The guides don't change and there has been no new development in the market that we haven't already hit on.
I wonder if "compliant" pages have anything to do with your issue. I just finished reading two posts written by Botify that make a lot of sense and that might help with your situation. Xenforo has lots of 301 redirects and 403 pages for search engines to crawl if you let them. Those pages have the potential to affect how much of the site gets crawled. Check out these posts. The first one is targeted towards classifieds sites, but it's actually quite applicable to many more types of sites than that.



Let me know what you think.
 
Top Bottom