XF 2.0 SEO Problems

nodle

Well-known member
Don't feel bad. I went though the same thing as you, I messed with everything that I could. I tried to research the errors as well and basically get a generic answer from Google on it. Honestly the only thing that I can narrow it down to is Google just doesn't like your content.

Screenshot_2019-02-20 Coverage.png
 

djbaxter

Well-known member
You appear to have additional rewrite rules that are not part of the normal XF .htaccess:

Code:
RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
RewriteCond %{REQUEST_URI} !^/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge/[0-9a-zA-Z_-]+$
Ok, I removed those 3 lines. They're inserted by cPanel automatically.

Hopefully I can get more help on figuring out the rest of my problem.
There's no point in blindly making changes without evidence to justify making those changes. Why are they added by cPanel? What do they do? They might be important.

The information as to what makes up the 60K+ is right there. The majority of them are listed as "Crawled - currently not indexed". I'm not certain but I believe you should be able to drill down into each of these down to specific URLs.

It's just going to be a case of going through the different error categories, trying to decipher what those Google messages are trying to tell you, and also analysing any of the URLs to see if there is anything pertinent that can be improved.

There's not going to be any silver bullet nugget of advice to give you on how you can fix it, and in some cases, there might not even be anything you can do. From what I've read, the "Currently not indexed" error is fairly generic and essentially boils down to "Google's prerogative".
You probably want to put those lines back.

Yes, they are inserted by cPanel when you enable the AutoSSL domain security certificate by Comodo.
 

djbaxter

Well-known member
Curious,

So because my robots.txt was missing for however long and I just replaced it an hour ago, do you think this will remove a good portion of the "crawled but not indexed" warnings and improve the SEO?

I found another issue in "coverage" area of the search console in Google:

URL is not available to Google
It cannot be indexed.
Page fetch - Failed: Crawl anomaly

Same with many other member links. Is that normal?

Again, this is my robots:

Code:
User-agent: *
Disallow: find-new/
Disallow: account/
Disallow: attachments/
Disallow: goto/
Disallow: posts/
Disallow: login/
Disallow: admin.php
Allow: /

Sitemap: https://www.talkjesus.com/sitemap.php
Why would you disallow posts?
 

Chris D

XenForo developer
Staff member
I made an edit to ours shortly after it was posted. As long as it roughly matches that, I’d say it’s ok.
 

Chad

Active member
I made an edit to ours shortly after it was posted. As long as it roughly matches that, I’d say it’s ok.
Thanks. Updated mine to the below since my installation is in root, not "community".

Code:
User-agent: *
Disallow: whats-new/
Disallow: account/
Disallow: attachments/
Disallow: goto/
Disallow: posts/
Disallow: login/
Disallow: admin.php
Allow: /

Sitemap: https://www.talkjesus.com/sitemap.php
Is the end trailing slash needed or is this correct?
 

usAdultAds

Active member
Don't feel bad. I went though the same thing as you, I messed with everything that I could. I tried to research the errors as well and basically get a generic answer from Google on it. Honestly the only thing that I can narrow it down to is Google just doesn't like your content.

View attachment 196181
I have the same issue, however, let's say you have 5 people having the similar discussions over and over, Google is going to choose what it thinks is the best content based on several factors, including engagement, then Google will ignore the rest. Even 5 people can create 100s of posts in a short period of time, but that does not mean Google will keep each and every one of those posts, as some or most could be similar in nature. There is another issue that forum has to deal with also, and that is called "thin" content, if the post appears too shallow, then Google may ignore those posts. 3 months ago, I had 10,000 results for my forum, today 2,300 results, and every blog post I have made is still indexed.
 
Last edited:

Chad

Active member
Just saw this in Google Search console. So apparently around end of December it started dropping steeply. I don't understand why.


196280
 

Ludachris

Well-known member
Just saw this in Google Search console. So apparently around end of December it started dropping steeply. I don't understand why.


View attachment 196280
I know this is old but I wanted to chime in to say that Chad, the drop in coverage may not have been due to anything you did on your end. It could have very well been an algorithm update on Google's side that negatively impacted your rankings/indexing. In order to try and find out what the impact is you have to research their algo updates and see if there is any info that you can find about it, and then see if you can remedy the situation. It's not always easy, and in many cases you simply have to accept it and find ways to improve elsewhere.

As for having a big chunk of pages not indexed, I'll say from experience that this can be pretty normal. Google is known to not always index all pages on a website. They focus on what they feel is the best and most "valuable" content on your site based on their algorithm. As someone said above, if Google feels that a lot of threads are focused on the same topic (repetitive in nature) the algo might decide to choose one thread to represent several of them in the search results. This is not only common on forums but also e-commerce sites where a lot of products have very similar naming schemes and descriptions. But there are other reasons the algo omits pages besides "duplicate content" too. Again, you have to keep up with all of the algo changes - and unfortunately, since they don't publish what has been changed, you have to rely on "SEO experts" who publish articles on what they think changed.

It's frustrating to hear this, but sometimes there truly is very little you can do to counteract these indexing and ranking problems. You just have to make sure you have all the technical settings right on the back end, make sure Google can spider the site effectively, make sure you're not violating any obvious rules, and then just focus on good content.
 
Top