Page indexing issue from google

[Alvin63]

When I mentioned to the server I was confused about the whole .htaccess thing when my file is htaccess.txt they just replied "It should be .htaccess could you try renaming it".

I think that's what Brogan suggested if there were issues - renaming it - but I'd quite like to understand what it is and what I'm doing! Don't like messing with things in the file manager! Also what changes would it make?

 

[Old Nick]

Your forum is brand new, don't worry too much about all these statistics for now everything looks normal. Check for 4xx errors by conscience but otherwise focus on the content of your forum and everything will be fine!

 

[Alvin63]

I have seen a few posts on here recommending changing htaccess.txt to .htaccess to activate it. I am just wondering why Xenforo install it as htaccess.txt in the first place!

 

[Alvin63]

Your forum is brand new, don't worry too much about all these statistics for now everything looks normal. Check for 4xx errors by conscience but otherwise focus on the content of your forum and everything will be fine!

Thanks. Yes but those pages should be being crawled shouldn't they?

 

[Old Nick]

I don't know LSWS well but with Apache when you activate friendly URLs you have to rename the .htaccess file. Apparently LSWS supports this type of file but I don't know if it needs it according to your forum settings. Wait for a more clear and precise answer in the meantime.

I have seen a few posts on here recommending changing htaccess.txt to .htaccess to activate it. I am just wondering why Xenforo install it as htaccess.txt in the first place!

Because some server doesn't need it.

Thanks. Yes but those pages should be being crawled shouldn't they?

They will be, Google does not only have your site to crawl.

 

[Alvin63]

Ok. Well as the server has said "try changing it to .htaccess - perhaps I should?

I realise Google has a lot of pages to crawl I am just going by my last forum which was crawled by google constantly from within the first couple of weeks!

 

[Old Nick]

Since he asks, do it.

 

[Alvin63]

Just checked file manager on my other forum, that's been going for 18 months and gets crawled regularly, and that one is still htaccess.txt and never had an issue! Aargh I think I'll sleep on it.

 

[Alvin63]

Aha - robot google has appeared. It only seems to crawl the articles page, not the forum

 

[Paul B]

You are unnecessarily stressing and worrying over something you have no control over.

As long as your robots.txt file looks like this there will be no issues:

User-agent: PetalBot
User-agent: AspiegelBot
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: DotBot
User-agent: MauiBot
User-agent: MJ12bot
Disallow: /

User-agent: Amazonbot
Disallow: /threads/*/reply

User-agent: *
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /login/
Disallow: /misc/language
Disallow: /misc/style
Disallow: /posts/
Disallow: /search/
Disallow: /whats-new/
Disallow: /admin.php
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Google will crawl and index as and when it sees fit.

New sites take time.

The .htaccess file is unrelated.
It is only required for friendly URLs on an Apache server.

This is all too common with new site owners.
Focus on the site and the content - the rest will fall into place.

 

[Alvin63]

Thank you Brogan for clarifying. Although my server last said I should have a .htaccess file so maybe I should change it to that. I haven't so far (as hadn't on my previous forum). I can't actually find the robots.txt file to check the above! Tried putting in the search box on the acp

 

[Old Nick]

robots.txt file is in the same place as your .htaccess file on the server.

Although my server last said I should have a .htaccess file so maybe I should change it to that.

It is only required for friendly URLs on an Apache server.

 

[TPerry]

I have seen a few posts on here recommending changing htaccess.txt to .htaccess to activate it. I am just wondering why Xenforo install it as htaccess.txt in the first place!

to work in default installs (including LiteSpeed) it HAS to be named .htaccess. Unlike OpenLiteSpeed (unless it has changed), LiteSpeed uses the same process as Apache to read the .htaccess in whatever directories it is in (it can be in more than the root to control specific needs for those directories).
robots.txt is a text file located in your vhost root directory.

 

[Mr Lucky]

When I mentioned to the server I was confused about the whole .htaccess thing when my file is htaccess.txt they just replied "It should be .htaccess could you try renaming it".

When you say you mentioned it to the server, do you mean your host tech support? What they probably mean is that if you want it to function, then unless you rename it as said it will be doing nothing. They would know whether you need it or not.

I have seen a few posts on here recommending changing htaccess.txt to .htaccess to activate it. I am just wondering why Xenforo install it as htaccess.txt in the first place!

because it it may or may] not be required. I found that redirects worked without it, so I just left it inactive instead of renaming it.

I can't actually find the robots.txt file to check the above! Tried putting in the search box on the acp

It is nothing specifically to do with xenforo, it is a text file that can be added to any website file system and purely tells robots what to index or not.

 

[Alvin63]

Thanks all. Yes I meant host tech support. As it seems related to telling robots what to index or not and Google is showing my forum pages aren't being indexed then perhaps I should rename it and activate it.

On my other, 18 month old, forum though, I didn't rename it and have no issue and plenty of google bots!

The only difference I can think of is the other forum had cloudflare set up automatically. The server now doesn't do this automatically and says you need to set up cloudflare yourself on the cloudflare site. Just wondering how important it is to have that. If setting up cloudflare myself it changes the DNS names from the Hostinger ones to the cloudflare ones so don't want any glitches!

 

[Mr Lucky]

then perhaps I should rename it and activate it.

The stock xenForo .htaccess haș nothing to do with Google indexing. You only need it if friendly URLs don't work without it (renamed and active)

 

[Alvin63]

The stock xenForo .htaccess haș nothing to do with Google indexing. You only need it if friendly URLs don't work without it (renamed and active)

Thank you

 

[Alvin63]

Ok I've checked this out and Google is just not indexing my forum page in its entirety. It is crawling it but not indexing it. It's indexing my home page (which is where my domain address leads) - the home page is Bobs AMS. But it's not indexing anything at all on the forum page. I've run analytics and speed insights and can't see why. This is annoying and perplexing (especially when I'm paying for google ads.

Any suggestions? Just wondering if I set the domain to go straight to the forum instead of the home page, if it might make a difference.

 

[Mr Lucky]

What do you mean by:

not indexing my forum page in its entirety

Surely it is either indexed or not indexed.

Even so if not indexed I would not worry as is just a list of links with little actual content. As long as Google is indexing useful content such as threads, based on keyword searches that’s what is important.

There won’t really be that many searches that would get the forum list anyway.

It's indexing my home page

So that’s good, more important IMO.

BTW what is the forum URL? It’s easier to help if we know the context.

 

[Alvin63]

The forum url goes to the home page. That is being indexed and shows on Google - this is an articles page basically. The url /forums is not being indexed in its entirety - nothing for that page, no threads, no posts.