Robots.txt and sitemap questions

[Alvin63]

Ok so I asked AI and it suggested a combination of Cloudflare Zero trust, plus editing htaccess with deny all except my ip address (rather than limiting the server to only allowing Cloudflare IP addresses). Does that sound like a plan?

 

[Alvin63]

Also, I still have this zip file in Public_html at the bottom of the list - when I first started the site I hadn't a clue how to upload it to the server, so the server did it for me or walked me through it - can't remember. And I think it involved unzipping the file within public_html. I assume that shouldn't still be there! Is it ok just to delete it?

 

[RichDevman]

I havent followed this post so not sure if this has all been covered but you can regenerate a sitemap using a cron task and you take the url for the sitemap and upload it to GSC-Google Search Console. It will be generated automatically.

Regarding robots.txt. It is very important for forums due tocrawl budget especially on large forums. We dont want google wasting time crawling portions of the site that are not useful to search.

 

[bzcomputers]

Apparently if I secure admin.php and /install via Cloudflare Zero Trust, then I need to ensure that only Cloudflare IP addresses pass through the server. Is that right?

No.

Otherwise someone could bypass Cloudflare to do something. But they also said there are downsides and limitations to restricting all IP addresses to the server going through Cloudflare.

No. We are just addressing securing two sections of your website and nothing more here.

Ok so I asked AI and it suggested a combination of Cloudflare Zero trust, plus editing htaccess with deny all except my ip address (rather than limiting the server to only allowing Cloudflare IP addresses). Does that sound like a plan?

No. To secure these 2 locations of your site you don't need to be limiting the server to only allowing Cloudflare IP addresses. That has no bearing on this at all. As for .htaccess you could do a similar ip only limitation through .htaccess but nothing to do with email addresses. If you setup the security for /admin.php and /install correctly through Cloudflare Zero Trust there is no need to mess with .htaccess at all.

 

[bzcomputers]

Also, I still have this zip file in Public_html at the bottom of the list - when I first started the site I hadn't a clue how to upload it to the server, so the server did it for me or walked me through it - can't remember. And I think it involved unzipping the file within public_html. I assume that shouldn't still be there! Is it ok just to delete it?

View attachment 323131

Yes, it is safe to delete this file.

 

[Alvin63]

No.


No. We are just addressing securing two sections of your website and nothing more here.


No. To secure these 2 locations of your site you don't need to be limiting the server to only allowing Cloudflare IP addresses. That has no bearing on this at all. As for .htaccess you could do a similar ip only limitation through .htaccess but nothing to do with email addresses. If you setup the security for /admin.php and /install correctly through Cloudflare Zero Trust there is no need to mess with .htaccess at all.

Thank you. So there is no risk from someone getting into the server bypassing Cloudflare then?

 

[Alvin63]

Yes, it is safe to delete this file.

It didn't like that at all! When I went to delete it - big red warning and said you don't have permission to access this and everything disappeared. Reloaded file manager and it all seemed normal but the zip file still in there. I think I'll just leave it there if it's not a security risk!

 

[bzcomputers]

Thank you. So there is no risk from someone getting into the server bypassing Cloudflare then?

No risk? We're dealing with the internet here, there is no such thing.

Your site has sat for years without either of these sections secured, without issue. You are now adding another layer of security through Cloudflare to better improve the robustness of your site.

Can Cloudflare be bypassed? Yes, it is possible, but you will always have your username/password combination as the first layer of security as a deterrent, the same way your site was secured before adding any of this.

If you are worried about Cloudflare being bypassed you can add additional security through .htaccess (directly on the server). Just do a search for securing files and directories with .htaccess on Google it is pretty straight forward.

This is getting way off topic to the original thread topic. You should do some searches for site security and/or start a new thread if you want additional site security help.

 

[Alvin63]

Yes it is getting off topic. Thank you for the help.

 

[bzcomputers]

It didn't like that at all! When I went to delete it - big red warning and said you don't have permission to access this and everything disappeared. Reloaded file manager and it all seemed normal but the zip file still in there. I think I'll just leave it there if it's not a security risk!

It's just a permission issue. The file is safe to delete.

The file is currently residing in public_html so there is a risk of someone accessing and running that file - you really should get it deleted when you can.

 

[Alvin63]

It's just a permission issue. The file is safe to delete.

The file is currently residing in public_html so there is a risk of someone accessing and running that file - you really should get it deleted when you can.

Thank you for the tip about permissions. Done.

 

[Alvin63]

Just as an update. Since doing the robots.txt it's gone from about 56 bots/guests at any given time to 180. Most of whom are guests rather than robots as I disallowed some of the robots. Also noting that Anthropic ignores robots text (it followed it for about two days and now it's back).

 

[JoyFreak]

Won’t adding “Disallow: /attachments/“ block robots such as Google from indexing images?

 

[Alvin63]

I think it does, yes. I left it out. Because google snippets use images.

 

[JoyFreak]

I think it does, yes. I left it out. Because google snippets use images.

Yeah, I would definitely keep attachments out from your robots.txt especially if you SEO those images and want them indexed. I tested this for 48 hours and created threads with attachments and none of those images were indexed as oppose to before without the disallow. Also, for Google Carousel or News, those images won’t show up otherwise.

 

[Alvin63]

This was my final version: However Anthropic is ignoring it. I've been quite alarmed at the high increase in bots crawling since adding the robots.txt. The recognised bots are just the same - so I should say a high increase in "guests" who are probably bots. It's tripled. Hence now looking at site security! I guess it depends what kind of images are on your site. Some may not want them appearing all over the place. Mine are just of pets.



User-agent: AspiegelBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: ImageSift
Disallow: /

User-agent: AnthropicBot
Disallow: /

User-agent: *
Disallow: /admin.php
Disallow: /account/
Disallow: /goto/
Disallow: /login/
Disallow: /register/
Disallow: /search/
Disallow: /help/
Disallow: /members/


Sitemap: https://www.xxxxxxxxxxxxxx.com/sitemap.xml

 

[ENF]

so I should say a high increase in "guests" who are probably bots

Would recommend you to use known bots, if you want to know what bots are actually coming.
Probably the best option to keep the bots data updated. It does nothing more than identify and tell you what bots (that are known) are coming to your site.

 

[Alvin63]

Thanks! I'll check that out. That might scare me Seeing what they are.

 

[Alvin63]

Following on - Google search console didn't seem that happy that I had disallowed "goto". It had an amber exclamation mark warning and says

"Indexed, though blocked by robots.txt
This is a non-critical issue. Pages with these issues are indexed, but their search result appearance can be improved"

The links in that section were all go to links. Clicking on one just brought up a thread page though. Not the goto link itself. So presumably it's because some things were already indexed and then stopped being indexed?

 

[bzcomputers]

Following on - Google search console didn't seem that happy that I had disallowed "goto". It had an amber exclamation mark warning and says

"Indexed, though blocked by robots.txt
This is a non-critical issue. Pages with these issues are indexed, but their search result appearance can be improved"

The links in that section were all go to links. Clicking on one just brought up a thread page though. Not the goto link itself. So presumably it's because some things were already indexed and then stopped being indexed?

Yes, you should keep /goto/ in the list of disallowed. The error is because some /goto/ links were already indexed, prior to you having a robots.txt and telling search engines not to index those /goto/ links. /goto/ links are duplicate content of the /threads/ links. Search engines should only index the /threads/ links. These errors are non-critical and you will likely see them for a long time. You may also see duplicate content warnings popup also, again non-critical.

When you have even more content the chance of you seeing even more non-critical indexing issues goes up. I've got thousands of /members/ pages indexed that Google still has not removed even though I have had them listed as disallowed in robots.txt for years. Again just non-critical errors.