Adding Lines to Robots.txt to Block Specific URLs

A

Alfuzzy

Well-known member
Hello all.

When my site was migrated from vB to XF...some custom member badge images came along for the ride. When I scan the site for errors...these badge image file URLs are showing up quite a bit in the scans. I would like to block them from the Google crawler. A couple other misc URLs thrown in as well I would like to block.

Here are the URLs:



I did some research on how to block things in the robots.txt. Kind of new to this...some of it was confusing. Want to get it right...so I don't end up blocking lots of possibly important stuff.

What lines would I need to add to block each of the 5 URLs above in the sites robots.txt? Just want to block these URLs. If each one requires a separate line in the robots.txt...that would be ok.

Thanks for the help:)
 
Adding entries to robots.txt doesn't prevent the URLs from being accessed.
Some crawlers abide by robots.txt, others don't.

If you want to prevent the URLs being accessed at all, use a rewrite rule in .htaccess (or the equivalent).
 
Yes I heard some crawlers don't obey robots.txt (Google crawler I read is supposed to follow robots.txt). Since Google is the "big-one"...I figured at least blocking Google for these URL's would be a good place to start.

I think working with rewrite rules is a little bit beyond my skills at this point. Robots.txt I understand a bit better...and figured this was at least a good place to start (rather than not doing anything at all). Lol

Thanks:)
 
Alfuzzy said:
.I figured at least blocking Google for these URL's would be a good place to start.
Click to expand...
Just keep in mind robots.txt doesn't block anything. Think of it as just a note saying "please don't index the following." Also, anything you include in robots.txt is public.
 
Max Taxable said:
Just keep in mind robots.txt doesn't block anything. Think of it as just a note saying "please don't index the following." Also, anything you include in robots.txt is public.
Click to expand...
Ohh yes understand all of that (including the public part).:) My understanding Google is supposed to obey robots.txt...and as we all know (at least at the moment)...Google pretty much runs the internet. Lol
 
Muddy Boots said:
Here is something to get you started with for .htaccess
Click to expand...
Hello Muddy Boots. Thanks very much for the help!:)

  • Do I follow the same format for the other 2 lines I mentioned I wanted to block?
  • Do these lines need to go somewhere special in the .htaccess?
  • I know my .htaccess already has some rewrite rules in it...how do I incorporate these lines without messing anything up (I know the "Rewrite Engine on" part is already definitely there)?
  • I guess maybe I get "rewrite rules" and "redirects" mixed up. How do these rewrite rules "block or hide" things from crawlers?
  • Is there a way I can test how rewrite rules work...to make sure I'm doing things correctly?

Apologies for all the questions...trying to learn a little bit at a time.

Thanks much!:)
 
Alfuzzy said:
Ohh yes understand all of that (including the public part).:) My understanding Google is supposed to obey robots.txt...and as we all know (at least at the moment)...Google pretty much runs the internet. Lol
Click to expand...
It's considered one of the "noble" crawlers that can adjust according to the robots.txt file. It's not a "supposed to" thing it's more of a "supposedly does" thing. It doesn't always happen. Anyhow, use of the word "block" regarding robots.txt is misleading for the uninitiated.
 
Max Taxable said:
It's considered one of the "noble" crawlers that can adjust according to the robots.txt file. It's not a "supposed to" thing it's more of a "supposedly does" thing. It doesn't always happen. Anyhow, use of the word "block" regarding robots.txt is misleading for the uninitiated.
Click to expand...
Appreciate the opinions & clarity of statements for the less IT experienced.:)

Any chance I can get some help with my request in post #1 above please?

Thanks
 
Alfuzzy said:
Appreciate the opinions & clarity of statements for the less IT experienced.:)

Any chance I can get some help with my request in post #1 above please?

Thanks
Click to expand...
Post numbers 2 and 5 are your huckleberry. Google can help too. Working with .htaccess or its equivalent isn't difficult to self teach. And if the results are wrong or objectionable, it's not permanent. You can't do irreparable harm. You can erase and start over.
 
Can someone help with my request in post #1 please?

Replying via Conversation/PM is perfectly cool too.

Thanks much:)
 
So, to me it is not clear what you want to do. If you don't want those URLs indexed in search engines, posting them here sort of defeats that purpose. You can add them to your robots.txt file but you're still going to find them indexed - maybe not by google itself. It's a dice roll.
 
Max Taxable said:
So, to me it is not clear what you want to do. If you don't want those URLs indexed in search engines, posting them here sort of defeats that purpose. You can add them to your robots.txt file but you're still going to find them indexed - maybe not by google itself. It's a dice roll.
Click to expand...
I wanted to add disallow lines to the robots.txt to prevent Google from crawling & indexing the URLs mentioned in post #1.

My apologies if this was unclear. Thanks
 
Alfuzzy said:
I wanted to add disallow lines to the robots.txt to prevent Google from crawling & indexing the URLs mentioned in post #1.
Click to expand...
I agree with max taxable. Google will crawl and index them via this site unless you can get xenForo to add them to their robots.txt ! Or redirect or delete / edit the post.

I would remove the links from the site, and remove the images from the server
 
Last edited:
Mr Lucky said:
I agree with max taxable. Google will crawl and index them via this site unless you can get xenForo to add them to their robots.txt !
Click to expand...
Please look at post #1 carefully...the URL is www.example.com. My website is NOT example.com! That's just a placeholder for the actual URLs I'm working with. The format of the examples in post #1 are correct for what I'm working with (example.com is used instead of the actual URL).
Mr Lucky said:
I would remove the links from the site, and remove the images from the server
Click to expand...
The images are in use on the website (I don't want to delete them). I just don't want Google to crawl them.

Please guys...I appreciate the opinions & extra information...but really not looking for a protracted conversation...just want a solution to my request in post #1.

I'm looking for the proper format to add "disallow" lines to my robots.txt for the URLs listed in post#1.

Thanks:)
 
When you said:

Alfuzzy said:
When my site was migrated from vB to XF...some custom member badge images came along for the ride.
Click to expand...

That implied you didn't want them, hence I suggested deleting them.

Alfuzzy said:
I'm looking for the proper format to add "disallow" lines to my robots.txt for the URLs listed in post#1.
Click to expand...
How about:

Code: 
User-agent: *
Disallow: /forums/images/

(each disallow on a new line)

or

Code: 
User-agent: *
Disallow: /forums/images/red_badge.gif
 
Hello Mr Lucky. Yes...that's pretty much what I was looking for.:)

Pretty easy I guess. Wanted to be 100% certain I got the syntax correct. I did some internet research on this before posting the thread...and some sites seemed to be making it overly complex to add disallow lines to robots.txt for each of the URLs mentioned in post #1.

If I follow the same format you gave...would the following be correct for all 5 URLs mentioned in post #1?:

User-agent: *
Disallow: /forums/images/red_badge.gif
Disallow: /images/badges/Banned.png
Disallow: /forums/images/badges/SeniorMember.png
Disallow: /forums/search.php?do=getdaily
Disallow: /forums/register.php


Also just to be sure...do I only need to add the User-agent: * line once for all 5 disallow lines?

Thanks very much for the help sir!:)

p.s. Here are the URLs from post #1 for easy reference:

https://www.example.com/forums/images/red_badge.gif
https://www.example.com/images/badges/Banned.png
https://www.example.com/forums/images/badges/SeniorMember.png
https://www.example.com/forums/search.php?do=getdaily
https://www.example.com/forums/register.php
 
You must log in or register to reply here.

Similar threads

JoyFreak
Adding Disallow: /attachments/ to robots.txt
Replies
10
Views
1K
JoyFreak
JoyFreak
JackieChun
  • Question Question
XF 2.0 Changes to robots.txt from Xenforo 1.5 to 2.0
Replies
8
Views
4K
djbaxter
D
Diana Cox
  • Question Question
XF 2.0 User wants to block threads from one or more specific forums from their view
Replies
5
Views
3K
Diana Cox
Diana Cox
Sadik B
  • Question Question
RM 1.1 Add Download Block to robots.txt for Resource Manager
Replies
0
Views
1K
Sadik B
Sadik B
Back
Top Bottom