XF 1.1 robots.txt

tekgirl

Active member
I have read quite a bit about the robots.txt file and still am confused about what files/directories shold be disallowed. Can someone help?

Thanks
 
I know this is an old thread, but wonder if that robots.txt file is complete....well, maybe some depends on add-ons!

But I am showing a vast number of google crawl errors on conversations.....

Also showing on bookmarks, but - yes- that's an add-on.

Strangely enough, the standard directory block doesn't seem to stop the conversation errors, so I am now trying a more complete URL like:
Disallow: /talk/conversations/add?to=*
Disallow: /talk/bookmarks/*
I also blocked the conversation direction in another line....but most of the crawler goog errors have that other text after them.
The second line is for bookmarks (add-on).
I will update this message after I watch the goog crawl errors for a week or two.
 
I usually disallow when I see things in GWT that shouldn't be there (like a year ago I saw my styles folder in phpBB was indexed). On top of that I have disallowed quite a bit more of the default stuff:
Code:
Disallow: /account/
Disallow: /admin.php
Disallow: /ajax/
Disallow: /attachments/
Disallow: /conversations/
Disallow: /data/
Disallow: /forums/-/
Disallow: /forums/tweets/
Disallow: /goto/
Disallow: /help/
Disallow: /internal_data/
Disallow: /js/
Disallow: /library/
Disallow: /login/
Disallow: /lost-password/
Disallow: /misc/contact/
Disallow: /members/
Disallow: /online/
Disallow: /recent-activity/
Disallow: /register/
Disallow: /posts/
Disallow: /search/
Disallow: /styles/
Allow: /

In addition I have disallowed my Intro forum, cause allot of the threads there are just repeating them self.
 
Yes, any attachment will not be indexed then, I don't remember if the ones inserted wont be indexed, I know the thumbnails are directly hotlinked, the inserted ones I am not sure about.
 
Yes, any attachment will not be indexed then, I don't remember if the ones inserted wont be indexed, I know the thumbnails are directly hotlinked, the inserted ones I am not sure about.

Google and Bing images are quite important in SOME searching, so folks should be careful about disallowing attachments if that is true.

In our case, we want goog to get the images and associate them with what is in the text nearby (which I think is how they do some of their categorization)
 
I don't see any traffic from image search, and my site isn't image heavy, so I decided to save bandwidth instead. If you are getting traffic from image search, it might be a different story.
 
How did you check your google stats for images? The regular report doesn't have them, but after searching around on google I saw an article which pointed to webmaster tools where you can at least see queries and clicks on your images searches.....
If the enclosed pic is anywhere near correct, most forums will not want to disable attachments!
Screen Shot 2013-01-27 at 5.49.18 PM.webp
 
None of my top ranking threads contain any images, and as I said, my site isn't image heavy (a total of 150 ish attachments, and allot of them is me posting screenshots on new site features). As I said, if image search is important to you, allow them. Be sure to allow guests to view attachments, cause that is why I blocked it (to prevent those 403 errors in GWT).
 
Top Bottom