Lack of interest XenForo HTTP Client should obey robots.txt

This suggestion has been closed automatically because it did not receive enough votes over an extended period of time. If you wish to see this, please search for an open suggestion and, if you don't find any, post a new one.
K

Kirby

Well-known member
Licensed customer
It would be nice if the XenForo HTTP Client which is being used for things like getting images for image proxy, URL titles, unfurling, etc. would obey robots.txt so site owners who do not want to have content fetched by XenForo could block it in a friendly & way.
 
Last edited:
Upvote 6
This suggestion has been closed. Votes are no longer accepted.

Similar threads

Mr Lucky
  • Suggestion Suggestion
Lack of interest Option for sitemap to obey robots.txt
Replies
0
Views
987
Mr Lucky
Mr Lucky
Back
Top Bottom