It would be nice if the XenForo HTTP Client which is being used for things like getting images for image proxy, URL titles, unfurling, etc. would obey
robots.txt so site owners who do not want to have content fetched by XenForo could block it in a friendly & way.
Last edited:
Upvote
6
