• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Lack of interest Automatically synch robots.txt with sitemaps

Alfa1

Well-known member
#1
Robots.txt and sitemaps should always be in synch. In other words: if the xenforo sitemap sends a URL to search engines, then it should not be disallowed in robots.txt or it will cause crawl errors.

Conversely: if a directory is not disallowed in robots.txt, then it should not be inaccessible to guests.

Since XenForo has sitemaps built in, it should also synch robots.txt to avoid conflicts between sitemaps and robots.txt and thereby avoid crawl errors.

What I am proposing is not on a url level, but on a directory level. If guests have no access to node_X, then robots.txt should disallow node_X.
Robots.txt should be rebuild every time the sitemap is rebuild.
 

ftsservice

Active member
#2
I respectfully disagree. Such a feature could conflict with other software which has rules in robots.txt or software packages which use a dynamically created robots.txt file to handle various crawlers.
 

Alfa1

Well-known member
#3
I respectfully disagree. Such a feature could conflict with other software which has rules in robots.txt or software packages which use a dynamically created robots.txt file to handle various crawlers.
Good point, but robots.txt rules that fall outside of xenforo could be catered to with a field with additional rules.
And of course such function should have an On/Off toggle.