Rigel Kentaurus
Well-known member
Could you try this in Google WebMaster Tools and see if it complains about anything?The generated robots.txt has some validation errors.
Go to http://tool.motoricerca.info/robots-checker.phtml
Type: http://www.controlbooth.com/robots.txt
There are several errors regarding wildchar "*" character usage, and directories that don't use a trailing slash character.
An interesting error is apparently there is a specific way you should do line breaks for robots.txt files:
ERROR: This file uses one or more "NL+CR" line breaks (New Line followed by Carriage Return). This is wrong. You should write the robots.txt file with a text editor that generates lines terminated by CR,CR+NL, or NL.
Who knew?
http://www.google.com/webmasters/ under Crawl -> Blocked Urls
If there are no errors about robots.txt I would not read too much into that. The robots.txt file has different semantics (technically Allow directive and wildcard is not permitted, but in practice all engines recognize them)