I've been trying to figure this out for a while. Every day I get a few 404 errors in the apache access_log when googlebot is trying to access robots.txt. This happens to a far lesser extent for other bots such as Baidu, and no problems with bingbot, yahoo, or several others. Is this occurring for anyone else? It appears googlebot successfully downloads robots.txt a few times a day based on the cPanel raw access logs that I've been going through, but I cannot understand where the 404 errors are coming from in the apache access log... I've no idea where the apache access log is traditionally kept, but on our account it is at /usr/local/apache/logs/access_log That file is a bit large... .