• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

how long until bots respect new robots.txt entries?

king8084

Active member
#1
my forum gets crazy /poll/ index and /members/ as well as /best-posts/ which is an add on

i added them to disallow 2 days ago but still they getting indexed

usually how long does it take?
 
#2
From google faq...

First, the cache of the robots.txt file must be refreshed (we generally cache the contents for up to one day). Even after finding the change, crawling and indexing is a complicated process that can sometimes take quite some time for individual URLs, so it's impossible to give an exact timeline. Also, keep in mind that even if your robots.txt file is disallowing access to a URL, that URL may remain visible in search results despite that fact that we can't crawl it. If you wish to expedite removal of the pages you've blocked from Google, please submit a removal request via Google Webmaster Tools.
Also, robots.txt won't prevent references to your URLs from other sites.