in vb3 I used for about 3 years this mod: http://www.vbulletin.org/forum/showthread.php?t=108070
It was possible to treat search engine bots as members, so that they had access to closed or hidden forums. You could creat an own usergroup for them and it worked pretty well.
Check out the link I hvae posted. Is it possible that anybody could program such an addon for XF? There is a free updated bot (spider) ip list on the web