Ryan Kent
Well-known member
I have an RSS feed which automatically posts in a forum. This forum is used exclusively for this feed. I want to block this forum from being crawled. What is the best method?
I tried adding the path to the node to my robots.txt file and that did not help. I realize the path to all threads is /threads. Can I use a wildcard with robot.txt inline?
What I mean is all the threads begin as:
www.mysite.com/threads/tweet-from-
Can I add www.mysite.com/threads/tweet-from-* to the robots.txt file?
I tried adding the path to the node to my robots.txt file and that did not help. I realize the path to all threads is /threads. Can I use a wildcard with robot.txt inline?
What I mean is all the threads begin as:
www.mysite.com/threads/tweet-from-
Can I add www.mysite.com/threads/tweet-from-* to the robots.txt file?