• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Google Crawl and 'No Follow' Settings

Core Freedom

Well-known member
#1
I have a question, if I may. One of my members just sent me the following message:

"XenForo tags all user posted links as "nofollow" which means their is no search engine benefit"

Yesterday I found a page in the Google webmaster tools area where it said that my site is not currently getting crawled. I downloaded and then uploaded a robot.txt file onto the 'highest level file' of my site but not knowing what the highest level file is I tried placing it into the public_html folder but that didn't work; then I tried placing it 'next to' the public_html folder but that didn't work either. And now I can't find that page on Google webmaster anymore. :-(

Obviously I do want my pages and posts get crawled by Google. And I do want to set my XF tags to be set as 'follow.' Does anyone know what I'm doing wrong?
 

Core Freedom

Well-known member
#3
oh cool, I didn't know that their links were a no-follow. Does that include their links in their signatures?

I'm not sure what the domain root is. I thought that's where I placed it, but apparently not. When I have Filezilla open in the right window I see various files, including the public_ftp, then public_html, and somewhere further below I placed this robots.txt file. It didn't work, so then I placed it inside the public_html file but that didn't work either.

Even if I placed it now, I don't know how to get back to that Google page to 'verify' that I actually placed the file. Is that important or will Google figure out automatically that it's there?

Thanks for your help, Brogan.
 

Core Freedom

Well-known member
#5
Yes, I log into the Google analytics account every day, but what I didn't know was about the robots.txt file. The Google page asked me yesterday to verify after placing the file in my root folder, which I did, but Google said there was an error. That's why I'm confused and not sure if my site is actually getting crawled or not. :)
 

MikeMpls

Well-known member
#7
Cool, this probably explains why my XF forums haven't been plagues by forum spammers anywhere nearly as much as as my vB forums.

(insert thumbs up smilie here)