Xenforo and Google
- Thread starter Markos
- Start date
Chromaniac
Well-known member
haha. everyone would have a different point of view on this. i would not like to have my 40,000 registered accounts indexed (majority of them are not actively participating on the forum of course) because that just add clutter to my google site search results. other folks might be more interested. i guess i can always use robots.txt file to block their indexing.
Anthony Parsons
Well-known member
That is there only real purpose, being that you can use them to not vote for a site you need to link towards, ie. say your talking about something on your general blog, and it refers to a porn site, hate site, etc. Now linking to such things is not essentially a good thing, because you are in essence voting for that site if you link to it. But for your users, it is easier for them to just click the link... so the nofollow tag was created and implemented for this purpose. You can make your readers life easy, but not lose your sites trust or credibility from a mathematical viewpoint, which is all an algorithm views your actions as... a 1 or 0.
People just completely confuse it with thinking all internal links should have a nofollow and that will fix things. In essence, all that does is stop their internal page structure voting for itself... it doesn't stop a bot following and indexing the page its pointed at. The only aspect a software should have builtin for nofollow, is to automatically be applied to everything external that is posted... so the site isn't punished for anything at the other end of links.
Which is exactly what the point of this subject was raised... being the thought of nofollow being implemented to stop bots indexing less relevant parts of the site... is incorrect. Only a robots.txt can accomplish that.
Anthony Parsons
Well-known member
The only thing you can do is use a robots.txt file to accomplish what you're asking... nofollow DOES NOT accomplish what you're otherwise asking.
Chromaniac
Well-known member
yeah. i agree. no follow would not prevent INDEXING of the content on links that are nofollowed. that is pretty much clear. 
Markos
Well-known member
I repeat, for seo purposes is better to use the robots.txt AND the nofollow attribute in profiles and non-relevant pages in a forum. See in the vbseo forums. The member page and the member profiles are disallowed in Robots.txt AND the profile links in post has the nofollow attribute. It's important to disallow indexing to the profile page AND disallow the "link consideration" of the link pointed in a profile.... Why? See my previous post (link juice, limits the links "counted" by Google in a page). You can choose to allow or disallow indexing profiles, but i don't see a "true" content in the profiles (however, i respect the Kier's view ).
).
I'm sorry, you are simply wrong.
dutchbb
Well-known member
Traffic is generated by content that is searched for. If that profile contains enough instances of 'Kier' and 'vb4' for example, that profile might eventually rank above some vB content in Google for the query 'Kier vb4' (which it doesn't atm, but that may change when the PageRank or content changes).
So you see, 'good content' is highly subjective, search engines look for keywords, people search with keywords. The goal is to get more people to find your site instead of somebody else's.
Anthony Parsons
Well-known member
Why would you want to disallow pages from being indexed? They would only contain relevant content that is spiderable.
Chromaniac
Well-known member
would not it be cool if XF includes a robots.txt generator! 
of course i can use a third party robots.txt generator... but that would not have following features!
it can provide options on what things can be blocked from indexing. like:
search result pages (makes no sense anyways)
member profile pages
pages
individual forums
and so on!
user selects the relevant option. it gives out raw content. and if robots.txt is writable, it does the needful.
i suppose this is probably too much for something too little!
haha. i guess this post has been a wastage of everyone's time.
of course i can use a third party robots.txt generator... but that would not have following features!
it can provide options on what things can be blocked from indexing. like:
search result pages (makes no sense anyways)
member profile pages
pages
individual forums
and so on!
user selects the relevant option. it gives out raw content. and if robots.txt is writable, it does the needful.
i suppose this is probably too much for something too little!
haha. i guess this post has been a wastage of everyone's time.
Cezz
Well-known member
The Robot.txt file is actually currently VERY baddly created as you should never included hidden/private/restricted pages like the admin.php file in a robot.txt file... This is a level 1 PCI violation and also tells would-be hackers where your areas are located. Sometimes, that's half the battle.
Markos
Well-known member
An alternative to robots.txt can be the metatag "robots", with the attribute "noindex". See this: http://www.robotstxt.org/meta.html
Shadab
Well-known member
I created something similar for vBulletin (in 2008, I think). It was a standalone product and not a plugin, per se. The user could select what type of pages to allow or disallow from being indexed. See the screenshot attached.
Probably the reason why I never published that code. It was an overkill!i suppose this is probably too much for something too little!
And unfortunately, a colossal waste of time.
Attachments
Chromaniac
Well-known member
well the focus these days is to have the most relevant content indexed instead of most content indexed.
Surely as soon as the software is sold, all that will become common knowledge anyway?
Similar threads
- Question
