DJ XtAzY
Active member
So I see a lot of people posting their robots.txt file here with some rules like these:
Disallow: /css.php
Disallow: /js
Disallow: /styles/
After watching this video:
are those rules above still necessary? I was looking at XF's and DigitalPoint's own robots.txt and theirs are far more simpler with at most like 7 rules.
Mine is currently (like most others):
User-agent: *
Disallow: /find-new/
Disallow: /goto/
Disallow: /account*
Disallow: /help*
Disallow: /misc/style*
Disallow: /misc/quick-navigation-menu*
Disallow: /login*
Disallow: /logout*
Disallow: /lost-password*
Disallow: /register*
Disallow: /reports*
Disallow: /search*
Disallow: /conversations*
Disallow: /css.php
Disallow: /cron.php
Disallow: /admin.php
Disallow: /js
Disallow: /styles/
Disallow: /attachments/*
Disallow: /members/*
Disallow: /profile-posts/*
Disallow: /online/*
Disallow: /recent-activity/*
Disallow: /posts/
Disallow: /css.php
Disallow: /js
Disallow: /styles/
After watching this video:
are those rules above still necessary? I was looking at XF's and DigitalPoint's own robots.txt and theirs are far more simpler with at most like 7 rules.
Mine is currently (like most others):
User-agent: *
Disallow: /find-new/
Disallow: /goto/
Disallow: /account*
Disallow: /help*
Disallow: /misc/style*
Disallow: /misc/quick-navigation-menu*
Disallow: /login*
Disallow: /logout*
Disallow: /lost-password*
Disallow: /register*
Disallow: /reports*
Disallow: /search*
Disallow: /conversations*
Disallow: /css.php
Disallow: /cron.php
Disallow: /admin.php
Disallow: /js
Disallow: /styles/
Disallow: /attachments/*
Disallow: /members/*
Disallow: /profile-posts/*
Disallow: /online/*
Disallow: /recent-activity/*
Disallow: /posts/
