• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

Another robots.txt question

DJ XtAzY

Active member
#1
So I see a lot of people posting their robots.txt file here with some rules like these:

Disallow: /css.php
Disallow: /js
Disallow: /styles/

After watching this video:

are those rules above still necessary? I was looking at XF's and DigitalPoint's own robots.txt and theirs are far more simpler with at most like 7 rules.

Mine is currently (like most others):

User-agent: *
Disallow: /find-new/
Disallow: /goto/
Disallow: /account*
Disallow: /help*
Disallow: /misc/style*
Disallow: /misc/quick-navigation-menu*
Disallow: /login*
Disallow: /logout*
Disallow: /lost-password*
Disallow: /register*
Disallow: /reports*
Disallow: /search*
Disallow: /conversations*
Disallow: /css.php
Disallow: /cron.php
Disallow: /admin.php
Disallow: /js
Disallow: /styles/
Disallow: /attachments/*
Disallow: /members/*
Disallow: /profile-posts/*
Disallow: /online/*
Disallow: /recent-activity/*
Disallow: /posts/