1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Another robots.txt question

Discussion in 'Forum Management' started by DJ XtAzY, Dec 10, 2013.

  1. DJ XtAzY

    DJ XtAzY Active Member

    So I see a lot of people posting their robots.txt file here with some rules like these:

    Disallow: /css.php
    Disallow: /js
    Disallow: /styles/

    After watching this video:


    are those rules above still necessary? I was looking at XF's and DigitalPoint's own robots.txt and theirs are far more simpler with at most like 7 rules.

    Mine is currently (like most others):

    User-agent: *
    Disallow: /find-new/
    Disallow: /goto/
    Disallow: /account*
    Disallow: /help*
    Disallow: /misc/style*
    Disallow: /misc/quick-navigation-menu*
    Disallow: /login*
    Disallow: /logout*
    Disallow: /lost-password*
    Disallow: /register*
    Disallow: /reports*
    Disallow: /search*
    Disallow: /conversations*
    Disallow: /css.php
    Disallow: /cron.php
    Disallow: /admin.php
    Disallow: /js
    Disallow: /styles/
    Disallow: /attachments/*
    Disallow: /members/*
    Disallow: /profile-posts/*
    Disallow: /online/*
    Disallow: /recent-activity/*
    Disallow: /posts/
     
  2. DJ XtAzY

    DJ XtAzY Active Member

    No one has thought on this?
     

Share This Page