• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

XF 1.1 Help setting up robots.txt file correct syntax

RoldanLT

Well-known member
#1
Here's my current robots.txt file: http://www.phcorner.net/robots.txt

User-agent: Mediapartners-Google
Disallow: /attachment*
Disallow: /cdn-cgi/
Disallow: /doubleclick/
Disallow: /goto/
Disallow: /images/
Disallow: /js/
Disallow: /posts/
Disallow: /styles/
Allow: /

User-agent: *
Disallow: /account/
Disallow: /conversations/
Disallow: /cdn-cgi/
Disallow: /doubleclick/
Disallow: /find-new/
Disallow: /goto/
Disallow: /js/
Disallow: /login/
Disallow: /posts/
Disallow: /search/
Allow: /

Sitemap: http://www.phcorner.net/sitemap/
Lines highlighted with red, Is it correct?

Is "Allow: /" necessary or obsolete already?