XF 1.3 Robot.txt advice

[Lexy]

How to make robot txt here http://www.mcanerin.com/EN/search-engine/robots-txt.asp any advice what allow what not

 

[Paul B]

Create a robots.txt file.

Upload it to your web root.

 

[Lexy]

so just like this

User-agent: *
Disallow:
Disallow: /cgi-bin/
Sitemap: http://example/sitemap.xml

 

[Martok]

You can see what XenForo have added here:

http://xenforo.com/robots.txt

 

[Lexy]

without community

 

[Lexy]

why disalow post it is only for community

 

[Lexy]

so just need to delete community and add rest ones

 

[Lexy]

and looks like this??

User-agent: *
Disallow: /cgi-bin/
Disallow: /find-new/
Disallow: /account/
Disallow: /attachments/
Disallow: /goto/
Disallow: /login/
Disallow: /admin.php
Allow: /
Sitemap: http://www.example.com/sitemap.xml

 

[Paul B]

Set it up to suit your own particular requirements.

This is mine: http://cliptheapex.com/robots.txt

 

[Lexy]

why disallow post?' it is only for xenforo who hosted on community

 

[Paul B]

http://xenforo.com/community/threads/why-disallow-posts-in-robots-txt.56006/#post-596780

 

[Lexy]

thanks