Robots.txt

[Asser]

Does the robot.txt file go in your public_html folder

For sure!
Upload it on public_html

 

[Gregsta]

For sure!
Upload it on public_html

Thanks for that

 

[dethfire]

Noticed Bing was indexing /search/ results so I disallowed them now.

 

[boeing747]

I was looking my google search console, I found there are more than 1.5 excluded urls :|
all of them affected by
Disallow: /posts/

Last crawl
Sep 25, 2018, 8:47:32 AM
Crawl allowed?No: blocked by robots.txt
Page fetch
Failed: Blocked by robots.txt

Whats your opinion?

Should I delete that line from robot.txt ?

 

[boeing747]

any suggestion?

 

[MySiteGuy]

any suggestion?

See post #68 of this thread.

 

[ERSAN.NET]

hello, which robots.txt file would you suggest?

 

[KSA]

Is there away to disallow Google bots crawling/seeing/recgonizing external links?

 

[htsumer]

Twitter is not showing meta images on my links.
The error says
what should I do

INFO: Page fetched successfully
INFO: 18 metatags were found
INFO: twitter:card = summary tag found
INFO: Card loaded successfully
WARN: The image URL https://www.maasmutemeti.com/forum/data/assets/logo/face_logo_uzman.png specified by the 'twitter:image' metatag may be restricted by the site's robots.txt file, which will prevent Twitter from fetching it.

robot.txt

User-agent: *
Allow: /

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/ajax/
Disallow: /forum/misc/contact/
Disallow: /forum/data/
Disallow: /forum/conversations/
Disallow: /forum/events/birthdays/
Disallow: /forum/events/monthly/
Disallow: /forum/events/weekly/
Disallow: /forum/find-new/
Disallow: /forum/help/
Disallow: /forum/internal_data/
Disallow: /forum/js/
Disallow: /forum/library/
Disallow: /forum/search/
Disallow: /forum/styles/
Disallow: /forum/login/
Disallow: /forum/lost-password/
Disallow: /forum/online/
Allow: /
Sitemap: https://www.maasmutemeti.com/forum/sitemap.php

 

[htsumer]

Twitter is not showing meta images on my links.
The error says
what should I do

View attachment 261783



robot.txt

current

 

[Jeremy P]

Twitter is not showing meta images on my links.
The error says
what should I do

View attachment 261783



robot.txt

You have Disallow: /forum/data/ in your robots.txt. Either remove that line or add additional rules to let certain bots bypass it.

 

[htsumer]

You have Disallow: /forum/data/ in your robots.txt. Either remove that line or add additional rules to let certain bots bypass it.

Thank you. Deleted it fixed it

 

[click here]

Would one disallow their off topic forum? Pros? Cons?

 

[snoopy5]

For what is this for?

Disallow: /forums/-/

 

[Mr Lucky]

Would one disallow their off topic forum? Pros? Cons?

No, I just noindex the forums (in the forum settings) I don’t want crawled and indexed. Robots.txt disallow doesn’t necessarily stop indexing.

Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers

Robots.txt is used to manage crawler traffic. Explore this robots.txt introduction guide to learn what robot.txt files are and how to use them.

developers.google.com

 

[mc nabulsy]

done added to my forum, hope this really help improving. is it the same with 2.2?