Robots.txt

boeing747

Active member
I was looking my google search console, I found there are more than 1.5 excluded urls :|
all of them affected by
Disallow: /posts/

Last crawl
Sep 25, 2018, 8:47:32 AM
Crawl allowed?No: blocked by robots.txt
Page fetch
Failed: Blocked by robots.txt

Whats your opinion?

Should I delete that line from robot.txt ?
 

KSA

Well-known member
Is there away to disallow Google bots crawling/seeing/recgonizing external links?
 

htsumer

Active member
Twitter is not showing meta images on my links.
The error says
what should I do

1639250989503.png

INFO: Page fetched successfully
INFO: 18 metatags were found
INFO: twitter:card = summary tag found
INFO: Card loaded successfully
WARN: The image URL https://www.maasmutemeti.com/forum/data/assets/logo/face_logo_uzman.png specified by the 'twitter:image' metatag may be restricted by the site's robots.txt file, which will prevent Twitter from fetching it.

robot.txt

User-agent: *
Allow: /

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /forum/find-new/
Disallow: /forum/account/
Disallow: /forum/attachments/
Disallow: /forum/goto/
Disallow: /forum/register/
Disallow: /forum/posts/
Disallow: /forum/login/
Disallow: /forum/admin.php
Disallow: /forum/ajax/
Disallow: /forum/misc/contact/
Disallow: /forum/data/
Disallow: /forum/conversations/
Disallow: /forum/events/birthdays/
Disallow: /forum/events/monthly/
Disallow: /forum/events/weekly/
Disallow: /forum/find-new/
Disallow: /forum/help/
Disallow: /forum/internal_data/
Disallow: /forum/js/
Disallow: /forum/library/
Disallow: /forum/search/
Disallow: /forum/styles/
Disallow: /forum/login/
Disallow: /forum/lost-password/
Disallow: /forum/online/
Allow: /
Sitemap: https://www.maasmutemeti.com/forum/sitemap.php
 
Top