XF 2.1 User-agent Googlebot for smartphone

kankan

Well-known member
Hello
I am facing issue with robots.txt for mobile
when testing my site for mobile here :


Test return that page is not reachable or blocked by a robots.txt file

User-agent
Googlebot for smartphone

As a consequence, i registered incorrect traffic for mobile and Adsense disabled my ads diffusion.

My set up is in /root/forums
my robots.txt is in /root/ and declared as follow :

Code:
User-agent: *

Disallow: /asus/
Disallow: /backup/
Disallow: /isos/
Disallow: /js/
Disallow: /forums/wiki/
Disallow: /wiki
Disallow: /www/
Disallow: /XF2_Addons/
Disallow: /forums/find-new/
Disallow: /forums/account/
Disallow: /forums/goto/
Disallow: /forums/login/
Disallow: /forums/admin.php
Disallow: /forums/conversations/
Disallow: /forums/devforums/

Allow: /

Sitemap: https://www.forum-des-portables-asus.fr/forums/sitemap.php

User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: grapeshot
Disallow: /
User-Agent: trendictionbot
Disallow: /

Thank you for your help.
 
Top