Robots.txt and sitemap questions

[Alvin63]

Thanks,

 

[Alvin63]

Page speed insights just gave me this message

" robots.txt is not valid - Lighthouse was unable to download a robots.txt file"

It says

Line 2 Content - Disallow * Error - no user agent specified

I think it's working but why does it say it's not valid. I have this:


User-agent: AspiegelBot
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MauiBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: ImageSift
Disallow: /

User-agent: AnthropicBot
Disallow: /

User-agent: Yandex
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: ByteDance
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: *
Disallow: /admin.php
Disallow: /account/
Disallow: /goto/
Disallow: /login/
Disallow: /register/
Disallow: /search/
Disallow: /help/
Disallow: /members/

Sitemap: https://www.xxxxxxxxxxxx.com/sitemap.xml

 

[RichDevman]

Looks valid. Do you have it in your root directory? Do you have cloudflare?

 

[Alvin63]

Yes and yes

I just edited the message above to show what the error was that was displayed.

 

[RichDevman]

Cloudflare is most likely just blocking lighthouse from crawling it.

 

[Alvin63]

Thanks

 

[RichDevman]

No problem.