Amazonbot can't view threads. Why?

[Levina]

For the last couple of days my site has been visited by Amazonbots, hundreds at a time. They can view tags and blog but not threads. All I see is this, page after page after page:


Now I have set a crawl-delay for them but I have not blocked them so they should be able to view threads, like the other bots that I allow on the site, but apparently they're blocked and I wonder why.

 

[Jeremy P]

You can hover over the triangle, but more than likely they're just trying to view a thread that guests don't have permission to view. There is nothing in the core which would cause them to be treated differently from any other bot.

 

[Chromaniac]

I do see a lot of bots with error Viewing unknown page A moment ago but usually they are on unknown page and not viewing thread like above with error triangle. I wish the error triangle would show the unknown page they are on lol.

 

[Levina]

I do see a lot of bots with error Viewing unknown page A moment ago but usually they are on unknown page and not viewing thread like above with error triangle. I wish the error triangle would show the unknown page they are on lol.

Yes, I get those "unknown page" errors too. But the Amazonbot is all "Viewing thread" with the error triangle.

You can hover over the triangle, but more than likely they're just trying to view a thread that guests don't have permission to view. There is nothing in the core which would cause them to be treated differently from any other bot.

Yes, it says "viewing an error" but what does that mean? What error? And it's over 20 pages of that. And it just goes on and on.

But it's good to know that "there is nothing in the core" causing it. I was wondering about that.

 

[Jeremy P]

Almost certainly just the regular no permission or not found error. You can check web server access logs to see what URLs they're trying to access. The bot permissions are no different from guests, but bots may be trying to scrape URLs guests don't have access to or that otherwise don't exist.

 

[Chromaniac]

so just a thought. i installed this user agent switcher app. changed the user agent to amazonbot. and accessed my site and i was able to see the cloudflare blocked page. it might be worth doing something like this to check what happens when you visit your website as amazonbot.

i used this one: https://webextension.org/listing/useragent-switcher.html

the user agent i used: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML\, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)

 

[Levina]

Almost certainly just the regular no permission or not found error. You can check web server access logs to see what URLs they're trying to access. The bot permissions are no different from guests, but bots may be trying to scrape URLs guests don't have access to or that otherwise don't exist.

I'm on Cloud hosting, Jeremy, so I don't think I have access to those logs. But at least I'm no longer concerned, so thank you!

 

[Alvin63]

It's probably just Alexa! I blocked Amazonbot in robots.txt as it was taking over a bit.

 

[Levina]

It's probably just Alexa! I blocked Amazonbot in robots.txt as it was taking over a bit.

I don't want to block it but have put in a crawl delay. Not sure if that is respected but my (small) site is not any slower so I'm not concerned in that way. But they're still crawling and still can't see any thread. Just tags, media and blogs.

One thing I don't get is how they seem to be stuck in viewing threads they don't have access to. And apparently only those. I would expect to also see them viewing threads where they can go.

 

[Levina]

so just a thought. i installed this user agent switcher app. changed the user agent to amazonbot. and accessed my site and i was able to see the cloudflare blocked page. it might be worth doing something like this to check what happens when you visit your website as amazonbot.

i used this one: https://webextension.org/listing/useragent-switcher.html

the user agent i used: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML\, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)

That wouldn't tell me which threads the real Amazonbot is trying to view but it might give an idea of what triggers the errors. Thanks for the suggestion!