XF 2.1 Unfurl leading to Robot Check (Amazon)

[NealC]

I started seeing my amazon links change to "Robot Check" for link titles. Anyone else seeing this? What's the cause and how to fix?

 

[Jeremy P]

Amazon probably (correctly) classified your traffic as bot traffic and is serving up a CAPTCHA or similar in response to the unfurl request from your server. There's not really anything you can do about it.

 

[NealC]

Sounds like we may need an unfurl exclusion list so we can prevent certain domains/urls from being unfurled so I can avoid hitting Amazon for a while but let other pages unfurl. I turned off unfurling for the time being.

 

[NealC]

I'm using one of the xf2addons that puts my affiliate tag in the link, I wonder if there are too many requests being hit on amazon such as the raw URL then the changed URL resulting in this mini ddos? I'm going to disable the add-on and leave unfurling off for a day or two and see what happens. I also use SkimLinks on the site and I'm not sure if that's playing into this. A lot of things in play here and the last thing I need not working is Amazon.

 

[NealC]

I think it's a combination of unfurling and getting link titles. I had to turn both off and am still monitoring.

 

[arn]

same thing happening on our site

test (guess it takes a certain volume)

Amazon.com: USB C Hub, 9-in-1 USB C Adapter with 4K USB C to HDMI,VGA, USB C Charging, 2 USB 3.0, SD/TF Card Reader, USB C to 3.5mm, Gigabit Ethernet, USB C Dock Compatible Apple MacBook Pro 13/15 (Thunderbolt 3): Computers & Accessories

Amazon.com: USB C Hub, 9-in-1 USB C Adapter with 4K USB C to HDMI,VGA, USB C Charging, 2 USB 3.0, SD/TF Card Reader, USB C to 3.5mm, Gigabit Ethernet, USB C Dock Compatible Apple MacBook Pro 13/15 (Thunderbolt 3): Computers & Accessories

www.amazon.com

 

[Kirby]

mini ddos

It can't be a DDoS unless you are using multiple servers.

 

[NealC]

It can't be a DDoS unless you are using multiple servers.

Whatever you want to call it, this is not about picking apart my words but indicating a problem we're seeing with Amazon and unfurling.

 

[arn]

I've turned off unfurling for now. would be nice for a simple mod to selectively exclude amazon urls from unfurling.

 

[NealC]

Unfurling alone didn't work for me, the title option I had to turn off as well.

 

[arn]

Changing Xenforo's UserAgent to this worked for now:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"

Found in src/XF/SubContainer/Http.php

1. Xenforo uses "Xenforo/2.x (https://yourboardurl)" by default for their unfurling bot
2. The above is the user agent that Apple uses for iMessages. I see they covered their bases.

Obviously editing XF files directly is bad... but hopefully an add-on or something can address it.

arn

 

[Ozzy47]

Changing Xenforo's UserAgent to this worked for now:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"

Found in src/XF/SubContainer/Http.php

1. Xenforo uses "Xenforo/2.x (https://yourboardurl)" by default for their unfurling bot
2. The above is the user agent that Apple uses for iMessages. I see they covered their bases.

Obviously editing XF files directly is bad... but hopefully an add-on or something can address it.

arn

Is there an online tool to check your sites useragent?

 

[arn]

I ended up pasting in our own website url and looking at the www logs.

Also... this solution may not have lasted. It didn't work in a recent post. So we may be triggering an IP rate limit.

 

[dethfire]

This is happening with my forum now too. Really really bad as we get a lot of amazon links.

 

[arn]

Using some sort of proxy service might be best long term solution. Anyone know an inexpensive / non sketchy proxy service?

 

[NealC]

I would like to have, promptly, an exclusion list. Allow me to add domains to be excluded from unfurling. ASAP

 

[dethfire]

Sounds like XF just needs to disable unfurl for Amazon. Other companies might be following suit.

 

[Kirby]

I would like to have, promptly, an exclusion list. Allow me to add domains to be excluded from unfurling. ASAP

Unfurl whitelist and blacklist

As the european union is about to decide on new legislation that might cause forum operators in the EU not being allowed to display snippets/titles for links (without having to pay a fee to do so), it would be nice if XF could implement a blacklist adn whitelist for URL unfurling and for...

xenforo.com

Not much interest so far

 

[nrep]

Same problem on my site, it's getting a lot of complaints from members.

 

[KensonPlays]

Same problem on my site, it's getting a lot of complaints from members.

I honestly do not want to disable these previews. I enjoy them; but for Andy's Affiliate manager addon, it's causing errors in the error log.