That helps a lot, thanksStill, could be styled better...
Another question, not sure how to solve this:
View attachment 339552
Tried disabling Redis caching but that didn't help. Nginx caching maybe?
Also do enable redis - bot guard will use it too!
XF Bot Guard 1.4.1
- Thread starter zeeb0t
- Start date
Also do enable redis - bot guard will use it too!
Sorry - on my list of things to do is support dark mode.
To he honest, I cant stand dark mode for this reason ... so disable it outright. But then, I forget about it!
Anyway, I will sort it out. Thanks for reporting it.
Hello @zeeb0t so its been over 48 hours of testing and I'd love to say a big thank you, this addon is a lifesaver. Before I started using it, my forum was swarmed with bot traffic, but now my Google Analytics, Microsoft Clarity and Statcounter analytics dashboards are all pretty clean. I've added screenshots from my admin dashboard.
However, there's one little issue tht popped up. Whenever i try to share a URL from my forum to any social media platform, the meta tags, og image e.t.c get stripped out and, never happened like this before i started using the addon. Is this a bug or something?
Attachments
That's amazing to hear! I would certainly appreciate it if you could share your experience by rating the add-on, you can do so at the top right near the download on this page: https://xenforo.com/community/resources/xf-bot-guard.10508/
A native solution for XF Bot Guard to accept social media crawlers is not yet published (coming soon), but there is an answer to help you out right now.
- Go to Setup -> Options -> XF Bot Guard
- Scroll down to the "Additional trusted crawler rDNS domains" setting
- Here you enter confirmed rDNS domains, I checked, and, a common list includes:
facebook.com
fbsv.net
whatsapp.net
twttr.com
twitter.com
x.com
linkedin.com
pinterest.com
pinterestcrawler.com
tumblr.com
telegram.org
skype.com
skype.net
- And Save.
Then, try sharing a new URL on one of those platforms to confirm.
Let me know how you go!
Done
Thank you very much!!
Just did. Sharing on Telegram and Twitter (X) works fine now, but all the Meta platforms (Facebook, Whatsapp, Instagram, Threads) still have the browser validation screen
Attachments
Is that a new URL you are trying, or the same one?
Same URL, I've tried 6 different URLs, even used Facebook URL debugger as well.
Ok - this is going to be a long list, but looks like Meta aren't sending out rDNS mapped IP's... so, add all of these in the attached text document to the "Excluded IPs/CIDRs" option, and Save.
Facebook and Threads work now, but Whatsapp still same issue.
I’ve looked into WhatsApp specifically.
Although WhatsApp is owned by Meta, WhatsApp link previews do not appear to behave like the normal Facebook/Meta crawler.
With Facebook, we can usually verify crawler traffic using a combination of known crawler behaviour, reverse DNS, ASN checks, and published crawler user-agents.
WhatsApp is different. For normal WhatsApp link previews, the request may come from the person’s own device or network rather than from a Meta-owned server range. That means the IP address reaching your site can be the user’s mobile carrier, home ISP, VPN, or another ordinary-looking IP, even though the user-agent says:
Code:
WhatsApp/...That creates a problem for XF Bot Guard because the only reliable signal left is the user-agent, and user-agents are trivial to spoof.
Allowing any request through just because it says it is WhatsApp would go against the core design of Bot Guard and would very quickly become an easy bypass for bots.
So at this stage I do not think it is safe to treat WhatsApp as a verified crawler in the same way as Googlebot, Bingbot, or properly verifiable Meta/Facebook crawler traffic.
Ok. Thanks for the clarification. Moving forward, do you plan on finding a workaround to Whatsapp in future updates?
Is this planned for ASN? So that users can be forced to pass a captcha or added to a whitelist?
Is there a way to check in reports which search engines are receiving captcha (or are they masquerading as such)?
Will there be support for common-crawlers.json? Working with large IP arrays is still difficult. Otherwise, you can get a file from the provider's website, or create one yourself and simply provide a link.
daily work report
Is there a way to check in reports which search engines are receiving captcha (or are they masquerading as such)?
Will there be support for common-crawlers.json? Working with large IP arrays is still difficult. Otherwise, you can get a file from the provider's website, or create one yourself and simply provide a link.
daily work report
I will definitely spend time trying to investigate a reliable method of validating a WhatsApp client. I can't promise a favourable result, but I will certainly try!
Yes, ASN support will be added for both include/exclude - more to the point, IP/CIDR will be expanded to ASN, and there will be an added Challenge IP/CIDR/ASN.
Not at this stage, but I just put it on the backlog for you.
Good idea, added to the backlog also!
That's looking nice and healthy!
I'd assume this is a catch22 with the way What's App works currently. The preview is probably rendered either on the sender's device or on the receiver's device. Given that what's app claims to be end-2-end encrypted as far as I know (I do use nothing that belongs to the meta-universe, so not too interested) it can't be done on the server.
So the only option would be to treat the what's app preview bot as a normal browser (no IP rDNS verification) and, as it probably lacks abilities, to use a cryptographic checksum for the crawler that Meta would need to provide for verification. Still a bit dirty, but as long as Facebook does nothing they will be locked out. The good thing is, that this will happen on other platforms, too, sooner or later, so if the ignore the issue they will shoot themselves in the foot and make their messenger less attractive. Which would not be a bad thing, when I think about it.
Hi there, I have this installed and running, but it does not actually seem to be doing anything. I do not use Cloudflare so I have that disabled.
Any ideas? (and yes Enable XF Bot Guard is checked)
Any ideas? (and yes Enable XF Bot Guard is checked)
