XF 2.2 Anyway of limiting bandwidth in order to prevent site scraping?

jromaine

Member
Hi guys,
I had a member recently use a "scraping tool" to essentially download over 300 Wistia videos, along with hundreds of resources and a lot of written content. While this was happening I could see their user profile "online" for a lengthy period, overnight infact which caught my attention. Especially given most users tend to drop off after a few minutes. Upon logging into the admin panel I could see a whole bunch of error messages. When I say a whole bunch I mean hundreds!

ErrorException: [E_NOTICE] Undefined offset: 1 src/addons/IntegratePro/Wistia/Wistia.php:31
Generated by: (username removed) Oct 31, 2022 at 12:58 AM

I wanted to ask....

Is there any way, I can force end of sessions for members that show high bandwidth usage or abnormal behaviour within Xenforo?

Even if I could have the system notify me via email so I could investigate.

Obviously I'd prefer this didn't happen again.

Thanks
 

DarkGizmo

Well-known member
Why not ban the user? This can be done via IP range, you can also set them as a discouraged user, also by username & IP range, so that they get slowdowns and other annoying occurrences and this may prevent them from scraping your content.
 

nocte

Well-known member
Why not ban the user?
I think the OP wants to be sure this does not happen with other accounts again.

On the XF-side this is not easy to accomplish. You could use this addon (maybe there are other similar ones, this is the only one I am aware of):


Then setup a user promotion, that adds a user-group with limited rights after some time spent online (I don't know how flexible this addon is regarding time spans and duration of online activity). To be clear: This addon does not measure bandwidth, but only time spent online.

Another idea: use this (free) addon by @Xon along with his Redis Cache addon:


It won't help you out of the box, but with a custom addon on top of this you may be able to limit the requests and/or take actions.

But, ideally this need to be done on the server-level, I guess. Maybe Cloudflare can be helpful for this.
 

jromaine

Member
Why not ban the user? This can be done via IP range, you can also set them as a discouraged user, also by username & IP range, so that they get slowdowns and other annoying occurrences and this may prevent them from scraping your content.

I've already banned them. I booted them immediately and cancelled their subscription.

Up until finding this out recently they've been a valued member, no issues at all.

I can't run around "banning" users unless I know they've actually done something wrong, which is exactly why I'm asking the question.

Is there any way to detect this type of behavior so I can intervene/investigate and action accordingly?
 

nocte

Well-known member
P.S. regarding clouldflare, look at this article:


Example:


previous-rate-limiting-create-rule.png
 
Top