Heavy bot traffic using the search feature was causing long post delays

[PumpinIron]

For a while now I've been having a big problem with my forum (https://wranglertjforum.com) in which when you go to make a new post and press the "post reply" button, it would take 10+ seconds for the reply to actually post. Same thing when deleting a thread, editing a thread, etc.

I used the web inspector console and found that the core-compiled.js file was taking the longest to load (upwards of 10+ seconds).

I decided to use ChatGPT to help me diagnose and it led me to monitor MySQL while testing the replies. That led me to this being the problem:




I was getting so many bots hitting the server using the "search" function that it was bogging down resources on the server.

I implemented a Cloudflare rule to block searches for guests (may change it to a rate limited rule potentially) and just within the past 24 hours that rule has been active it's blocked over 7k requests.

I just wanted to share this here in case anyone ever runs into a similar issue. This took me a long time to figure out and it was ChatGPT that finally helped me to do it.

Some people rip on Cloudflare but I really, really like it. The amount of bot traffic it's allowed me to block is amazing. Not only that but I love how much control I have over things with the security rules and analytics. For $20 a month it's highly worth it to me and has been a very effective tool at blocking out increasing bot traffic.

 

[briansol]

Set these search items to no in your unregistered group too

 

[PumpinIron]

Set these search items to no in your unregistered group too

View attachment 335663

Go figure. Didn’t even know those existed! Thanks for sharing this.

 

[smallwheels]

First: Congrats for solving this issue and thanks for sharing! But to play a little devil's advocate:

I implemented a Cloudflare rule to block searches for guests (may change it to a rate limited rule potentially) and just within the past 24 hours that rule has been active it's blocked over 7k requests. (...)

Some people rip on Cloudflare but I really, really like it. The amount of bot traffic it's allowed me to block is amazing. Not only that but I love how much control I have over things with the security rules and analytics. For $20 a month it's highly worth it to me and has been a very effective tool at blocking out increasing bot traffic.

Hmm, if before you added the rule you had 7k more bots on your forum it does not seem to be too effective, to be honest. At least not in your config. What you configured then would have been a 2 second permission adjustment within XF and completely free, so I don't the the reason to praise Cloudflare here either.

Not saying that it cannot do the job - on the other hand 20$/month is more than I spend for my hosting and with the IP thread monitor add on (which is not free but way cheaper than your CF account level) I do successfully block the majority of the bots. So Cloudflare may work, but it is by far not the only tool that does the job and not the cheapest one either. It can however do other things (that I don't need at my forum size).

 

[PumpinIron]

First: Congrats for solving this issue and thanks for sharing! But to play a little devil's advocate:

Hmm, if before you added the rule you had 7k more bots on your forum it does not seem to be too effective, to be honest. At least not in your config. What you configured then would have been a 2 second permission adjustment within XF and completely free, so I don't the the reason to praise Cloudflare here either.

Not saying that it cannot do the job - on the other hand 20$/month is more than I spend for my hosting and with the IP thread monitor add on (which is not free but way cheaper than your CF account level) I do successfully block the majority of the bots. So Cloudflare may work, but it is by far not the only tool that does the job and not the cheapest one either. It can however do other things (that I don't need at my forum size).

I agree, I may have been too quick to praise Cloudflare. However, I've got a list of other security rules I've put in place (at least 10) that have been very effective so far.

I can't believe I didn't think to look under the user group permissions to begin with! Of course there was a permission for guest searches.