XF 2.2 Site continues to load every 10-30 minutes

[MR X]

Spoiler: faq

• the version of XenForo you are running: v2.2.10 Patch 1
• a list of all add-ons you have installed: way to many to name off. "im working on removing used addons"
• any outdated templates or phrases: no
• custom styles in use: yes
• confirmation that all add-ons and styles are compatible with the XenForo version you are running: yes/no
• whether the problem occurs in a specific browser, or more than one browser: any browser
• whether the issue is experienced by all users or just some: yes it is
• browser add-ons in use, e.g. Adblock Plus, NoScript, etc.
• any changes which were made prior to the problem occurring: Server moved to a dedicated server & an addon update
• details of any errors in the ACP server error log: none
• steps to be taken to reproduce the problem and whether it can consistently be reproduced: happens randomly
• confirmation whether the problem can be reproduced here on XenForo.com: No

So we recently switched to a new dedicated server with no issues,
we were using v2.7.1 of https://xenforo.com/community/resources/ap-style-suite.8282/ and noticed an update to v2.9.1 so we updated it, and after that all hell broke lose and our site now lags the hell out, we checked our server itself and is running smoothly.

Spoiler: server image inspect

image when xenforo is lagging :

Image when server is running fine:

in other words we believe xenforo itself is having issues as if i close the site from public it runs fine no issues, but if its open we have issues id say every 10-30 minutes of the site just constantly trying to load, I've disabled all addons one by one, uninstalled addons and still problems. "none of this occurred before the addon was updated I've asked the developer @apathy but he doesn't really know either any help would be appreciated.

the site is NSFW and so i will not publicly say it, please PM me for the link thank you.

Spoiler: site specs

30tb bandwidth
4 cores, 8Gb RAM
apache version 2.4.54
php version 8.0.22-1+1.1.cpanel
mysql 8.0.30

 

[nocte]

As the problem occurs periodically, maybe it has something to do with XF cron jobs. Are you running them activity-triggered or via CLI (cmd.php xf:run-jobs)?

 

[MR X]

As the problem occurs periodically, maybe it has something to do with XF cron jobs. Are you running them activity-triggered or via CLI (cmd.php xf:run-jobs)?

we run them actively-triggered "default xf install"

 

[apathy]

Did you check execution times for queries in debug mode? You never got back to me about that and it could tell us exactly where the problem is. See below for what a debug output looks like, and specifically what you're looking for. Make note of any queries which have an unusually long execution time (several seconds).


It'd complain in your admin panel if your job queue is clogged, but regardless like nocte suggested it still might be worth running php cmd.php xf:run-jobs in a terminal just to be sure it's not clogged.

Additionally it's worth noting that you have some raw edits made to various PHP files - are you reverting these each time you disable addons? They wont get reverted automatically.

Finally, you have an addon I think you said you wrote thats generating one of those "Your forum is in an inconsistent state" errors. If you haven't already I'd honestly get that sorted before I start worrying about anything else, weird **** can happen when you have one of those present.

 

[MR X]

Did you check execution times for queries in debug mode? You never got back to me about that and it could tell us exactly where the problem is. See below for what a debug output looks like, and specifically what you're looking for. Make note of any queries which have an unusually long execution time (several seconds).
View attachment 272410

It'd complain in your admin panel if your job queue is clogged, but regardless like nocte suggested it still might be worth running php cmd.php xf:run-jobs in a terminal just to be sure it's not clogged.

Additionally it's worth noting that you have some raw edits made to various PHP files - are you reverting these each time you disable addons? They wont get reverted automatically.

Finally, you have an addon I think you said you wrote thats generating one of those "Your forum is in an inconsistent state" errors. If you haven't already I'd honestly get that sorted before I start worrying about anything else, weird **** can happen when you have one of those present.

will look further into it,
about the addon that i wrote yea ill be removing it today.