MG 2.2 Media Gallery causing excessive database load

[Zig]

We've noticed that XFMG is causing an undue load on our database. The most common XFMG queries account for more than half our database CPU time, despite requests numbering in the low hundreds of thousands over a 7-day period. Meanwhile, the XF core and our numerous other add-ons make hundreds of millions of requests without issue in the same time period.

Recently, XFMG has caused spikes to 100% CPU usage on our very large DB (10 vCPUs and 64GB RAM), which in turn causes response delays that ultimately result in load balancer timeouts and site downtime. Disabling XFMG stops the 100% spikes altogether and drops our typical baseline CPU usage from ~25% to ~10%.

While our XFMG install is quite large (over 30k pages of content on the index), it remains our least commonly used feature. That said, a number of our diehard members do use the add-on and, as we can all agree, this kind of passionate member is key to any community. With all that in mind, I'm wondering: is there any chance of optimization in the near future? Or are there perhaps some steps we could take to reduce the load caused by XFMG? We would like to continue using the add-on, as we do have users interacting with it and a large number of historical records we'd like to retain.

Any help you could offer would be greatly appreciated!

 

[StarArmy]

I know this is late, but I am wondering if these spikes were happening because robots were aggressively indexing your site's images for use in training AI or something like that. Something similar happened to me but with my wiki's images. I ended up having to set rules in Cloudflare to reduce the bot traffic to manageable levels. If you're still having this issue, you could try that,or putting most of the media content behind a login so randoms and bots won't be able to load it.

 

[LemmeSeeHim]

Same issue.

 

[Martin Elsworth]

I have a forum (with the assistance of someone else who is far more knowledgeable than I am) The use of media is an integral part of the forum but it is causing severe server bloat making updates / backing up more difficult. Is there anything I can do to reduce this problem. Should I possibly be hosting images on a third party server? Apologies in advance if this has already been answered. I don't use this forum as often as I should.

 

[Seeker-Smith]

We've noticed that XFMG is causing an undue load on our database. The most common XFMG queries account for more than half our database CPU time, despite requests numbering in the low hundreds of thousands over a 7-day period. Meanwhile, the XF core and our numerous other add-ons make hundreds of millions of requests without issue in the same time period.

Recently, XFMG has caused spikes to 100% CPU usage on our very large DB (10 vCPUs and 64GB RAM), which in turn causes response delays that ultimately result in load balancer timeouts and site downtime. Disabling XFMG stops the 100% spikes altogether and drops our typical baseline CPU usage from ~25% to ~10%.

While our XFMG install is quite large (over 30k pages of content on the index), it remains our least commonly used feature. That said, a number of our diehard members do use the add-on and, as we can all agree, this kind of passionate member is key to any community. With all that in mind, I'm wondering: is there any chance of optimization in the near future? Or are there perhaps some steps we could take to reduce the load caused by XFMG? We would like to continue using the add-on, as we do have users interacting with it and a large number of historical records we'd like to retain.

Any help you could offer would be greatly appreciated!

Are you by chance using Dragon-Byte Social Groups? There is a known issue.

 

[chillibear]

Should I possibly be hosting images on a third party server?

You could certainly offload the media elsewhere - XF 2.2 supports using S3 via this official add-on and in XF 2.3 I understand the functionality is built in - although you may want to refer to the discussion on the add-on page for details about configuration.

Otherwise there are various image optimisation options you could consider to reduce file sizes somewhat, eg moving to webp, etc

 

[Seeker-Smith]

I know this is late, but I am wondering if these spikes were happening because robots were aggressively indexing your site's images for use in training AI or something like that. Something similar happened to me but with my wiki's images. I ended up having to set rules in Cloudflare to reduce the bot traffic to manageable levels. If you're still having this issue, you could try that,or putting most of the media content behind a login so randoms and bots won't be able to load it.

This pretty standard on this site and they consume literally nothing in resources.