Fixed Add-on list susceptible to exceeding max_user_connections due to add-on icon approach

[RSI]

Then why not instead of deleting my posts just send me a PM first and tell me to create another topic, or etc?

But anyway that's the thing, I've know exactly why I was having the problem this whole time. My hosting company has two of the limits restricted. There's the one that most newer hosts will set to around 1000 today and my host has that one set to either 700 or 750, I forget which. Then there's the one that most newer hosts set to around 70 or 100 and mine is set to 35 and that's the one which was doing it. Other than that, the server is fully up to date.

The question is why does the ACP want to use more connections to load those icons next to the add-ons in the first place?

My server info


Then the additional missing info

.

 

[Chris D]

Can you let us know specifically which limits you suspect you’re hitting? Also how many add-ons you have which have image based icons (actual images rather than Font Awesome icons). I can’t recall if you’ve provided that information at all.

Essentially, the more information you provide, the better chance we have of being able to understand the issue, reproduce it and perhaps come up with a solution.

 

[RSI]

Well, I looked around for the other thread I referred to back further in this topic where I quoted myself, but it's gone. I think it was moved to the Trouble shooting and problem forum, but it must have eventually been deleted by someone. At least I cannot find it using the "Your content" link. I had described in detail what was happening about the connections and what was going on with that one limit. I had even posted the error logs showing the problem that was happening when it was calling up the icons.

Anyway, I currently have 46 add-ons installed and that's about the norm. I usually run close to 50. Part of it is because I use a bunch of Andy's add-ons and his only do about one thing each so there winds up being a bunch of them installed. It might be wise to just modify the templates manually instead and eliminate many of them, but I really don't like doing that. Using add-ons is easier and safer.

I'll have to go back and look at the server again when I have time to figure out the connection detail on which socket it was. It really doesn't matter now though since the add-on is being developed which will solve the problem, but I do think it would be wise to look at what's going on because say if someone is on a new server and that limit is set to 75 or even a 100, but the person has 50 or more add-ons installed. I might be the only person you have had complain about this just because of the large number I have installed. Sooner or later someone else may too. I did try uninstalling all the add-ons then removing them from the SRC folder back then and the problem went away as well.
.

 

[Chris D]

We wouldn't have had any reason to delete it, and if we had we'd have notified you by alert or conversation with the reasons why.

I think I found it, you might have overlooked it because it was all buried within another thread about something unrelated:
https://xenforo.com/community/threads/error-message-when-changing-avatars.145402/post-1239446

max_user_connections is actually a value which is, by default, set to 0 i.e. unlimited. I was curious so I just had a look at a few servers I have access to, ranging from shared to dedicated and all of them use the default value of 0.

I still wasn't entirely convinced so I just signed up for a hosting account with Namecheap (it was the easy option because I already use them for domain names). It's a package which costs less than $3 a month. Even that has max_user_connections set to 0.

I concur that the approach we took with Admin CP icons can theoretically pose an issue in this case, but I certainly wouldn't expect many customers to experience this sort of issue at all as throttling connections in such a way is unusual for even the cheapest shared hosts.

That said, I'm not going to be awkward over it, simply ignore the problem and write it off. As I said earlier, it is on our radar. However, the root cause here is what appears to be an over-throttled server. Theoretically, and feel free to test this if you like, if you were to create a thread in a private forum on your site and fill 10 posts with 10 full-size attachments (rather than inserting thumbnails) you will most likely find you hit the same limit before you're half-way through creating them.

I appreciate that's a fairly contrived example, but I just wanted to demonstrate that there would be other ways to trigger this and actually not something that would necessarily be limited to XF2, or even XF specifically.

Ideally, your host wouldn't be setting max_user_connections or you were using a different host which didn't set that (as most don't).

But, as I said, we won't write this off and we may implement something that may help in a future version.

 

[RSI]

We wouldn't have had any reason to delete it, and if we had we'd have notified you by alert or conversation with the reasons why.

I think I found it, you might have overlooked it because it was all buried within another thread about something unrelated:
https://xenforo.com/community/threads/error-message-when-changing-avatars.145402/post-1239446

max_user_connections is actually a value which is, by default, set to 0 i.e. unlimited. I was curious so I just had a look at a few servers I have access to, ranging from shared to dedicated and all of them use the default value of 0.

I still wasn't entirely convinced so I just signed up for a hosting account with Namecheap (it was the easy option because I already use them for domain names). It's a package which costs less than $3 a month. Even that has max_user_connections set to 0.

I concur that the approach we took with Admin CP icons can theoretically pose an issue in this case, but I certainly wouldn't expect many customers to experience this sort of issue at all as throttling connections in such a way is unusual for even the cheapest shared hosts.

That said, I'm not going to be awkward over it, simply ignore the problem and write it off. As I said earlier, it is on our radar. However, the root cause here is what appears to be an over-throttled server. Theoretically, and feel free to test this if you like, if you were to create a thread in a private forum on your site and fill 10 posts with 10 full-size attachments (rather than inserting thumbnails) you will most likely find you hit the same limit before you're half-way through creating them.

I appreciate that's a fairly contrived example, but I just wanted to demonstrate that there would be other ways to trigger this and actually not something that would necessarily be limited to XF2, or even XF specifically.

Ideally, your host wouldn't be setting max_user_connections or you were using a different host which didn't set that (as most don't).

But, as I said, we won't write this off and we may implement something that may help in a future version.

That's the thread, it went off topic because this one was still fresh in my head when I came back from dinner that night and it happened again. Then it went back on topic until that guy asked or replied about it later on in the thread. I saw the thread earlier listed via my your content link but didn't connect the dots.

Anyway no, that internal max_user_connections setting on my server is set to 75. It's not set to 0. I was remebering 35 connections, but now I remember it was actually the 75 after reading the posts. Then many of the other hosts here set that to like 100, but my host wouldn't raise it. I called them right after I checked it. They base that number on the number of accounts on the server. It's the total allowed connections on the server divided by the number of accounts minus a little headroom. It's usually like 100 connections per account per server and I think some even go to 300 connections per account, but it depends and that may be where I got the 35 from. The 3 from 300 and the 5 from 75

Then on the external side it's usually like 1000 connections per account and mine is set to either 700 or 750. Those old brain cells haven't re-fired yet.
.

 

[RSI]

And after running show variables like "max_connections"; just now it says 750 connections so that's settled.

And that's max_connections, not max_user connections. The max_user_connections is set to 75. The norm is usually 100 with most hosts here.

So the issue isn't with the max_connections, 750 is fine, it's the 75 max_user connections. The additional connections the icons or whatever are trying to make push it past that limit. That's what was going on. Actually, the 75 connections should be enough, but they aren't and that's the whole issue.

Anyway, that's about all the info I can give you on what's going on.
.

 

[RSI]

Oh, and nope... "if you were to create a thread in a private forum on your site and fill 10 posts with 10 full-size attachments (rather than inserting thumbnails) you will most likely find you hit the same limit before you're half-way through creating them."

Nothing happens like that. You can post away on my board with multiple attachments until the one I/O resource limit is reached, then it just slows it down a bit, but never stops and it still finishes every time. I've been uploading like 10 to 12 pdf files at time in Resources lately that are around 50MB each and don't have any issues there either.

The only problem I have had is when in the ACP due to the add-on icons or whatever being called up. Everything else work fine and causes no issues.
.

 

[DragonByte Tech]

Oh, and nope... "if you were to create a thread in a private forum on your site and fill 10 posts with 10 full-size attachments (rather than inserting thumbnails) you will most likely find you hit the same limit before you're half-way through creating them."

Nothing happens like that.

Try posting 10 attachments per post across the 20 posts on the first page (so 200 attachments in total) then see if it happens again.

It is actually possible to accept the fact that your host has made a decision that, while great for them (less resources taken by each customer means more overselling) is bad for you. That doesn't mean you have to say the host is the worst host of all time, it just means that what you're trying to do with the software has gone beyond the capabilities of that particular host, if they are unwilling to uncap max_user_connections for you.


Fillip

 

[Chris D]

Nothing happens like that. You can post away on my board with multiple attachments until the one I/O resource limit is reached, then it just slows it down a bit, but never stops and it still finishes every time. I've been uploading like 10 to 12 pdf files at time in Resources lately that are around 50MB each and don't have any issues there either.

To clarify, it's not related to the uploading of them. It's related to the display of them. I should have clarified but I specifically meant "full size images".

The reason why it is similar is a full size image attachment gets inserted into a post with this BB code [ATTACH]123[/ATTACH] and the resulting HTML is more or less:

<img src="https://example.com/attachments/image.123" />

The XF attachment URLs do not reference an image directly on the file system. The reason for this is because of the permissions system - that attachment URL actually starts the XF application, sets up the current visitor, checks they have permission to view the attachment and THEN it gets the image data and returns that. By doing this, each time an attachment is viewed, it creates a new database connection and therefore if there are <existing number of connections> + <number of image attachment tags in posts> connections and that exceeds your max_user_connections value then you will hit the issue.

Why do add-on icons behave similarly? Simply out of convenience mostly, and similar reasons though not really directly related to user group permissions. When an add-on developer creates an add-on, the majority of their code is created inside a single folder. We then pull that together and put it in a Zip file. We decided to add icon/image support here because it makes for a much better looking, interesting list and makes it much easier to find things. But the problem is that the image file is buried within a directory that a normal web user doesn't and shouldn't have access to (for security reasons). So what we have to do is, like attachments, load an image like this:

<img src="https://example.com/admin.php?add-ons/XFMG/icon/" />

Like attachments this creates a new DB connection.

I concede that with regards to add-on icons, this is a much more likely reproduction case, but I just wanted to be clear that the reasons for this aren't really constrained to add-on icons and certainly aren't anything new - you've just so happened not to have anything trigger the issue before.

We are making changes here, and I'd like to send you the updated file later so you might be able to confirm if it makes an improvement. Though longer term, I'd still strongly recommend that you explore hosting with a company that doesn't restrict database connections in such a way. I'm sure some hosts do, but most hosts don't. I appreciate their efforts in reducing over-subscription on shared servers and the impact on other customers but there has to be a balance between doing that and not over-throttling a customer's growth.

 

[XF Bug Bot]

Thank you for reporting this issue. The issue is now resolved and we are aiming to include that in the next XF release (2.0.11).

Change log:

Use a different approach to loading icons on the add-ons list in the Admin CP. To avoid issues with multiple database connections, the icon image data is instead converted to a data URI.

 

[RSI]

Thank you...

And, if you want to send me the file then yes, I can try it. I'm still going to use the add-on to turn them off even if its fixed, but at least it might prevent someone else from having the same problem in the future. My guess it someone would sooner or later. My issue was always with the response I received which was XF2 is not well suited for shared servers, but I disagree with that. I don’t think you guys would want to lose business just because someone is on a shared server and has the problem.
.