Lack of interest Allow addons to not hog as much memory (when listening for load_class_controller)

[tenants]

For each addon, it seem just extending the core will increase the memory by about 0.06 MB (even though we can get the calculation of the addon it's self down to below 0.01 MB)

This 0.06 MB might not seem much, but at this point we have done nothing other than add the addon, and this value is arithmetic. The more addons a user has, the more memory will be hogged.

Since a lot of addons can be very memory cheap (and 0 query), is it not worth also keeping the memory down when extending the core classes

A simple test case is this empty thread extension addon (attached)
It does nothing other than
1) Listen to load_class_controller
2) extend the thread (no classes are overridden, no new classes are used)
- It's possibly the smallest addon possible that listens to load_class_controller and extends thread (but then it does do nothing)

Once installed, going to a thread and refreshing a few times, we get a rough steady state of about 2.300 MB (depending on what other addons are present)
Timing: 0.0521 seconds Memory: 2.300 MB DB Queries: 10

Turning the addon off and refreshing a few times, we get a rough steady state of about 2.247 MB (depending on what other addons are present)
Timing: 0.0513 seconds Memory: 2.247 MB DB Queries: 10

Un-installing the addon and refreshing a few times, we get a rough steady state of about 2.247 MB (depending on what other addons are present)
Timing: 0.0647 seconds Memory: 2.247 MB DB Queries: 10


If one addon listening to the load_class_controller adds 0.06 MB, then 25 addons would add about 1.5 Meg, even if they did very little/nothing

In this case, the 25 empty addons increase the memory used by 66% (and that's just addons that do nothing other than extend the thread class with no calculations).

 

[Adam Howard]

Interesting find.

Have you run a test on a true empty add-ons with no code (xml file only)? ..ie... The simple process of going into XenForo telling it to make an add-on, but not assigning it to anything (templates or otherwise).

If there is still an increase in memory, you may have found what I'd call a memory leak ... not in the strictest sense of the word ... But rather a design flaw in resource usage.

 

[Valhalla]

Interesting find.

Have you run a test on a true empty add-ons with no code (xml file only)? ..ie... The simple process of going into XenForo telling it to make an add-on, but not assigning it to anything (templates or otherwise).

If there is still an increase in memory, you may have found what I'd call a memory leak ... not in the strictest sense of the word ... But rather a design flaw in resource usage.

You would not be able to install an add-on if the extended classes didn't exist.

 

[Chris D]

Good find tenants.

I think that this may have already been resolved to a certain extent. Could you re-run your test, but specify an event listener hint of XenForo_ControllerPublic_Thread

I know a lot of your add-ons (admirably) still support below XF 1.2, but the event hint feature added in XF 1.2 I believe should mitigate this memory usage somewhat.

 

[Chris D]

Have you run a test on a true empty add-ons with no code (xml file only)? ..ie... The simple process of going into XenForo telling it to make an add-on, but not assigning it to anything (templates or otherwise).

An add-on which literally has nothing. No event listeners, no template mods, no options. i.e. I've just created an add-on in the Admin CP and saved it and done nothing else uses no resources. There's nothing for it to do. It's just a row in the database.

 

[tenants]

Good find tenants.

I think that this may have already been resolved to a certain extent. Could you re-run your test, but specify an event listener hint of XenForo_ControllerPublic_Thread

I know a lot of your add-ons (admirably) still support below XF 1.2, but the event hint feature added in XF 1.2 I believe should mitigate this memory usage somewhat.

I'll give it a go, but adding event hints does have it's draw backs... it means it will always be run after the other addons (and specifically in my case, I want it to be the very first for performance reasons and exit immediately, so I can't use the event hints for what I want to do, but I will test the event hints any way)

I did notice, by simply going to the listener and commenting out

case 'XenForo_ControllerPublic_Thread':
               $extend[] = 'EmptyAddon_ControllerPublic_Thread';
               break;

The memory is no longer 0.06 Meg (it's virtually nothing)
So it's not the act of the load_class_controller being present, but what load_class_controller then does with the listened for class (which I guess makes sense)

Interesting find.

Have you run a test on a true empty add-ons with no code (xml file only)? ..ie... The simple process of going into XenForo telling it to make an add-on, but not assigning it to anything (templates or otherwise).

If there is still an increase in memory, you may have found what I'd call a memory leak ... not in the strictest sense of the word ... But rather a design flaw in resource usage.

It's not the XML file it's self (or the adding of the XML data), it's what the listener then does with the listened for classes:

I did notice, by simply going to the listener and commenting out

case 'XenForo_ControllerPublic_Thread':
               $extend[] = 'EmptyAddon_ControllerPublic_Thread';
               break;

The memory is no longer 0.06 Meg (it's virtually nothing)

 

[Adam Howard]

An add-on which literally has nothing. No event listeners, no template mods, no options. i.e. I've just created an add-on in the Admin CP and saved it and done nothing else uses no resources. There's nothing for it to do. It's just a row in the database.

Good.

That is what I hoped it would do and what one would expect. I was a little concern if maybe there was more to this. Thanks for checking @Chris D

 

[Adam Howard]

It's not the XML file it's self (or the adding of the XML data), it's what the listener then does with the listened for classes:

I understood that... I was wondering if maybe there was something more.

But never the less, you did find a good find.

 

[Snog]

I think this has more to do with PHP itself than XenForo.

Every time you extend a class, a constructor is called and that uses memory. Even if the extension does nothing, PHP is going to allocate memory to construct it.

By commenting out the code that extends the class, you don't cause a constructor to be called so no change in memory will happen.

 

[tenants]

I think this has more to do with PHP itself than XenForo.

Every time you extend a class, a constructor is called and that uses memory.

You might be right, you are extending, but you're not really extending the class in xenforo, you're extending the "XFCP" class (Which I really don't know enough about how that really works ... but this might be where the performance could be improved ???),
and 0.06 meg seems an awful lot of memory for just for extending a class (do it 100 times and you've used 6 meg... that seems a lot).

 

[Adam Howard]

@Snog this is true, but usually I wouldn't expect that to amount to anything exactly readable.

But it does beg the question... @tenants what php version are you using and do you have any cache enabled in either XenForo or php?

 

[tenants]

No cache, local test environment php 5.4.0-ZS5.6.0

 

[Mike]

PHP is not a memory efficient language. High level languages generally aren't. Create a single array entry with an integer and see how much space it takes. If I recall, it was actually around 100 bytes (though I tested that years ago). Run memory_get_usage() in an empty script and see how much memory PHP uses just to start up.

What you're not accounting for here is that you're actually testing (not entirely in isolation) the memory allocated to store the opcodes to execute your code. You're testing the memory used by PHP to setup a class (mapping all the methods for example). You are of course testing different code paths in XF and some additional data to be loaded.

Another key thing is that if you have an opcode cache, you're talking about shared memory here. If we load 0.1MB of data and 1.9MB of opcodes, 10 simultaneous requests doesn't necessarily mean that 20MB of memory will be allocated. It could well be 2.9MB if there isn't a separate allocation for the shared memory. You'll see this in other (non-PHP places): it's very easy to double count memory usage when there are shared components (such as between threads).

If we're talking about simultaneous requests as well, that would also necessarily involve them all occurring while PHP is running. This is something that can often (and should generally) be below 100ms. The (non-shared) memory would be freed during PHP's clean up.

 

[Snog]

Without doing a ton of research of the actual PHP interpreter, I would bet on 64bit systems the minimum memory allocation is 64k and on 32bit systems it's 32k. That would explain the 0.06 memory allocation for an empty class extension. The easy way to prove that would be to do the same test on a 32bit system.

 

[tenants]

I understand the array comment, since you don't define the size of the array, so in the lower level language this array is probably always as big as it can be, thus it's going to take up a lot more resources than it should when using arrays in PHP (but then it's easier to program in, since you don't have to define the size of arrays, data type ... and garbage collect etc etc)

What you're not accounting for here is that you're actually testing (not entirely in isolation) the memory allocated to store the opcodes to execute your code. You're testing the memory used by PHP to setup a class (mapping all the methods for example).

I don't understand this as much. To extend the class, store it and set it up in memory, 0.06 Mb still seems like an awful lot of memory (but I guess not as much not when you take into account your original comment about arrays)

Another key thing is that if you have an opcode cache, you're talking about shared memory here. If we load 0.1MB of data and 1.9MB of opcodes, 10 simultaneous requests doesn't necessarily mean that 20MB of memory will be allocated. It could well be 2.9MB if there isn't a separate allocation for the shared memory. You'll see this in other (non-PHP places): it's very easy to double count memory usage when there are shared components (such as between threads)..

There was no opcode/cache/accelerator (other than that which comes with PHP / Zend)


As Mr Always Right (aka @Chris D) suggested , I did the same thing with event hints, and there was indeed a massive improvement (from 0.06 Mb originality used, to 0.001 Mb used with hints)

EmptyAddon On
Steady State: Timing: 0.0391 seconds Memory: 2.448 MB DB Queries: 10

EmptyAddon Off
Steady State: Timing: 0.0333 seconds Memory: 2.447 MB DB Queries: 10


So, event hints are without a doubt necessary for multiple addons (and where you want to keep low resource usage)


Same Add-on with event hints: lies, see below

 

[Mike]

Your event hint example is broken. The hint should be the class name. If you do that, you'll see little difference. The benefit in hints is not when the hint matches but when it doesn't, as it doesn't load a class unnecessarily (it can do a hash table look up instead).

The point I'm trying to make is that most (not all) of what you're counting here is internal to PHP, either through opcodes from compilation (which can be shared between requests) or from wiring up a class due to inheritance. The amount of memory used to define a class increases (linearly presumably) based on the number of methods and members it has. This is the total number of number of methods/members, not just ones that have been overridden.

The only real overhead that comes from XenForo is that our dynamic inheritance system requires an extra level of indirection between add-ons, which creates another level of inheritance. Of course, without that, the add-on system would like very different and be much more limited (we could not use an inheritance system and would have to define events for every single location where something should be changed).

In terms of seeming like a lot, as noted there is some overhead because of the extra layer, but have a look at something like this: http://nikic.github.io/2011/12/12/How-big-are-PHP-arrays-really-Hint-BIG.html This relates to the array thing, but it's relevant here. There is a lot of overhead to make the structs in C that power PHP.

 

[tenants]

I just saw the event hint issue, no wonder there was such a difference (or more importantly, an insignificant increase in memory when using event hints)

I tried that again (and tested that the event hint was actually doing something this time)

Hint: XenForo_ControllerPublic_Thread

EmptyAddon (with event hints) On:
Timing: 0.0377 seconds Memory: 2.474 MB DB Queries: 10

EmptyAddon (with event hints) Off:
Timing: 0.0387 seconds Memory: 2.424 MB DB Queries: 10

It's not as 'fair' a test any more, since I've some how lost my original starting conditions

So each empty addon (with or without event hints) still seems to be using about 0.05->0.06 MB
and this additional 0.06 MB per addon is simply unavoidable if we want to use the beautiful XenForo framework (and in most cases the memory is just allocation and not really used). Even if we didn't use the XenForo framework, the very nature of extending would probably still take a fair amount of memory

http://nikic.github.io/2011/12/12/How-big-are-PHP-arrays-really-Hint-BIG.html This relates to the array thing, but it's relevant here. There is a lot of overhead to make the structs in C that power PHP.

This was interesting for arrays

$array = new SplFixedArray(100000);

(56 bytes per element as opposed to 144 bytes when defining)

I'm more impressed that the starting conditions (XenForo without plugins) is a low as ~2.2MB, when a simple array is a factor of 18 times bigger than it should be in C

I guess that's just the way it is, each plugin will increase memory by about 0.06MB per plugin (before work)


Same Add-on with event hints (that actually work)