XF 2.2 Getting errors similar to "ErrorException:Fatal Error:Allowed memory size of 1073741824 bytes exhausted(tried to allocate 2621440 bytes)" every moment

[Simplilearn]

Hi All,
we have written custom code to provide forum access to the emails VIA API.

We are using the existing code of Xenforo.
PHP:
$permissionUpdater = $this->service('XF:UpdatePermissions');
$permissionUpdater->setContent('node', $forumId)->setUser($user);
$permissionUpdater->updatePermissions($permissions);

After we released this code we experienced Deadlok issue as mentioned in https://xenforo.com/community/threads/deadlock-on-permission-api.205113/ and we are getting following errors every moment.

We are unable to do any edits on the nodes from the Admin Panel to see if we can provide permissions.It throws memory size errors with different stack trace. So we are completely blocked.

Could you please help us? Its been pending for over a month now.
Appreciate any insights provided

 

[Sim]

You're running out of memory - allocate more memory to PHP.

What is your memory_limit currently set to?



How much RAM do you have in your machine?

 

[Simplilearn]

Hi Sim,
Thanks for the quick response, following are the details

 

[Sim]

1G of RAM allocated to PHP is incredibly high.

How much physical RAM do you have?

How many PHP processes are running?

 

[Simplilearn]

Hi,

How much physical RAM do you have? 16GB

How many PHP processes are running? 26



Max Requests: 20

Process Idle Timeout: 10

Max Children: 5

Please let me know if you require more details

 

[Sim]

How much physical RAM do you have? 16GB

How many PHP processes are running? 26

26 PHP processes, each using 1GB of RAM for a total of 26GB used in a machine with only 16GB of physical RAM? That's not going to work very well.

I'd be setting memory_limit back to 128MB or there abouts and see how you go - you may have to boost it to 256MB, but you are unlikely to need to go higher than that unless you have some particularly poorly behaved scripts running.

I've got one of my larger sites running with it set to 256MB, and another with 384MB - but only because I have thousands and nodes, prefixes and gallery categories on that particular site.

My other 6 sites are all on the same server, running with a memory_limit of 128MB - which should be more than enough for most sites.

 

[Arantor]

That's not going to work very well.

PHP doesn’t immediately request all that memory at once. That’s just the limit it’s allowed to go to.

In any case what happened here is not the server running out of memory, but a script hitting that “allowed to go to” level. If it had actually run out of memory, the error would be “Out of memory” rather than “Allowed memory size”.

I’d suggest not changing the memory_limit at this point - something clearly wants all that memory! It does sound like there might be a bug in the permissions system, it shouldn’t really be deadlocking like that.

 

[Sim]

Yes - but the point is that no script should be using 1GB of RAM, so the server should not need that much available.

There is definitely a bug somewhere - but I doubt it is in the XF permissions system. The stack trace showing that error simply indicates what was executing at the point where the application tried to request more memory, but none was available - but that doesn't mean that was what used all the memory in the first place, it's just the proverbial "straw that broke the camel's back" so to speak.

we have written custom code to provide forum access to the emails VIA API.

You need to go back through your custom code and examine what you are doing to identify what is consuming all of your memory.

I pretty much guarantee your custom code is the problem here. You may need to rethink how your code is structured.

Perhaps if you give more details about your custom code and what it's trying to do - someone here might be able to identify the issue.

 

[Simplilearn]

Yes - but the point is that no script should be using 1GB of RAM, so the server should not need that much available.

There is definitely a bug somewhere - but I doubt it is in the XF permissions system. The stack trace showing that error simply indicates what was executing at the point where the application tried to request more memory, but none was available - but that doesn't mean that was what used all the memory in the first place, it's just the proverbial "straw that broke the camel's back" so to speak.



You need to go back through your custom code and examine what you are doing to identify what is consuming all of your memory.

I pretty much guarantee your custom code is the problem here. You may need to rethink how your code is structured.

Perhaps if you give more details about your custom code and what it's trying to do - someone here might be able to identify the issue.

this is our custom code

public function actionPostForumPermission(ParameterBag $params) {
    $user = $this->finder('XF:User')->where('email', $this->filter('user_email', 'str'))->fetchOne();
    if(empty($user)) {
        return $this->apiSuccess(["status"=>false,"message"=> 'not valid user']);
    }
    $record = $this->assertRecordExists('XF:Node', $this->filter('forum_id', 'uint'));
    if(empty($record)) {
        return $this->apiError("invalid Forum",400);
    }
    $nodePlugin = $this->plugin('SL\Atlas:Api:Smpl');
    $nodePlugin->setPermissions($user, $this->filter('forum_id', 'uint'));
    return $this->apiSuccess(["status"=>true,"message"=> 'access given to user']);

}

public function setPermissions($user, $forumId) {
    $permissionType = $this->filter('permission_type', 'str');
    $forumType = $this->filter('forum_type', 'str');

    if($permissionType == 'access') {
        $permissions = $this->getForumPermissions();
    } else if($permissionType == 'revoke') {
        $permissions = $this->getForumRevokePermissions($forumType);
    }
   

    /** @var \XF\Service\UpdatePermissions $permissionUpdater */
    $permissionUpdater = $this->service('XF:UpdatePermissions');
    $permissionUpdater->setContent('node', $forumId)->setUser($user);
    $permissionUpdater->updatePermissions($permissions);
}

 

[Sim]

this is our custom code

Exactly what is it your code is trying to achieve?

Can you explain the workflow that causes this to be triggered?

You mentioned in the other thread that this gets called frequently via the API - but I don't understand why you need to call it at all?

I suspect there is a better way to achieve what you are trying to do.

From what i can tell - you are giving someone access to do something in a node by editing the node permissions?

That's fine - but every time you call updatePermissions - you cause multiple database writes and then trigger a cache rebuild, which can all add up to a lot of database activity.

If this is being called frequently by some external process - then that will be causing the database to get hammered with requests and will impact on performance. I'm not surprised you are seeing deadlocks occur!

I suspect you need to rethink your approach. If you could provide more detail about what you are doing, we might be able to suggest something.

 

[Simplilearn]

Exactly what is it your code is trying to achieve?

Can you explain the workflow that causes this to be triggered?

You mentioned in the other thread that this gets called frequently via the API - but I don't understand why you need to call it at all?

I suspect there is a better way to achieve what you are trying to do.

From what i can tell - you are giving someone access to do something in a node by editing the node permissions?

That's fine - but every time you call updatePermissions - you cause multiple database writes and then trigger a cache rebuild, which can all add up to a lot of database activity.

If this is being called frequently by some external process - then that will be causing the database to get hammered with requests and will impact on performance. I'm not surprised you are seeing deadlocks occur!

I suspect you need to rethink your approach. If you could provide more detail about what you are doing, we might be able to suggest something.

we are having a forum for each course when a user purchased the course then we need to give access to that forum tagged to the course. so we created this custom API to call on every course assignment. can we do without cache rebuild

 

[Sim]

we are calling this API from our system through the Kafka cluster. 3 instances are calling this API frequently because of this in the community system we are getting a deadlock error.

How frequently is the API being called?

 

[Xon]

How many forums do you have? XenForo isn't designed to have more than a few thousand forums, and many add-ons don't handle large numbers of forums well.

What 3rd party add-ons do you have installed? It is very possible one of them isn't handling a large number of forum nodes gracefully.

 

[Simplilearn]

How frequently is the API being called?

per day nearly 5000

 

[Simplilearn]

How many forums do you have? XenForo isn't designed to have more than a few thousand forums, and many add-ons don't handle large numbers of forums well.

What 3rd party add-ons do you have installed? It is very possible one of them isn't handling a large number of forum nodes gracefully.

no third-party addon we have only one which is developed by us. node wise we are only having 600

 

[Sim]

per day nearly 5000

So every 20 seconds or so - yes, that's going to cause an issue.

You'll need to rethink your approach - the XenForo permissions system is not designed to be updated so frequently.

If you look at how that updatePermissions method is called elsewhere in the codebase, it is typically only the result of a manual admin action and thus the time taken to rebuild the permissions is not an issue.

Why do you actually need to update the permissions directly? Can't you just add a user to a group which gives them the desired permissions?

 

[Simplilearn]

So every 20 seconds or so - yes, that's going to cause an issue.

You'll need to rethink your approach - the XenForo permissions system is not designed to be updated so frequently.

If you look at how that updatePermissions method is called elsewhere in the codebase, it is typically only the result of a manual admin action and thus the time taken to rebuild the permissions is not an issue.

Why do you actually need to update the permissions directly? Can't you just add a user to a group which gives them the desired permissions?

We need give forum permission only for the course they purchased. And many forums we marked as a private forum. So if we go with permission group we can't create permission group for each forum right

 

[Sim]

We need give forum permission only for the course they purchased. And many forums we marked as a private forum. So if we go with permission group we can't create permission group for each forum right

But why are you doing this 5,000 times a day? Are you selling 5,000 courses a day?

 

[Xon]

So if we go with permission group we can't create permission group for each forum right

You should go for a user-group per private forum. Adding a group to a user is vastly cheaper in both memory and CPU time.

XenForo is optimized for a bound number of per-user permissions. This is why you are running out of memory, it expects the per-user permission-sets to be low.

 

[Simplilearn]

You should go for a user-group per private forum. Adding a group to a user is vastly cheaper in both memory and CPU time.

XenForo is optimized for a bound number of per-user permissions. This is why you are running out of memory, it expects the per-user permission-sets to be low.

If we have a user group per private forum it will be more user groups right. Is there any other way?
1. What will happen if we stop refreshing cache from API?
2. If we are not refreshed the cache when it will refresh automatically?
3. Can we use an external cache like Redis?
4. What is the default cache in xenforo?