Not a bug Possible Importer Memory Leak

[Warchamp7]

Affected version

Xenforo 2.0.2

Currently trying to a very large import for XenTorneo2 in a XF1 -> XF2 upgrade.

I've added some echo statements to the importer so I could have some better insight into when and where it's failing using CLI

Most recently, I got this error

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 32 bytes) in /---/XF/Db/Mysqli/Statement.php on line 140

This happened around the 140,000th row out of 900,000~

With PHP set to 256M, it failed around 65,000~ which is why I suspect a memory leak.

Before I added more verbose logging to see how far it was getting I had failures on this import step even with PHP set to 6144M

Edit:
Jaxel has sent me an updated importer that uses XenForo's StepRunner system, will report back if that does not solve the issue.

Spoiler

Here's the importer code with my quick and dirty changes. I'm sure there are XenForo functions to properly do the things I'm doing, but this was just to quickly get a read on where it was failing.

public function stepMatches(StepState $state)
    {
        $total = $this->sourceDb->fetchAll('SELECT count(*) AS totalMatches FROM EWRtorneo_matches'); // Logging
        $totalMatches = $total[0]['totalMatches']; // Logging
     
        $this->app->db->emptyTable('ewr_torneo_matches');
     
        $offset = 0; // Logging
        $chunk = 5000; // Logging
     
        while($offset < $totalMatches) {
         
            echo "\n - - Match Import Chunk " . $offset / $chunk . ". Offset: " . $offset . ", Chunk: " . $chunk . ", Total: " . $totalMatches . "\n"; // Logging
     
            $matches = $this->sourceDb->fetchAll('SELECT * FROM EWRtorneo_matches ORDER BY match_id LIMIT ' . $chunk . ' OFFSET ' . $offset);
         
            $ticker = 0; // Logging
         
            foreach ($matches AS $match)
            {
                if($ticker >= 100) {
                    echo "|"; // Logging
                    $ticker = 0; // Logging
                }
                $import = $this->app->em->create('EWR\Torneo:Match');
                $import->bulkSet([
                    'event_id' => $match['event_id'],
                    'match_id' => $match['match_id'],
                    'match_date' => $match['match_date'],
                    'match_round' => $match['match_round'],
                    'match_scores' => $match['match_scores'],
                    'match_p1' => $match['match_p1'],
                    'match_p2' => $match['match_p2'],
                    'match_winner' => $match['match_win'],
                    'match_media' => $match['match_media'],
                ], ['forceSet' => true]);
                $import->save();
             
                $ticker++; // Logging
             
                $state->imported++;
            }
         
            unset($matches);
         
            $offset += $chunk; // Logging
     
        }

        return $state->complete();
    }

I'm probably going to modify the importer to break every 50,000 rows and run it multiple times without emptying the table, but I thought I'd bring it to your attention as a potential bug.

 

[Warchamp7]

Jaxel has sent me an updated importer that uses XenForo's StepRunner system, will report back if that does not solve the issue.

 

[Chris D]

Yeah that code in its current form would certainly cause problems.