Empty media after big board import

[The Geek]

Migrated from vb3.8x and used the Big Board importer instead of the standard XF one. Have spent the last several hours doing the import of Media and its complete, but nothing is in the media section.

I guess this is because of missing import_log data for albums, but I don't think the Big Board importer creates this. So - now everything is imported and live apart from the albums to media - how can I complete this?

Thanks in advance.

 

[teletubbi]

Only nothing to see or no data at all?

For the first one rebuild the indexes.

 

[The Geek]

Nothing to see and no data. That is after like 4 hours of importing. Its also busy tallying up media counts. But if there is no media... why would it take so long!?
Also my xf_import_log is empty, but I didn't specify it to be archived after the media import? I needed it for the 301's.

Really confused now.

 

[Chris D]

Yes, you need an import log.

This is courtesy of @Slavik

INSERT INTO xf_import_log (content_type, old_id, new_id)
    SELECT 'user', user_id, user_id
    FROM xf_user
ON DUPLICATE KEY UPDATE
    old_id = VALUES(old_id)

Before restarting the Gallery import process it should prompt you to archive the import log. The name of that archived import log is what you need to supply when setting up the import.

Usually the import log would contain a lot more, but for the Gallery import we only need to know about the users.

 

[Chris D]

Nothing to see and no data. That is after like 4 hours of importing. Its also busy tallying up media counts. But if there is no media... why would it take so long!?
Also my xf_import_log is empty, but I didn't specify it to be archived after the media import? I needed it for the 301's.

Really confused now.

In your case the length of time to import is the length of time it takes to query your old database, fetch the data to be imported, discover that it can't find the appropriate user to map it to and then repeat.

If it's doing the media counts right now, then it's currently going through each of your users, and counting their media and moving on.

The xf_import_log is empty after the Gallery import because nothing was imported.

 

[The Geek]

Thanks for such a quick reply Chris. I will Run the big board export on a test site to rebuild the log as I need really the whole thing. But reassuring to know that I just need the userids. Was getting frazzled there.

 

[The Geek]

The xf_import_log is empty after the Gallery import because nothing was imported.

I think the confusion is when I start the import, it gives me the option to Empty the import log without archiving (ah, no thanks... I need it!) or to Archive the import log to a new table. I must have chosen that option but that will leave xf_import_log empty and give me the failed import. What am I missing?

 

[Chris D]

Just to put it into some clear steps:

1. Make sure the xf_import_log table is empty.
2. Run the query I posted above, that will populate the user ID map into the xf_import_log table.
3. Go to the Import External Data page in the Admin CP.
4. You should get a notice about previously imported data being found. Choose the option to "Archive the import log" and give it a name. It defaults to archived_import_log.
5. When you set up the vBulletin Albums to XFMG importer it will ask you for the import log name, if you chose the default, this would be "archived_import_log".

Set everything else up as normal, and the import should complete successfully.

 

[The Geek]

Thats really helpful @Chris D (and @Slavik ). I am now importing and can verify I am getting data (and stuff is starting to appear) but have just run into:

Mysqli statement execute error : Duplicate entry '675' for key 'PRIMARY'

1. Zend_Db_Statement_Mysqli->_execute() in Zend/Db/Statement.php at line 297
2. Zend_Db_Statement->execute() in Zend/Db/Adapter/Abstract.php at line 479
3. Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 574
4. Zend_Db_Adapter_Abstract->insert() in XenForo/DataWriter.php at line 1624
5. XenForo_DataWriter->_insert() in XenForo/DataWriter.php at line 1613
6. XenForo_DataWriter->_save() in XenForo/DataWriter.php at line 1405
7. XenForo_DataWriter->save() in XenGallery/Model/Importers.php at line 166
8. XenGallery_Model_Importers->importMedia() in XenGallery/Importer/vBulletin38x.php at line 500
9. XenGallery_Importer_vBulletin38x->stepMedia() in XenForo/Importer/Abstract.php at line 97
10. XenForo_Importer_Abstract->runStep() in XenForo/ControllerAdmin/Import.php at line 187
11. XenForo_ControllerAdmin_Import->_runStep() in XenForo/ControllerAdmin/Import.php at line 135
12. XenForo_ControllerAdmin_Import->actionImport() in XenForo/FrontController.php at line 347
13. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
14. XenForo_FrontController->run() in /var/www/html/admin.php at line 13

 

[Chris D]

Was the import process interrupted at all?

What version of XFMG are you running?

The error suggests it is trying to import again, an item it has already imported.

Make a note of the media ID (675) and remove that from the database:

DELETE FROM xengallery_media WHERE media_id = 675

Then try the process again. If you get a different media ID, repeat the step above and remove that from the database too.

If you get to a point where it keeps repeating the same error "Duplicate entry '675'" then there may be a problem with the source data. I have seen, sometimes, an issue where some IDs can overlap across records, although there are some steps taken that should prevent this being an issue.

 

[The Geek]

I did that (and had to for next 3 keys) and now I just get an error "No data received" in the browser.

This is pulling the data remotely though I can query it no problem:

SELECT albumpicture.*, picture.*, user.*,
                    album.*, albumpicture.pictureid
                FROM albumpicture AS albumpicture
                LEFT JOIN album AS album ON
                    (albumpicture.albumid = album.albumid)
                LEFT JOIN picture AS picture ON
                    (albumpicture.pictureid = picture.pictureid)
                LEFT JOIN user AS user ON
                    (picture.userid = user.userid)
                WHERE
                     picture.state = 'visible'
                ORDER BY albumpicture.pictureid

And there are no dupe keys there. Perhaps its the next item that is causing a problem?

 

[The Geek]

This is the error log entry I get:

[Sat Apr 11 23:45:58.215088 2015] [core:notice] [pid 969] AH00052: child pid 15048 exit signal Segmentation fault (11)

 

[Slavik]

This is the error log entry I get:

[Sat Apr 11 23:45:58.215088 2015] [core:notice] [pid 969] AH00052: child pid 15048 exit signal Segmentation fault (11)

Are you runing any opcode caching or similar?

 

[The Geek]

APC?

 

[Slavik]

Try disabling that and restarting httpd

 

[The Geek]

Ok. I changed apc.enabled to 0 and restarted but got same issue (duplicate key) so I deleted the key(s) and got the same error. Then commented out the extension and restarted but same issue:
[Sun Apr 12 00:12:35.293108 2015] [core:notice] [pid 969] AH00052: child pid 24269 exit signal Segmentation fault (11)
apc isn't showing in phpinfo
Only difference is that after the "No data received" in browser, it reloads and gives me the duplicate key error again.

 

[Chris D]

Is the duplicate key message happening on the same IDs each time after the error?

If so it would likely suggest that perhaps it is a specific image that is causing the issue. It is likely going to be the "next one" so if the highest duplicate key is 675, then you might need to check the vBulletin albumpicture table for which pictureid comes after 675.

If you have a back up of this table, the easiest thing to do is probably just delete that row, then resolve the duplicate key errors and it will skip the now deleted image and if that was causing the problem then hopefully it will proceed as normal.

 

[The Geek]

We're on the same page, but so far haven't had much luck.

Each iteration it does 675 (ok), then 676 (ok) then 678 (ok). It then crashes so next run those 3 are there. I tried skipping the next row (681) but no good. Going to try a few and see if I can get any love.

 

[The Geek]

Still having problems and have got to this point:

In XenGallery_Model_Importers->importMedia(), a var_export after the $upload = new XenForo_Upload($xfAttachmentData['filename'], $tempFile); produces

XenForo_Upload::__set_state(array(
   '_fileName' => '8481-1227039917-580b7646d24350b0e7f9db83fc248963.jpg',
   '_extension' => 'jpg',
   '_tempFile' => '/var/www/html/internal_data/temp/xfIesf3c',
   '_imageInfo' => NULL,
   '_errorsChecked' => false,
   '_errors' => 
  array (  ),
   '_allowedExtensions' => 
  array (  ),
   '_maxFileSize' => 0,
   '_maxWidth' => 0,
   '_maxHeight' => 0,
   '_autoResize' => true,))

Then the next line: $upload->isImage() is where the fault comes.

Any ideas?

 

[Chris D]

Well, there'a definitely nothing out of the ordinary there.

The next step would be interrogating the image for obscure content (the XF code checks for executable PHP code hidden within an image, for example), storing its properties such as its width and height, applying any relevant constraints and automatically rotating the image based on EXIF properties (these doesn't apply here).

So, really, nothing here is screaming out at me as potentially being an issue unless there is something so dodgy in the image itself it's just causing PHP to crash.

If it is still only this image (or maybe a series of images in the same album) causing a problem, do what you can to skip them. As an alternative to manipulating the source database you could actually just change the name of the file on the file system; if it can't find it on the file system it will skip it.