1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Empty media after big board import

Discussion in 'Media Gallery Support' started by The Geek, Apr 11, 2015.

  1. The Geek

    The Geek Active Member

    Migrated from vb3.8x and used the Big Board importer instead of the standard XF one. Have spent the last several hours doing the import of Media and its complete, but nothing is in the media section.

    I guess this is because of missing import_log data for albums, but I don't think the Big Board importer creates this. So - now everything is imported and live apart from the albums to media - how can I complete this?

    Thanks in advance.
     
  2. teletubbi

    teletubbi Active Member

    Only nothing to see or no data at all?

    For the first one rebuild the indexes.
     
  3. The Geek

    The Geek Active Member

    Nothing to see and no data. That is after like 4 hours of importing. Its also busy tallying up media counts. But if there is no media... why would it take so long!?
    Also my xf_import_log is empty, but I didn't specify it to be archived after the media import? I needed it for the 301's.

    Really confused now.
     
  4. Chris D

    Chris D XenForo Developer Staff Member

    Yes, you need an import log.

    This is courtesy of @Slavik

    Code:
    INSERT INTO xf_import_log (content_type, old_id, new_id)
        SELECT 'user', user_id, user_id
        FROM xf_user
    ON DUPLICATE KEY UPDATE
        old_id = VALUES(old_id)
    Before restarting the Gallery import process it should prompt you to archive the import log. The name of that archived import log is what you need to supply when setting up the import.

    Usually the import log would contain a lot more, but for the Gallery import we only need to know about the users.
     
    Slavik likes this.
  5. Chris D

    Chris D XenForo Developer Staff Member

    In your case the length of time to import is the length of time it takes to query your old database, fetch the data to be imported, discover that it can't find the appropriate user to map it to and then repeat.

    If it's doing the media counts right now, then it's currently going through each of your users, and counting their media and moving on.

    The xf_import_log is empty after the Gallery import because nothing was imported.
     
  6. The Geek

    The Geek Active Member

    Thanks for such a quick reply Chris. I will Run the big board export on a test site to rebuild the log as I need really the whole thing. But reassuring to know that I just need the userids. Was getting frazzled there.
     
  7. The Geek

    The Geek Active Member

    I think the confusion is when I start the import, it gives me the option to Empty the import log without archiving (ah, no thanks... I need it!) or to Archive the import log to a new table. I must have chosen that option but that will leave xf_import_log empty and give me the failed import. What am I missing?
     
  8. Chris D

    Chris D XenForo Developer Staff Member

    Just to put it into some clear steps:
    1. Make sure the xf_import_log table is empty.
    2. Run the query I posted above, that will populate the user ID map into the xf_import_log table.
    3. Go to the Import External Data page in the Admin CP.
    4. You should get a notice about previously imported data being found. Choose the option to "Archive the import log" and give it a name. It defaults to archived_import_log.
    5. When you set up the vBulletin Albums to XFMG importer it will ask you for the import log name, if you chose the default, this would be "archived_import_log".
    Set everything else up as normal, and the import should complete successfully.
     
    jeffwidman likes this.
  9. The Geek

    The Geek Active Member

    Thats really helpful @Chris D (and @Slavik ). I am now importing and can verify I am getting data (and stuff is starting to appear) but have just run into:

    Mysqli statement execute error : Duplicate entry '675' for key 'PRIMARY'

    1. Zend_Db_Statement_Mysqli->_execute() in Zend/Db/Statement.php at line 297
    2. Zend_Db_Statement->execute() in Zend/Db/Adapter/Abstract.php at line 479
    3. Zend_Db_Adapter_Abstract->query() in Zend/Db/Adapter/Abstract.php at line 574
    4. Zend_Db_Adapter_Abstract->insert() in XenForo/DataWriter.php at line 1624
    5. XenForo_DataWriter->_insert() in XenForo/DataWriter.php at line 1613
    6. XenForo_DataWriter->_save() in XenForo/DataWriter.php at line 1405
    7. XenForo_DataWriter->save() in XenGallery/Model/Importers.php at line 166
    8. XenGallery_Model_Importers->importMedia() in XenGallery/Importer/vBulletin38x.php at line 500
    9. XenGallery_Importer_vBulletin38x->stepMedia() in XenForo/Importer/Abstract.php at line 97
    10. XenForo_Importer_Abstract->runStep() in XenForo/ControllerAdmin/Import.php at line 187
    11. XenForo_ControllerAdmin_Import->_runStep() in XenForo/ControllerAdmin/Import.php at line 135
    12. XenForo_ControllerAdmin_Import->actionImport() in XenForo/FrontController.php at line 347
    13. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
    14. XenForo_FrontController->run() in /var/www/html/admin.php at line 13
     
  10. Chris D

    Chris D XenForo Developer Staff Member

    Was the import process interrupted at all?

    What version of XFMG are you running?

    The error suggests it is trying to import again, an item it has already imported.

    Make a note of the media ID (675) and remove that from the database:
    Code:
    DELETE FROM xengallery_media WHERE media_id = 675
    Then try the process again. If you get a different media ID, repeat the step above and remove that from the database too.

    If you get to a point where it keeps repeating the same error "Duplicate entry '675'" then there may be a problem with the source data. I have seen, sometimes, an issue where some IDs can overlap across records, although there are some steps taken that should prevent this being an issue.
     
  11. The Geek

    The Geek Active Member

    I did that (and had to for next 3 keys) and now I just get an error "No data received" in the browser.

    This is pulling the data remotely though I can query it no problem:

    Code:
    SELECT albumpicture.*, picture.*, user.*,
                        album.*, albumpicture.pictureid
                    FROM albumpicture AS albumpicture
                    LEFT JOIN album AS album ON
                        (albumpicture.albumid = album.albumid)
                    LEFT JOIN picture AS picture ON
                        (albumpicture.pictureid = picture.pictureid)
                    LEFT JOIN user AS user ON
                        (picture.userid = user.userid)
                    WHERE
                         picture.state = 'visible'
                    ORDER BY albumpicture.pictureid
    And there are no dupe keys there. Perhaps its the next item that is causing a problem?
     
  12. The Geek

    The Geek Active Member

    This is the error log entry I get:

    [Sat Apr 11 23:45:58.215088 2015] [core:notice] [pid 969] AH00052: child pid 15048 exit signal Segmentation fault (11)
     
  13. Slavik

    Slavik XenForo Moderator Staff Member

    Are you runing any opcode caching or similar?
     
  14. The Geek

    The Geek Active Member

  15. Slavik

    Slavik XenForo Moderator Staff Member

    Try disabling that and restarting httpd
     
  16. The Geek

    The Geek Active Member

    Ok. I changed apc.enabled to 0 and restarted but got same issue (duplicate key) so I deleted the key(s) and got the same error. Then commented out the extension and restarted but same issue:
    [Sun Apr 12 00:12:35.293108 2015] [core:notice] [pid 969] AH00052: child pid 24269 exit signal Segmentation fault (11)
    apc isn't showing in phpinfo
    Only difference is that after the "No data received" in browser, it reloads and gives me the duplicate key error again.
     
  17. Chris D

    Chris D XenForo Developer Staff Member

    Is the duplicate key message happening on the same IDs each time after the error?

    If so it would likely suggest that perhaps it is a specific image that is causing the issue. It is likely going to be the "next one" so if the highest duplicate key is 675, then you might need to check the vBulletin albumpicture table for which pictureid comes after 675.

    If you have a back up of this table, the easiest thing to do is probably just delete that row, then resolve the duplicate key errors and it will skip the now deleted image and if that was causing the problem then hopefully it will proceed as normal.
     
  18. The Geek

    The Geek Active Member

    We're on the same page, but so far haven't had much luck.

    Each iteration it does 675 (ok), then 676 (ok) then 678 (ok). It then crashes so next run those 3 are there. I tried skipping the next row (681) but no good. Going to try a few and see if I can get any love.
     
  19. The Geek

    The Geek Active Member

    Still having problems and have got to this point:

    In XenGallery_Model_Importers->importMedia(), a var_export after the $upload = new XenForo_Upload($xfAttachmentData['filename'], $tempFile); produces

    PHP:
    XenForo_Upload::__set_state(array(
       
    '_fileName' => '8481-1227039917-580b7646d24350b0e7f9db83fc248963.jpg',
       
    '_extension' => 'jpg',
       
    '_tempFile' => '/var/www/html/internal_data/temp/xfIesf3c',
       
    '_imageInfo' => NULL,
       
    '_errorsChecked' => false,
       
    '_errors' => 
      array (  ),
       
    '_allowedExtensions' => 
      array (  ),
       
    '_maxFileSize' => 0,
       
    '_maxWidth' => 0,
       
    '_maxHeight' => 0,
       
    '_autoResize' => true,))
    Then the next line: $upload->isImage() is where the fault comes.

    Any ideas?
     
  20. Chris D

    Chris D XenForo Developer Staff Member

    Well, there'a definitely nothing out of the ordinary there.

    The next step would be interrogating the image for obscure content (the XF code checks for executable PHP code hidden within an image, for example), storing its properties such as its width and height, applying any relevant constraints and automatically rotating the image based on EXIF properties (these doesn't apply here).

    So, really, nothing here is screaming out at me as potentially being an issue unless there is something so dodgy in the image itself it's just causing PHP to crash.

    If it is still only this image (or maybe a series of images in the same album) causing a problem, do what you can to skip them. As an alternative to manipulating the source database you could actually just change the name of the file on the file system; if it can't find it on the file system it will skip it.
     

Share This Page