XF 2.1 XG2XFMG importer with [bd]AttachmentStore

Zig

Member
In light of the radio silence from sonnb on the progress/ETA for XenGallery on XF2, my team has decided to drop the addon and migrate to XFMG. I'm currently tasked with modifying ChrisD's XG2XFMG importer to work with our media items, which are stored in an AWS bucket via Xfrocks' [bd] Attachment Store addon. I've replaced \XF\Import\Importer\XenForoSourceTrait in the import script with a custom version to load up the necessary parts of Attachment Store and then check for directory existence using the has() method in Flysystem's EventableFilesystem class. For example:

Code:
            $engine = $this->storage()->getDefaultEngine();
            $options = $this->storage()->getDefaultOptionsPure($engine);
            $fs = new \Xfrocks\AttachmentStore\Fs(Fs::getAdapterDefault(), Fs::getAdapterConfig($engine, $options));

            // Big section of code that's identical to the original here. Removed for brevity.

            if ($this->requiresDataPath()) {
                if ($fullConfig['data_dir']) {
                    $data = rtrim($fullConfig['data_dir'], '/\\ ');

                    if (!$fs->has($data)) {
                        $errors[] = \XF::phrase('directory_x_not_found_is_not_readable', ['dir' => $data]);
                    }
                    $baseConfig['data_dir'] = $data; // to make sure it takes the format we expect
                } else {
                    $missingFields = true;
                }
            }

This allows the Importer to verify the existence of the directory in which our XenGallery images are stored. In addition I've modified the postSave behavior of XFMG\Import\Data\MediaItem in a similar fashion. For example:

Code:
            if ($this->originalPath)
            {
                $engine = $this->storage()->getDefaultEngine();
                $options = $this->storage()->getDefaultOptionsPure($engine);
                $fs = new \Xfrocks\AttachmentStore\Fs(Fs::getAdapterDefault(), Fs::getAdapterConfig($engine, $options));
                if ($fs->has($this->originalPath))
                {
                    \XF\Util\File::copyFileToAbstractedPath(
                        $this->originalPath, $mediaItem->getOriginalAbstractedDataPath()
                    );
                }
            }

We have almost a million albums and many more images, so it takes quite a while for the importer to run. All steps run, including the MediaItem import step. However, after about 12 hours (including the finalization phase) the result is a full import of our albums, album comments, and categories, but no imported Media Items. Newly generated Media Items work properly , so we know the addon itself plays nice with Attachment Store. It is only the Importer that doesn't work at this time.

Has anyone worked with the Importer system that could shed some light on what I'm missing? Should I be running Fs:putStream() instead of keeping the existing File Util command? Due to the time commitment required for each test I want to make sure I'm at least headed in the right direction before running it again.

Thanks!
zig
 
Last edited:
Top Bottom