moving internal_data to block storage: symlink or $config['internalDataPath'] ?

[Sim]

Due to the rapidly growing disk usage for our galleries, I have added an external block storage volume (Linode) and then moved the entire internal_data directory to the new volume and set the $config['internalDataPath'] to that directory.

It's all working as expected - no issues.

However, I have noted that the latency of accessing files from the external volume is quite a bit higher than those on the native SSD drives (as expected based on @eva2000 's early testing).

Rough figures I'm seeing from NGINX Aplify monitoring shows an average of around 0.6ms io.wait_r the SSD compared to around 4.5ms for the block storage volume - so approximately 7-8 times slower on block storage.

Given that the internal_data directory contains a lot more than just attachments:

• attachments
• image_cache
• page_cache
• sitemaps
• temp
• templates

... I'm wondering whether this is actually the ideal setup (performance wise) to have the entire directory on the block storage volume?

Would we be better to keep internal_data on the main SSD storage, move only attachments to block storage volume and symlink to it?

I'm guessing it's not largely going to be an issue for the php files (templates) because the Opcache should take care of latency.

image_cache could be a little more problematic - but given that these files are not served directly by the web server but are served via php - there is already latency there anyway - not sure an extra 4.5ms to read the image from the block storage volume is going to have a significant impact.

Is there anything I haven't considered that might cause problems using a symlink and only moving attachments? Should I not bother an just leave it as is with the entire internal_data directory on the block storage volume?

 

[Kirby]

I'd keep code_cache on local SSD while moving internal_dataand configure paths in config.php accordingly - this is how we've setup our systems.

If you are using guest page caching I would also configure this for local storage or if possible Redis/Memcache.

 

[Sim]

I'd keep code_cache on local SSD while moving internal_dataand configure paths in config.php accordingly - this is how we've setup our systems.

If you are using guest page caching I would also configure this for local storage or if possible Redis/Memcache.

Ahh - I should have pointed out that I'm still on XF 1.5 ... will be a few months before I'm ready to upgrade to 2.x yet

I hadn't noticed the new variables in XF 2.x - I think I will revisit this post-upgrade.

I'm wondering if performing an XF1.5 - XF2.x upgrade with the $config['internalDataPath'] variable set is likely to cause an issue?

 

[Xon]

Have a look at my Attachment Improvements add-on (for XF1/XF2), as it support using X-Accel-Redirect to make nginx serve the attachments instead of php. Probably need to extend it to support XFMG however

I'm guessing it's not largely going to be an issue for the php files (templates) because the Opcache should take care of latency.

You want templates on local, as opcache will still do stat-calls to check if the file exists & is unchanged.

I'm wondering if performing an XF1.5 - XF2.x upgrade with the $config['internalDataPath'] variable set is likely to cause an issue?

The installer does respect those options.

 

[dknife]

I'd keep code_cache on local SSD while moving internal_dataand configure paths in config.php accordingly - this is how we've setup our systems.

Could you elaborate on how this is configured? My current setup is simply having my DO block storage mounted via fstab and then data/ and /internal_data as symlinks. This was from when I was using 1.5 so there was no concept of code cache then and after moving to 2.1 I realized that it's now pulling the code cache from block storage which is probably not ideal.

Is there a $config option to set code_cache outside of internal data?

Unless there are $config options for the various folders, I'm probably going to need to symlink specific folders to block, like attachments, image_cache, xengallery and leave the rest as local.

 

[Kirby]

We are running XF in a cluster with a clustered filesystem (GlusterFS) for storage, though we might phase this out in the future in favour of Ceph.
Using Ceph/RadosGW was not a viable option before XF 2 as XF 1 did require a filesystem for storage.

For performance reasons, (would be way too slow to have code-cache on shared storage) code-cache is kept local, but must be synced to all cluster nodes so we've got two paths

• One for writing/synchronizing to all nodes
• One for local readonly runtime/production access

So basically & simplified (I've left out chroot fuu, etc.) the config is

$config['fsAdapters'] = array(
    'data' => function () { return \XF\FsMounts::getLocalAdapter('/path/to/cluster-filesystem/data/public'); },
    'internal-data' => function () { return \XF\FsMounts::getLocalAdapter('/path/to/cluster-filesystem/data/internal'); },
    'code-cache' => function () { return \XF\FsMounts::getLocalAdapter('/path/to-cluster-sync/code-cache-write'); },
);

$config['codeCachePath'] = '/path/to/local/readnonly/code-cache';

 

[Sim]

You want templates on local, as opcache will still do stat-calls to check if the file exists & is unchanged.

FYI - I've reconfigured and decided to mount the volume directly in internal_data/attachments, so everything else in internal_data is now served from the local SSD, while the attachments are on the block storage volume. Much cleaner (other than the pesky lost+found directory which I have to exclude from my backups because it is not world readable) and solves any opcache performance issues.