XF 2.3 cmd.php xf-rebuild:attachment-optimization

[MattW]

I set a job going for a customer who's site I'm hosting to optimise their attachments. During this, they upgraded 2 addons, and suggested that this stopped the optimise job:

total 28284     
-rw-rw-rw-. 1 30408 Aug 9 23:02 17899-c75ba0b2e7203815e21c141e2fc25731.data
-rw-rw-rw-. 1 8484 Aug 9 23:02 17900-15fafe33bfd8623972363ab39a57b4e3.data
-rw-rw-rw-. 1 44758 Aug 9 23:02 17901-627656ef0b7d0f3da5d0bec4b91d0b71.data
-rw-rw-rw-. 1 11972 Aug 9 23:02 17902-54fbcb8730eb9d8c3dc44ed10f5775e0.data
-rw-rw-rw-. 1 13116 Aug 9 23:02 17903-632391108e1cf095b85276258bf3c63e.data
-rw-rw-rw-. 1 11220 Aug 9 23:02 17905-a101b70c769f2889c6dd6886e115596b.data
-rw-rw-rw-. 1 13206 Aug 9 23:02 17906-09638f54a281727b8db38ad9f9c97b49.data
-rw-rw-rw-. 1 757854 Aug 9 23:02 17907-f2dbe941b83e3f19284fdf217137b180.data
-rw-r--r--. 1 5177597 Dec 28 2020 17908-9df0f913be89c31031e0c1664813a188.data
-rw-r--r--. 1 320507 Dec 28 2020 17909-b89e0a14537d0672f524bc7e9a9f2997.data
-rw-r--r--. 1 1548872 Dec 31 2020 17910-ea5f2549bfad7d43f4844e3d7afcfe28.data
-rw-r--r--. 1 117025 Jan 1 2021 17911-fc413d2bef4a50bf51fabe3a57db13f3.data
-rw-r--r--. 1 760 Jan 1 2021 17912-4b6ef48a3c95fcdb039909dddb2b91cc.data
-rw-r--r--. 1 760 Jan 1 2021 17913-4b6ef48a3c95fcdb039909dddb2b91cc.data
-rw-r--r--. 1 760 Jan 1 2021 17914-4b6ef48a3c95fcdb039909dddb2b91cc.data
-rw-r--r--. 1 609 Jan 1 2021 17915-6ebbf23c39a3194cee7957ef68e61c82.data
-rw-r--r--. 1 760 Jan 1 2021 17916-4b6ef48a3c95fcdb039909dddb2b91cc.data
-rw-r--r--. 1 760 Jan 1 2021 17917-4b6ef48a3c95fcdb039909dddb2b91cc.data
-rw-r--r--. 1 2052026 Jan 3 2021 17919-cc3b4c58e06bf6eb537e9c9637d61e66.data
-rw-r--r--. 1 300799 Jan 10 2021 17923-4e733d7cdd99a58c0b9d49ea28d02b57.data
-rw-r--r--. 1 164336 Jan 10 2021 17924-ddbaab25e60efc0899e76d74f5fe3161.data
-rw-r--r--. 1 43793 Jan 10 2021 17925-4d347727c88465afd301cee1fc1f7775.data
-rw-r--r--. 1 465682 Jan 11 2021 17926-d56ab9beb8a702186f8f0773f7e8421b.data
-rw-r--r--. 1 3814006 Jan 11 2021 17927-e8200b95998e0c9b683cf420bb33b72a.data
-rw-r--r--. 1 3323012 Jan 11 2021 17928-a284412eeace2984ca7d3b971852ea0e.data
-rw-r--r--. 1 389473 Jan 22 2021 17929-2d551db3aa45a955e3f82c78ed2cf517.data
-rw-r--r--. 1 182380 Jan 22 2021 17930-12dbffdbd31609db35d904f267c98b06.data
-rw-r--r--. 1 361261 Jan 22 2021 17931-fa19d493473a91aeb1467bf78c9187c6.data
-rw-r--r--. 1 1865569 Jan 22 2021 17932-556153549d26e000c01992710069d164.data
-rw-r--r--. 1 245933 Jan 23 2021 17933-e3fb969d61d3b9993151b7317692ce3b.data
-rw-r--r--. 1 3227919 Jan 28 2021 17994-3c744f1a4589068d56776e79574e929d.data
-rw-r--r--. 1 2251623 Jan 29 2021 17995-d9766a3e875649cb9dd3e2c06855a106.data
-rw-r--r--. 1 2001118 Jan 30 2021 17998-faf435b0109f8bb99c9cf43c0f11002e.data
-rw-r--r--. 1 119923 Feb 2 2021 17999-52ecf09983d4716986052b436f234e05.data

I can see the files stopped at 17907 at 23:02 last night, 15 minutes before they messaged me to tell me they upgraded the 2 addons.

Firstly, is this something that can happen when a job is being ran via command line (they upgraded the addons via the ACP)?
Secondly, I've ran a --resume command, and no further optimisation occurs:

php cmd.php xf-rebuild:attachment-optimization --resume

There is still a full folder of files that aren't optimised

# du -sh 18/
187M    18/

 

[Jeremy P]

Upgrading add-ons wouldn't interrupt the job, no. Were they running the job from a browser and stopped it or closed the tab to do the upgrades? Is the job still present in the xf_job table?

Secondly, I've ran a --resume command, and no further optimisation occurs:

If --resume fails then there is no matching job, though just running the command regularly will optimize any attachments which were not already optimized. Are the attachments marked as optimized in xf_attachment_data?

 

[MattW]

Upgrading add-ons wouldn't interrupt the job, no. Were they running the job from a browser and stopped it or closed the tab to do the upgrades? Is the job still present in the xf_job table?

I was running the optimisation job from the command line, they upgraded the addons via the ACP.

Is the job still present in the xf_job table?

Yes:

> SELECT * FROM xf_job;
+--------+--------------------------------------------------+-------------------------------+--------------------------------------------------------------------------------------------------------------------+----------------+--------------+---------------+----------+----------+
| job_id | unique_key                                       | execute_class                 | execute_data                                                                                                       | manual_execute | trigger_date | last_run_date | attempts | priority |
+--------+--------------------------------------------------+-------------------------------+--------------------------------------------------------------------------------------------------------------------+----------------+--------------+---------------+----------+----------+
|      2 | cron                                             | XF\Job\Cron                   | a:0:{}                                                                                                             |              0 |   1723267927 |    1723267868 |        0 |      100 |
|      9 | xfCollectStats                                   | XF\Job\CollectStats           | a:0:{}                                                                                                             |              0 |   1724188590 |    1723192210 |        0 |      100 |
|   8023 | xfUpgradeCheck                                   | XF\Job\UpgradeCheck           | a:0:{}                                                                                                             |              0 |   1723312284 |    1723195420 |        0 |      100 |
| 406796 | xfRebuildJob-attachment-optimization             | XF\Job\AttachmentOptimization | a:4:{s:5:"steps";i:784;s:5:"start";i:17907;s:5:"batch";d:9;s:6:"resume";b:0;}                                      |              1 |   1723235918 |    1723265096 |        0 |      100 |
| 406958 | trendingContent_66da1e01f1c91b1e2823560fae7dbc74 | XF\Job\TrendingResult         | a:4:{s:5:"order";s:3:"hot";s:8:"duration";i:7;s:12:"content_type";s:0:"";s:20:"content_container_id";i:0;}         |              0 |   1723267920 |          NULL |        0 |      100 |
| 406959 | trendingContent_1c46003ca3d844c59a0a83bf5f3fa70b | XF\Job\TrendingResult         | a:4:{s:5:"order";s:3:"hot";s:8:"duration";i:7;s:12:"content_type";s:7:"sc_item";s:20:"content_container_id";i:60;} |              0 |   1723268088 |          NULL |        0 |      100 |
+--------+--------------------------------------------------+-------------------------------+--------------------------------------------------------------------------------------------------------------------+----------------+--------------+---------------+----------+----------+

 

[MattW]

Are the attachments marked as optimized in xf_attachment_data?

No, the optimized flag stops on the ID 17907 which is the same as the timestamp on the files in the folder:

 

[Jeremy P]

Interesting. When you run the --resume, does it just return immediately with no feedback? There may be an error getting swallowed somewhere if there's nothing in the logs. I did uncover a few weeks ago that imagewebp (used for optimization with GD) can crash PHP in some cases without any opportunity for error handling, which is terrible.

 

[MattW]

Interesting. When you run the --resume, does it just return immediately with no feedback?

Correct, no feedback to the command prompt.

I’ve asked them to make my account admin so I can check if something is erroring or flagged in the admin panel.

 

[Jeremy P]

If so, it may be enough to simply mark attachment 17908 (and possibly 17909 as they seem related) as optimized so it gets skipped over. It's likely failing to decode/encode that particular image somehow.

 

[MattW]

 

[Jeremy P]

Yeah that'll be it then. There's an open bug report about this, I think we'll have to register a shutdown function to try and mark these as optimized. Marking them as optimized manually should work around this in the meantime.

 

[MattW]

Yeah that'll be it then. There's an open bug report about this, I think we'll have to register a shutdown function to try and mark these as optimized. Marking them as optimized manually should work around this in the meantime.

Thanks, marking those as optimized in the table has allowed the job to continue. Assuming this is specific to GD, and probably wouldn't occur with ImageMagick? Issue here is PHP8.3 on cPanel doesn't compile Imagick correctly.

https://support.cpanel.net/hc/en-us/community/posts/19653628594071-Installing-PHP-PECL-imagick-fails-on-PHP-8-3

They've been waiting 8 months so far!

 

[Jeremy P]

I haven't tested this with Imagick yet so I'm not sure. It may work due to differences in decoding, but they both use libwebp for encoding so it's also possible there's some kind of issue with the source image.

I do have Imagick working on PHP 8.3 (without any trickery, some how).

 

[MattW]

I do have Imagick working on PHP 8.3 (without any trickery, some how).

It seems it’s specific to cPanel. I have it working as well on centminmod, and cPanel with Cloudlinux (and Cloudlinux’s hardened PHP versions).