Speed up / Allow --jobs on php cmd.php xf-rebuild:attachment-optimization

[ekool]

I've been optimizing the attachments on one of our sites for over 24 hours... currently at 89,240 seconds. The process isn't halfway done.

It would be nice if there was some way to speed up this process or specify --jobs for this on the command line. I don't feel like this process should take nearly this long. The load on the machine is very low and there is clearly way more hardware that could be put to use. The site is live, and the server isn't breaking a sweat.

CPU: AMD EPYC 7251 (16) @ 2.10 GHz
Memory: 27.23 GiB / 62.27 GiB (44%)
Swap: 0 B / 4.00 GiB (0%)

 

[ekool]

There are some issues here...

The first major one is you say you want Google to index attachments, but in robots.txt you are specifically disallowing them from crawling the /attachments/ directory.

You have numerous references to directories/links only seen by logged in users. Search engines will never see these, unless you specifically give them login credentials which is uncommon.

Recommend putting directories in alphabetical order, this will help eliminate duplicates.

Your current robots.txt with notes...

The URL to viewing the attachments directly and viewing attachments in threads is different. I know there are duplicates, but I wanted to be doubly sure

This thread gives a little info on the attachments: https://xenforo.com/community/threa...to-disallow-attachments-in-robots-txt.143658/