Implemented Rebuild Caches through CLI

Alpha1

Alpha1

Well-known member
Now that we have a CLI tool to install and upgrade addons, I would like to request a function to Rebuild Caches through CLI.
This allows us to simply initiate the task and get on with whatever we need to do. No need to wait for deferred to complete.

What I mean by this is any task that is in XF1 here: /admin.php?tools/rebuild
 
Last edited:
Upvote 14
This suggestion has been implemented. Votes are no longer accepted.
Yeah, something like this would be a life saver. Especially when rebuilding search index takes countless hours without Enhanced Search
 
We've added the ability to rebuild various things via the CLI using the following commands as of the next release (a mix of XF and add-on rebuilds here):
Code: 
    xf-rebuild:xfrm-user-resource-counts
    xf-rebuild:xfrm-resource-items
    xf-rebuild:xfmg-media-items
    xf-rebuild:xfmg-user-counts
    xf-rebuild:xfmg-albums
    xf-rebuild:threads
    xf-rebuild:search
    xf-rebuild:forums
    xf-rebuild:users
This is only a selected portion of the available rebuilds (the important ones), though we could add more in the future, and doing so is crazily simple. As an example, here's the rebuild forums command class:
PHP: 
<?php

namespace XF\Cli\Command\Rebuild;

class RebuildForums extends AbstractRebuildCommand
{
    protected function getRebuildName()
    {
        return 'forums';
    }

    protected function getRebuildDescription()
    {
        return 'Rebuilds forum counters.';
    }

    protected function getRebuildClass()
    {
        return 'XF:Forum';
    }
}
 
@Chris D Any change you can alter the rebuild logic for the CLI rebuild logic + search handler to work down from a known maximum id? (ie the entity does a select max(primary-id) from table).

This makes it easy to fork a bunch and hand out known id-range batch sizes with the stop check being a simple >0 id check without any additional queries.

I'm using a hacky version of this approach for my XF1 Elastic Search Essentials add-on, and it is a 3-3.5 times speedup with 4 worker processes. Which helps massively with +20 million posts.
 
Last edited:
That would have to be a totally separate suggestion.

The job command code itself just essentially calls the normal job and runs it in a loop so there's not particularly anything we can do in the command class itself.
 
@Chris D We are busy migrating our custom forum to XF2. All is working quite well but rebuilding XFMG thumbnails is a thing...

We have over a million files so rebuilding with the GUI is a pain in the *ss. Can you add this to the CLI cmd tool?
(Maybe it's an idea to add all the GUI options in the CLI tool).
 
Are all the CLI commands in the manual somewhere? They might be but I'm struggling to find them and have to keep searching the forums and hope to stumble on them like the example above of how to rebuild the search index.
 
You can pull down a list of commands using the php cmd.php list command:
Code: 
Console Tool

Usage:
  command [options] [arguments]

Options:
  -h, --help            Display this help message
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi            Force ANSI output
      --no-ansi         Disable ANSI output
  -n, --no-interaction  Do not ask any interactive question
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

Available commands:
  help                                  Displays help for a command
  list                                  Lists commands
 xf
  xf:addon-disable                      Disables the specified add-on. If no add-on ID is provided, all enabled add-ons will be disabled.
  xf:addon-enable                       Enable the specified add-on. If no add-on ID is provided, all disabled add-ons will be enabled.
  xf:addon-install                      Installs the specified add-on
  xf:addon-rebuild                      Rebuilds the specified add-on data.
  xf:addon-sub-action                   Runs a add-on sub-action in a separate process. Do not run directly!
  xf:addon-uninstall                    Uninstalls the specified add-on
  xf:addon-upgrade                      Upgrades the specified add-on
  xf:convert-utf8mb4                    Converts XenForo tables to utf8mb4
  xf:file-check                         Performs a file health check
  xf:import                             Executes an import configured via the control panel
  xf:import-finalize                    Finalize an import configured via the control panel
  xf:install                            Installs XenForo
  xf:legacy-cleanup                     Creates a new _legacy directory in your XF root and moves all known unused XF1 files there.
  xf:rebuild-master-data                Rebuilds the core XF master data.
  xf:upgrade                            Upgrades XenForo
 xf-addon
  xf-addon:build-release                Performs an export of the current XML data and saves a ZIP file to _releases
  xf-addon:bump-version                 Bumps the version of the specified add-on
  xf-addon:create                       Creates an XF add-on and writes out the basic addon.json file.
  xf-addon:disable                      Disables the specified add-on. If no add-on ID is provided, all enabled add-ons will be disabled.
  xf-addon:enable                       Enable the specified add-on. If no add-on ID is provided, all disabled add-ons will be enabled.
  xf-addon:export                       Exports the XML files for an add-on
  xf-addon:install                      Installs the specified add-on
  xf-addon:install-step                 Runs a specific step from the specified add-on Setup class.
  xf-addon:rebuild                      Rebuilds the specified add-on data.
  xf-addon:sync-json                    Syncs the contents of the add-on JSON file to the database, updating the title, version and JSON hash as necessary
  xf-addon:uninstall                    Uninstalls the specified add-on
  xf-addon:uninstall-step               Runs a specific step from the specified add-on Setup class.
  xf-addon:upgrade                      Upgrades the specified add-on
  xf-addon:upgrade-step                 Runs a specific step from the specified add-on Setup class.
  xf-addon:validate-json                Validates the contents of the add-on JSON file to ensure all of the required values exist and are of the correct type.
 xf-designer
  xf-designer:disable                   Disables designer mode on the specified style
  xf-designer:enable                    Enables designer mode on the specified style
  xf-designer:export                    Exports modified templates from the database to the file system for the specified designer mode.
  xf-designer:export-style-properties   Exports style properties for the specified designer mode ID.
  xf-designer:export-templates          Exports templates for the specified designer mode ID.
  xf-designer:import                    Imports designer files from the file system for the specified designer mode.
  xf-designer:import-style-properties   Imports style properties from specified designer mode files
  xf-designer:import-templates          Imports templates from specified designer mode files
  xf-designer:rebuild-metadata          Rebuilds metadata hashes based on file system content
  xf-designer:revert-template           Reverts the specified template.
  xf-designer:sync-templates            Syncs the contents of the template files to the DB for the specified designer mode, applying version number updates
  xf-designer:touch-template            Marks the specified template as modified in the specified style and exports it.
 xf-dev
  xf-dev:class-lint                     Checks that all classes can be loaded without conflicts
  xf-dev:compare-schema                 Compares database schemas for consequential differences
  xf-dev:entity-class-properties        Applies class properties to type hint columns, getters and relations
  xf-dev:export                         Exports all data to development files
  xf-dev:export-admin-navigation        Exports Admin navigation to development files
  xf-dev:export-admin-permissions       Exports Admin permissions to development files
  xf-dev:export-advertising-positions   Exports advertising positions to development files
  xf-dev:export-bb-code-media-sites     Exports bb code media sites to development files
  xf-dev:export-bb-codes                Exports bb codes to development files
  xf-dev:export-class-extensions        Exports class extensions to development files
  xf-dev:export-code-event-listeners    Exports code event listeners to development files
  xf-dev:export-code-events             Exports code events to development files
  xf-dev:export-content-types           Exports content types to development files
  xf-dev:export-cron-entries            Exports cron entries to development files
  xf-dev:export-help-pages              Exports help pages to development files
  xf-dev:export-member-stats            Exports member stats to development files
  xf-dev:export-navigation              Exports Navigation to development files
  xf-dev:export-options                 Exports options to development files
  xf-dev:export-permissions             Exports Permissions to development files
  xf-dev:export-phrases                 Exports phrases to development files
  xf-dev:export-routes                  Exports Routes to development files
  xf-dev:export-style-properties        Exports style properties to development files
  xf-dev:export-template-modifications  Exports template modifications to development files
  xf-dev:export-templates               Exports templates to development files
  xf-dev:export-widget-definitions      Exports widget definitions to development files
  xf-dev:export-widget-positions        Exports widget positions to development files
  xf-dev:generate-entity-dw             Generates an entity from a legacy DataWriter
  xf-dev:generate-phpstorm-meta         Generates a .phpstorm.meta.php file for dynamic return type hinting
  xf-dev:generate-schema-entity         Generates schema code from an entity
  xf-dev:import                         Imports all data from development files
  xf-dev:import-admin-navigation        Imports admin navigation from development files
  xf-dev:import-admin-permissions       Imports admin permissions from development files
  xf-dev:import-advertising-positions   Imports advertising positions from development files
  xf-dev:import-bb-code-media-sites     Imports bb code media sites from development files
  xf-dev:import-bb-codes                Imports bb codes from development files
  xf-dev:import-class-extensions        Imports class extensions from development files
  xf-dev:import-code-event-listeners    Imports code event listeners from development files
  xf-dev:import-code-events             Imports code events from development files
  xf-dev:import-content-types           Imports content types from development files
  xf-dev:import-cron-entries            Imports cron entries from development files
  xf-dev:import-help-pages              Imports help pages from development files
  xf-dev:import-member-stats            Imports member stats from development files
  xf-dev:import-navigation              Imports navigation from development files
  xf-dev:import-options                 Imports options from development files
  xf-dev:import-permissions             Imports permissions from development files
  xf-dev:import-phrases                 Imports phrases from development files
  xf-dev:import-routes                  Imports routes from development files
  xf-dev:import-style-properties        Imports style properties from development files
  xf-dev:import-template-modifications  Imports template modifications from development files
  xf-dev:import-templates               Imports templates from development files
  xf-dev:import-widget-definitions      Imports widget definitions from development files
  xf-dev:import-widget-positions        Imports widget positions from development files
  xf-dev:metadata-clean                 Cleans up the metadata JSON files and removes missing entries
  xf-dev:prep-releases                  Takes the necessary steps to ensure you have the latest data and export it for releases
  xf-dev:rebuild-caches                 Rebuilds various caches
  xf-dev:rebuild-metadata               Rebuilds metadata hashes based on file system content
  xf-dev:recompile                      Recompiles all template/phrase data
  xf-dev:recompile-phrases              Recompiles phrases
  xf-dev:recompile-style-properties     Recompiles style properties
  xf-dev:recompile-templates            Recompiles parsed templates
  xf-dev:sync-templates                 Syncs the contents of the template files to the DB, applying version number updates
  xf-dev:touch-template                 Updates the version number of the specified template
  xf-dev:update-version                 Updates the XF version to the same version as in the XF files.
  xf-dev:upgrade-step                   Runs a specific upgrade step from a specified upgrade class.
 xf-rebuild
  xf-rebuild:forums                     Rebuilds forum counters.
  xf-rebuild:search                     Rebuilds the search index.
  xf-rebuild:threads                    Rebuilds thread counters.
  xf-rebuild:users                      Rebuilds user counters and caches.
  xf-rebuild:xfmg-albums                Rebuilds album counters.
  xf-rebuild:xfmg-media-items           Rebuilds media item counters.
  xf-rebuild:xfmg-user-counts           Rebuilds media related user counters.
  xf-rebuild:xfrm-resource-items        Rebuilds resource item counters.
  xf-rebuild:xfrm-user-resource-counts  Rebuilds resource related user counters.
 
Hi @Jake B. , created the php file and ran the php cmd.php xf-rebuild:attachment-thumbnails but that doesn't work:
Code: 
Command "xf-rebuild:attachment-thumbnails" is not defined.
 
duderuud said:
Hi @Jake B. , created the php file and ran the php cmd.php xf-rebuild:attachment-thumbnails but that doesn't work:
Code: 
Command "xf-rebuild:attachment-thumbnails" is not defined.
Click to expand...

So I got this too, but then I realised that the instructions above are missing an important detail...
Jake B. said:
src/XF/Cli/Command/RebuildAttachmentThumb.php
Click to expand...

should be;

src/XF/Cli/Command/Rebuild/RebuildAttachmentThumb.php

ensure you delete the file you placed in ../Command/ before running the process
 
Last edited:

Similar threads

A
XF 2.2 Add Jobs To Rebuild Caches For Custom Addon
Replies
1
Views
24
Jeremy P
Jeremy P
kbas
  • Question Question
XF 2.3 error message when entering the cache rebuild
Replies
0
Views
26
kbas
kbas
Stuart Wright
  • Solved
XF 2.2 Is there a list of CLI commands to run the rebuild caches routines?
Replies
2
Views
331
Stuart Wright
Stuart Wright
Chromaniac
Fixed Optimizing profile banners and avatars to WebP through CLI does not seem to do anything
Replies
18
Views
778
nodle
nodle
texkev
  • Question Question
XF 2.1 Where did "Rebuild Caches" tool go??
Replies
3
Views
480
texkev
texkev
Back
Top Bottom