CLI Cache Rebuilds 1.0

[Slavik]

Slavik submitted a new resource:

CLI Cache Rebuilds - Rebuild your XenForo caches from the command line.

This is a set of tools which allows for the rebuilding of XenForo caches from the command line, these have usually been reserved for my big board import customers (for threads and search), however recently have been expanded to include all default rebuilds and am now releasing these publicly.

Importantly and specifically, it avoids using the deferred system, so other actions which trigger the deferred system from the web will not interfere with the rebuilds.

Usage: Upload the cli...

Read more about this resource...

 

[Liam W]

Could you create a github repo for this? I have a few things I wouldn't mind contributing.

Liam

 

[maxicep]

Is Searchindex rebuild feature supports the elasticsearch search system if it set? @Slavik

 

[Slavik]

Is Searchindex rebuild feature supports the elasticsearch search system if it set? @Slavik

Yes

 

[maxicep]

I run the search rebuild last night and it was stopped when morning. Rebuild was not completed, i will go to run from web interface today :/

P.S: Have about 7million posts

 

[Slavik]

I run the search rebuild last night and it was stopped when morning. Rebuild was not completed, i will go to run from web interface today :/

P.S: Have about 7million posts

Any error thrown?

 

[maxicep]

No i didn't see any error record on server error logs. It may stop while database backup process i think but i don't know where is the stop point so i must start rebuild from beginning

You may modify this program for write a log while processing. So, if you write a process log real-time in cli directory, we can know where stopped the long rebuilds and start again where it left off.

 

[Slavik]

No i didn't see any error record on server error logs. It may stop while database backup process i think but i don't know where is the stop point so i must start rebuild from beginning

You may modify this program for write a log while processing. So, if you write a process log real-time in cli directory, we can know where stopped the long rebuilds and start again where it left off.

If the backup caused the database to become inaccessible that would stop it.

 

[maxicep]

Yes it may related with that issue but you should think the log file while processing rebuilds

 

[Slavik]

Could you create a github repo for this? I have a few things I wouldn't mind contributing.

Liam

https://github.com/SlavikXF/cli-rebuilds

 

[Slavik]

Yes it may related with that issue but you should think the log file while processing rebuilds

Incidentally, if the database is unavailable due to said backup, it wouldn't be able to write out an error to the error log anyway

I probably wouldnt include file-logging.

 

[Travis-Mc]

I can't figure out how to make this work for XMG cache rebuilds.. It's over my head. I've spent hours rebuilding using the regular web based tools. How much faster is this compared to using the regular web based tools?

 

[Slavik]

I can't figure out how to make this work for XMG cache rebuilds.. It's over my head. I've spent hours rebuilding using the regular web based tools. How much faster is this compared to using the regular web based tools?

It's marginally faster, the main reason for these tools is to kick them off in the background.

 

[Travis-Mc]

Thanks Slavik. I guess I'll just stick with the normal process.

 

[Accounting]

Hello, very usefull tool,

but what could i change in Threads.php in order to rebuild post position either please ?

 

[Slavik]

Hello, very usefull tool,

but what could i change in Threads.php in order to rebuild post position either please ?

Near the top just change the variable form false to true

 

[Accounting]

i tried, but it didn't change anything posts are still upside down in threads

 

[Wildcat Media]

Yeah, I know...still on 1.5. But, upgrade to 2.2 coming in November as I work out the last of the issues with this project.

But in the meantime, I'm having do deal with 1.5, Due to our hosting company updating to a newer default version of Java, I had to install ElasticSearch 6.8 and dump our indexes.

The web-based rebuild kept crashing. The CLI rebuild works until I get to Posts, as there are over 32 million posts, and it seems as though it's a random amount of time/posts that get indexed before I get "An unexpected error occurred. Please try again later." I can't babysit this thing for hours at a time, but the last time when it got to over 9 million posts, I noticed a high server load, from what appeared to be an influx of traffic. The ES log isn't showing anything amiss, and the XF server error log shows nothing either.

I don't think there is a way with the CLI to continue indexing where I left off, is there?

What I might do is shut down the forum overnight and let it run during low-traffic hours to see if I can get over the hump.

 

[Slavik]

I don't think there is a way with the CLI to continue indexing where I left off, is there?

Unfortunately not.