XF 1.5 Elasticsearch issues after power failure. How to rebuild from scratch?

[Derek Zeanah]

I had a power loss yesterday. My Xenforo server came back up fine, but Elasticsearch is now having issues. Rebuilding the index isn't a fix, and I've seen this once before (also related to a power loss event) but I don't remember what I did to fix it.

We're a big board, but far from the largest.



I'm just not sure where to go from here. I think all I want to do is delete the existing search index and have it recreated, but I'd prefer to get some guidance before doing database surgery.

Here's the list of errors I'm seeing. This list represents deleting the index and having it automatically rebuilt this morning (I'd disabled it overnight.)

 

[Kirby]

php cmd.php xf-rebuild:search --truncate

You might also want to increase the batch size with --batch=<size>

I usually use a batch size of 10K, default is 500.

Edit: Forget what I wrote.
You are using 1.5, not 2.X

 

[Derek Zeanah]

Yeah, I was searching google trying to figure out where cmd.php is because I sure couldn't find it.

Someday I'll upgrade to 2 on that site - I've got it running successfully on two other sites - but the moderators there have been with me for 20 years and are....a bit adverse to change, so we've been holding off.

 

[Derek Zeanah]

I wonder. Is the fix going to be to go into MySQL and truncate one of the xf_search tables, then rebuild the cache? If so, does anyone remember which one?

 

[AndyB]

You might like this add-on:

https://xenforo.com/community/resources/elasticsearch.4218/

 

[Derek Zeanah]

Thanks, Andy. I'll check it out.

 

[Derek Zeanah]

Well, Andy's add-on helped me delete the index, which I promptly rebuilt, but searches still say "The search could not be completed. Please try again later." And I'm still getting similar errors in my error log.

XenForo_Exception: Elasticsearch error: {"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [discussion_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"thehighroad","node":"kNx6SbmuSiiInhZdnBzOqA","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [discussion_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [discussion_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.","caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [discussion_id] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}} - library/XenES/Search/SourceHandler/ElasticSearch.php:1056

 

[Kirby]

Place the attached script in XF root directory and execute via CLI, this should rebuild the index - but I am not sure if XFES on XenForo 1.5 would work with ES 7.10 at all?

No guarantees whatsoever.

 

[Derek Zeanah]

I'll wait for another reply before running that. Not that I don't trust you, but the "no guarantees" line strikes me as ominous.

Not sure about XFES and the newer versions of Elasticsearch. It's been working fine until the power outage.

 

[Derek Zeanah]

Well, I tried that script. Got "an unexpected error occurred, please try again later."

The index was successfully deleted by the script though:



Now, I'm about a third of the way into rebuilding the index and I'm getting search results. So fingers crossed. I'll report back in a bit to see if the issue is resolved.

 

[Derek Zeanah]

Yep. This issue is now resolved. Something in Kirby's script did it.

TYVM all.

 

[Xon]

In the future;

• stop elasticsearch
• removing /var/lib/eleasticsearch/data (path may vary)
• start elasticsearch

I've had some weird edge cases with older version where restarting from scratch was the only way to get elasticsearch operation again.

 

[Kirby]

Well, I tried that script. Got "an unexpected error occurred, please try again later."

Weird. What was the last message that was printed on the console before that error?
You did execute it on the console and not via browser, didn't you?

 

[Derek Zeanah]

Weird. What was the last message that was printed on the console before that error?
You did execute it on the console and not via browser, didn't you?

I ran this over SSH; the error I mentioned is one I saw that grabbed my attention, so I maximized the window on my monitor, and when it reformatted the text the error wasn't there; instead I saw this:

# php rebuild-es.php Recreating ElasthcSearch Index ... Done
Your guess is as good as mine, but that wording is really similar to the errors Xenforo was giving users when trying to execute a search.