Server issue Enhanced Search Crashing Elasticsearch

[MattW]

Affected version

2.3

I’m currently away, so haven’t looked through logs yet, but each time someone is upgrading to XF2.3, the search index rebuild is crashing elasticsearch. Seen it on ES8.14 and 8.15

 

[MattW]

Just tested on my own development site.

If you create a new index, it crashes Elasticsearch:

[2024-08-17T16:05:35,095][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [host.mattw.services] fatal error in thread [elasticsearch[host.mattw.services][masterService#updateTask][T#2300]], exiting
java.util.ServiceConfigurationError: Error loading SPI class list from URL: jar:file:///usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.14.3.jar!/META-INF/services/org.elasticsearch.index.SlowLogFieldProvider
        at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:136) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.plugins.spi.SPIClassIterator.hasNext(SPIClassIterator.java:148) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.plugins.PluginsService.createExtensions(PluginsService.java:379) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.plugins.PluginsService.loadServiceProviders(PluginsService.java:334) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.indices.IndicesService.loadSlowLogFieldProvider(IndicesService.java:1399) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:742) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:699) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:471) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:594) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:421) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService$1.execute(MetadataCreateIndexService.java:300) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:568) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1059) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1022) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1675) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1672) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1267) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.action.ActionListener.run(ActionListener.java:356) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1246) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.14.3.jar:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.14.3.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.lang.Thread.run(Thread.java:1570) ~[?:?]
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/modules/x-pack-security/x-pack-security-8.14.3.jar
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
        at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56) ~[?:?]
        at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171) ~[?:?]
        at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
        at java.nio.file.Files.readAttributes(Files.java:1858) ~[?:?]
        at java.util.zip.ZipFile$Source.get(ZipFile.java:1491) ~[?:?]
        at java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:724) ~[?:?]
        at java.util.zip.ZipFile.<init>(ZipFile.java:251) ~[?:?]
        at java.util.zip.ZipFile.<init>(ZipFile.java:180) ~[?:?]
        at java.util.jar.JarFile.<init>(JarFile.java:345) ~[?:?]
        at sun.net.www.protocol.jar.URLJarFile.<init>(URLJarFile.java:100) ~[?:?]
        at sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:69) ~[?:?]
        at sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:168) ~[?:?]
        at sun.net.www.protocol.jar.JarFileFactory.getOrCreate(JarFileFactory.java:91) ~[?:?]
        at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:117) ~[?:?]
        at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:160) ~[?:?]
        at org.elasticsearch.plugins.spi.SPIClassIterator.loadNextProfile(SPIClassIterator.java:112) ~[elasticsearch-8.14.3.jar:?]
        ... 25 more

 

[MattW]

Hmm, tested again after updating to 8.15, and it's not crashing now on new index creation with my development site. Might just be related to 8.14

 

[Jeremy P]

I had used 8.14 since it came out without issue. Looking at the logs, it seems like an Elasticsearch bug or configuration issue.