Cannot rebuild post cache

[duderuud]

Hello all,

I'm testing out a conversion of our forum, just bought ES because we run a big board.

Installation works, rebuilding of profileposts and topics works great but the rebuilding of posts keeps crashing.

A bit of info:

XenForo_Exception: Elasticsearch indexing failed for post- Elasticsearch error: post:252001 - IndexFailedEngineException[[xf][2] Index failed for [post#252001]]; nested: FileNotFoundException[/usr/local/share/elasticsearch/data/elasticsearch/nodes/0/indices/xf/2/index/_b1j.fdt (Too many open files)]; - library/XenES/Search/SourceHandler/ElasticSearch.php:711

#0 /local/web/xf.forum.nl/html/library/XenES/Search/SourceHandler/ElasticSearch.php(748): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(false, true, 'Elasticsearch i...')
#1 /local/web/xf.forum.nl/html/library/XenES/Search/SourceHandler/ElasticSearch.php(67): XenES_Search_SourceHandler_ElasticSearch->_assertIndexSuccessful(false, 'post')
#2 /local/web/xf.forum.nl/html/library/XenForo/Search/Indexer.php(125): XenES_Search_SourceHandler_ElasticSearch->finalizeRebuildSet()
#3 /local/web/xf.forum.nl/html/library/XenForo/CacheRebuilder/SearchIndex.php(93): XenForo_Search_Indexer->finalizeRebuildSet()
#4 /local/web/xf.forum.nl/html/library/XenForo/ControllerHelper/CacheRebuild.php(26): XenForo_CacheRebuilder_SearchIndex->rebuild(1, Array, NULL)
#5 /local/web/xf.forum.nl/html/library/XenForo/ControllerAdmin/Tools.php(78): XenForo_ControllerHelper_CacheRebuild->rebuildCache(Array, 'http://xf...', 'admin.php?tools...', true)
#6 /local/web/xf.forum.nl/html/library/XenForo/FrontController.php(310): XenForo_ControllerAdmin_Tools->actionCacheRebuild()
#7 /local/web/xf.forum.nl/html/library/XenForo/FrontController.php(132): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#8 /local/web/xf.forum.nl/html/admin.php(13): XenForo_FrontController->run()
#9 {main}

ES logging:

[2012-04-27 22:08:38,578][WARN ][index.shard.service      ] [Ani-Mator] [xf][3] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException: [xf][3] Refresh failed
        at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:768)
        at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:440)
        at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:765)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /usr/local/share/elasticsearch/data/elasticsearch/nodes/0/indices/xf/3/index/_ay6.fdx (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:70)
        at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:97)
        at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:92)
        at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
        at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:458)
        at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:133)
        at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212)
        at org.apache.lucene.index.SegmentReader.openDocStores(SegmentReader.java:137)
        at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:714)
        at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:663)
        at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:157)
        at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:38)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453)

This looks a lot like this problem but I can't figure it out...delete a shard?

Edit: After these crashes ES seems to run (a curl http://127.0.0.1:9200 gives a return) but it really doesn't, I cannot rebuild again. I have to restart ES before I can try to rebuild again.

 

[Slavik]

EAm mobile so cant grab the links. Check the es bug reports forum for my post on upping the file limit. To delete the shard remove all files from the es data directory.

 

[duderuud]

Thx for the info. I'm seeing your post about installing ES, you recommend 1gb per milion posts. That's gonna be a problem with 22mln posts

 

[Slavik]

Thx for the info. I'm seeing your post about installing ES, you recommend 1gb per milion posts. That's gonna be a problem with 22mln posts

Check out IGN's custom mapping, that would reduce the requirement by approximately 30%.

 

[duderuud]

Thx, will try it out...