1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Cannot rebuild post cache

Discussion in 'Enhanced Search Support' started by duderuud, Apr 27, 2012.

  1. duderuud

    duderuud Active Member

    Hello all,

    I'm testing out a conversion of our forum, just bought ES because we run a big board.

    Installation works, rebuilding of profileposts and topics works great but the rebuilding of posts keeps crashing.

    A bit of info:

    Code:
    XenForo_Exception: Elasticsearch indexing failed for post- Elasticsearch error: post:252001 - IndexFailedEngineException[[xf][2] Index failed for [post#252001]]; nested: FileNotFoundException[/usr/local/share/elasticsearch/data/elasticsearch/nodes/0/indices/xf/2/index/_b1j.fdt (Too many open files)]; - library/XenES/Search/SourceHandler/ElasticSearch.php:711
    Code:
    #0 /local/web/xf.forum.nl/html/library/XenES/Search/SourceHandler/ElasticSearch.php(748): XenES_Search_SourceHandler_ElasticSearch->_logSearchResponseError(false, true, 'Elasticsearch i...')
    #1 /local/web/xf.forum.nl/html/library/XenES/Search/SourceHandler/ElasticSearch.php(67): XenES_Search_SourceHandler_ElasticSearch->_assertIndexSuccessful(false, 'post')
    #2 /local/web/xf.forum.nl/html/library/XenForo/Search/Indexer.php(125): XenES_Search_SourceHandler_ElasticSearch->finalizeRebuildSet()
    #3 /local/web/xf.forum.nl/html/library/XenForo/CacheRebuilder/SearchIndex.php(93): XenForo_Search_Indexer->finalizeRebuildSet()
    #4 /local/web/xf.forum.nl/html/library/XenForo/ControllerHelper/CacheRebuild.php(26): XenForo_CacheRebuilder_SearchIndex->rebuild(1, Array, NULL)
    #5 /local/web/xf.forum.nl/html/library/XenForo/ControllerAdmin/Tools.php(78): XenForo_ControllerHelper_CacheRebuild->rebuildCache(Array, 'http://xf...', 'admin.php?tools...', true)
    #6 /local/web/xf.forum.nl/html/library/XenForo/FrontController.php(310): XenForo_ControllerAdmin_Tools->actionCacheRebuild()
    #7 /local/web/xf.forum.nl/html/library/XenForo/FrontController.php(132): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
    #8 /local/web/xf.forum.nl/html/admin.php(13): XenForo_FrontController->run()
    #9 {main}
    ES logging:

    Code:
    [2012-04-27 22:08:38,578][WARN ][index.shard.service      ] [Ani-Mator] [xf][3] Failed to perform scheduled engine refresh
    org.elasticsearch.index.engine.RefreshFailedEngineException: [xf][3] Refresh failed
            at org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:768)
            at org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:440)
            at org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:765)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
            at java.lang.Thread.run(Thread.java:636)
    Caused by: java.io.FileNotFoundException: /usr/local/share/elasticsearch/data/elasticsearch/nodes/0/indices/xf/3/index/_ay6.fdx (Too many open files)
            at java.io.RandomAccessFile.open(Native Method)
            at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
            at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:70)
            at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:97)
            at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:92)
            at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
            at org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:458)
            at org.apache.lucene.index.FieldsReader.<init>(FieldsReader.java:133)
            at org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:212)
            at org.apache.lucene.index.SegmentReader.openDocStores(SegmentReader.java:137)
            at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:714)
            at org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:663)
            at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:157)
            at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:38)
            at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:453)


    This looks a lot like this problem but I can't figure it out...delete a shard?

    Edit: After these crashes ES seems to run (a curl http://127.0.0.1:9200 gives a return) but it really doesn't, I cannot rebuild again. I have to restart ES before I can try to rebuild again.
     
  2. Slavik

    Slavik XenForo Moderator Staff Member

    EAm mobile so cant grab the links. Check the es bug reports forum for my post on upping the file limit. To delete the shard remove all files from the es data directory.
     
  3. duderuud

    duderuud Active Member

    Thx for the info. I'm seeing your post about installing ES, you recommend 1gb per milion posts. That's gonna be a problem with 22mln posts :)
     
  4. Slavik

    Slavik XenForo Moderator Staff Member

    Check out IGN's custom mapping, that would reduce the requirement by approximately 30%.
     
  5. duderuud

    duderuud Active Member

    Thx, will try it out...
     

Share This Page