Unable to run initial indexing - Elasticsearch indexing error: Uknown Error (HTTP code: 413)

[defender110]

Hi,

I'm running XFES 2.1.0 and attempting to perform the initial indexing. Every time I run it, it gets to around post 922868 and errors out. It's very consistent in making it nearly all the way to the end (my forum has 1,073,873 posts) before dying. Before re-running the indexing, I delete the index on the Elasticsearch side and try again.

Rebuilding... Search index (Post 922868)

In Elasticsearch.php line 87:
                                                                                                                   
  An error was triggered while indexing. See the <a href="/admin.php?logs/server-errors/">XenForo error log</a> for details.
                                                                                                              
xf-rebuild:search [--log-queries LOG-QUERIES] [-b|--batch BATCH] [--type TYPE] [--truncate]

The stack trace doesn't seem very helpful:

#0 src/addons/XFES/Elasticsearch/Api.php(425): XFES\Elasticsearch\Api->request('post', '_bulk', '{"index":{"_ind...', Array, Object(GuzzleHttp\Psr7\Request))
#1 src/addons/XFES/Elasticsearch/Api.php(180): XFES\Elasticsearch\Api->bulkRequest('{"index":{"_ind...')
#2 src/addons/XFES/Search/Source/Elasticsearch.php(82): XFES\Elasticsearch\Api->indexBulk(Array)
#3 src/addons/XFES/Search/Source/Elasticsearch.php(57): XFES\Search\Source\Elasticsearch->flushBulkIndexing()
#4 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#5 src/XF/Search/Search.php(59): XF\Search\Search->index('post', Object(XF\Entity\Post))
#6 src/XF/Search/Search.php(85): XF\Search\Search->indexEntities('post', Object(XF\Mvc\Entity\ArrayCollection))
#7 src/XF/Job/SearchRebuild.php(57): XF\Search\Search->indexRange('post', 922868, '1000')
#8 src/XF/Job/Manager.php(253): XF\Job\SearchRebuild->run(8)
#9 src/XF/Job/Manager.php(195): XF\Job\Manager->runJobInternal(Array, 8)
#10 src/XF/Job/Manager.php(146): XF\Job\Manager->runJobEntry(Array, 8)
#11 src/XF/Cli/Command/JobRunnerTrait.php(30): XF\Job\Manager->runUnique('xfRebuildJob-se...', 8)
#12 src/XF/Cli/Command/JobRunnerTrait.php(13): XF\Cli\Command\Rebuild\AbstractRebuildCommand->runJob('xfRebuildJob-se...', Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 src/XF/Cli/Command/Rebuild/AbstractRebuildCommand.php(82): XF\Cli\Command\Rebuild\AbstractRebuildCommand->setupAndRunJob('xfRebuildJob-se...', 'XF:SearchRebuil...', Array, Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 src/vendor/symfony/console/Command/Command.php(255): XF\Cli\Command\Rebuild\AbstractRebuildCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 src/vendor/symfony/console/Application.php(953): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 src/vendor/symfony/console/Application.php(248): Symfony\Component\Console\Application->doRunCommand(Object(XF\Cli\Command\Rebuild\RebuildSearch), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 src/vendor/symfony/console/Application.php(148): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 src/XF/Cli/Runner.php(63): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 cmd.php(15): XF\Cli\Runner->run()
#20 {main}

My platform is as follows:

PHP 7.3.1
Elasticsearch 6.6 (10 GB max heap)
nginx reverse proxy fronting ES (multi_accept is on, 12 worker processes, 1024 worker_connections).

I run a lot of ES clusters at my day job and I've never seen this behavior before so I'm inclined to think that this is on the XFES side.

Any ideas?

 

[Mike]

413 would indicate that it's getting a content too large error, which I believe is controlled by the http.max_content_length setting in Elasticsearch (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html)

According to the ES docs, this defaults to 100MB, which I'm sure we're not exceeding. However, you mention Nginx fronting ES, and that would be covered by client_max_body_size I believe, which defaults to 1MB. With a large batch size, I think that could possibly be what's being hit, so increasing that would likely resolve the issue. Alternatively, I think dropping the batch size could also work.

 

[defender110]

Bam! Bumping client_max_body_size to 10m fixed it! Thank you!