Elasticsearch indexing error: Elasticsearch bulk action error first error: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

[joroom]

I am getting this error after upgrading to 2.1

     XFES\Elasticsearch\BulkRequestException: Elasticsearch indexing error: Elasticsearch bulk action error (first error: [page-159] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];) src/addons/XFES/Elasticsearch/Api.php:430

Stack trace

#0 src/addons/XFES/Elasticsearch/Api.php(180): XFES\Elasticsearch\Api->bulkRequest('{"index":{"_ind...')
#1 src/addons/XFES/Search/Source/Elasticsearch.php(82): XFES\Elasticsearch\Api->indexBulk(Array)
#2 src/XF/Search/Source/AbstractSource.php(36): XFES\Search\Source\Elasticsearch->flushBulkIndexing()
#3 src/XF/Search/Search.php(98): XF\Search\Source\AbstractSource->disableBulkIndexing()
#4 src/XF/Search/Search.php(62): XF\Search\Search->disableBulkIndexing()
#5 src/XF/Search/Search.php(85): XF\Search\Search->indexEntities('page', Object(XF\Mvc\Entity\ArrayCollection))
#6 src/XF/Job/SearchRebuild.php(57): XF\Search\Search->indexRange('page', 0, '2000')
#7 src/XF/Job/Manager.php(253): XF\Job\SearchRebuild->run(G)
#8 src/XF/Job/Manager.php(195): XF\Job\Manager->runJobInternal(Array, G)
#9 src/XF/Job/Manager.php(79): XF\Job\Manager->runJobEntry(Array, G)
#10 src/XF/Admin/Controller/Tools.php(139): XF\Job\Manager->runQueue(true, 8)
#11 src/XF/Mvc/Dispatcher.php(321): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#12 src/XF/Mvc/Dispatcher.php(248): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', Object(XF\Mvc\RouteMatch), Object(XF\Admin\Controller\Tools), NULL)
#13 src/XF/Mvc/Dispatcher.php(100): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(XF\Admin\Controller\Tools), NULL)
#14 src/XF/Mvc/Dispatcher.php(50): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#15 src/XF/App.php(2177): XF\Mvc\Dispatcher->run()
#16 src/XF.php(390): XF\App->run()
#17 admin.php(13): XF::runApp('XF\\Admin\\App')
#18 {main}

Request State

array(4) {
  ["url"] => string(30) "/forum/admin.php?tools/run-job"
  ["referrer"] => string(51) "https://***/forum/admin.php?tools/run-job"
  ["_GET"] => array(1) {
    ["tools/run-job"] => string(0) ""
  }
  ["_POST"] => array(3) {
    ["_xfRedirect"] => string(16) "/forum/admin.php"
    ["_xfToken"] => string(8) "********"
    ["only_ids"] => string(0) ""
  }
}

I think this is causing the forum search to malfunction.

 

[joroom]

Heyy

Did anyone face this before?

 

[markus68]

Hi,
you ran out of space (less than 5% free) on the partition/filesystem where your ES index is stored.

Increase space and then switch back the index to write mode by CURLing into ES:

PUT your_index_name/_settings
    {
    "index": {
    "blocks": {
        "read_only_allow_delete": "false"
        }
    }
}

 

[webbouk]

Does anyone know how to switch back to write mode from 'read-only' on a Windows Server?

I've got the same problem, scoured the internet, installed Curl and tried the above to no avail (commands not recognised) and have reached the end of my tether with it.

I thought there might be a switch in the registry but cannot find anything there.

I've uninstalled ElasticSearch, removed all traces of it in the registry, reinstalled the latest version (7.1.1) into the root of the drive and still get the same read-only error as in the original post.

In addition, there is over 60GB of free space on the drive, so that can not still be an issue, plus I've increased the Elasticsearch parameters to use up to 4GB of memory compared to the default 1GB.

 

[webbouk]

just an update...

Despite trying just about everything known to man, and Google, I could not change the 'read_only_delete' index to 'false' in order to make it writable.
It had locked into 'true' and despite the increase in disc space would not alter back to 'false' automatically.

Then the obvious struck me right there inbetween the eyes - ignore the current Elasticsearch server as set by XF Enhanced Search, the default option based upon the database name, and reconfigure the settings giving it a new Index name.
This then creates a new index with the default setting 'read_only_delete:"false"

Rebuild the cache, creating the new index and voila, it works once again


Simple but effective

 

[Xon]

@webbouk you don't need to re-install ElasticSearch; just deleting where it stores application data would be been enough (not 100% sure where that is under Windows). I've had to-do that when I've really screwed up elasticsearch before.

 

[webbouk]

That was part of the problem, not being able to locate where the data was stored despite extensive searches of my server and no information was available on the internet.
I thought by uninstalling the previous version and reinstalling the latest it would have removed any app data, but it didn't. The read-only setting (true) remained applied to the index 'same name as forum database' and the same index was used when ElasticSearch was reinstalled and XF Exhanced Search was enabled via the default XFES settings.

Changing the name of the index by reconfiguring XFES created a new index which was then marked 'false' instead of 'true'

 

[gerryvz]

I had the same issue that just cropped up out of the blue, and your "fix" solved the issue. Thank you, webbouk (and Xon).

One question though -- on a Linux-based system, does anyone know where the old search index is, so that I can delete it?

    XFES\Elasticsearch\BulkRequestException: Elasticsearch indexing error: Elasticsearch bulk action error (first error: [thread-1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];) src/addons/XFES/Elasticsearch/Api.php:430

    Generated by: gerryvz Jun 14, 2019 at 1:59 PM

Stack trace

#0 src/addons/XFES/Elasticsearch/Api.php(180): XFES\Elasticsearch\Api->bulkRequest('{"index":{"_ind...')
#1 src/addons/XFES/Search/Source/Elasticsearch.php(82): XFES\Elasticsearch\Api->indexBulk(Array)
#2 src/addons/XFES/Search/Source/Elasticsearch.php(57): XFES\Search\Source\Elasticsearch->flushBulkIndexing()
#3 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#4 src/XF/Search/Search.php(59): XF\Search\Search->index('post', Object(vw\vw\XF\Entity\Post))
#5 src/XF/Search/Search.php(85): XF\Search\Search->indexEntities('post', Object(XF\Mvc\Entity\ArrayCollection))
#6 src/XF/Job/SearchRebuild.php(57): XF\Search\Search->indexRange('post', 0, '1000')
#7 src/XF/Job/Manager.php(253): XF\Job\SearchRebuild->run(G)
#8 src/XF/Job/Manager.php(195): XF\Job\Manager->runJobInternal(Array, G)
#9 src/XF/Job/Manager.php(111): XF\Job\Manager->runJobEntry(Array, G)
#10 src/XF/Admin/Controller/Tools.php(120): XF\Job\Manager->runByIds(Array, 8)
#11 src/XF/Mvc/Dispatcher.php(321): XF\Admin\Controller\Tools->actionRunJob(Object(XF\Mvc\ParameterBag))
#12 src/XF/Mvc/Dispatcher.php(244): XF\Mvc\Dispatcher->dispatchClass('XF:Tools', 'RunJob', Object(XF\Mvc\RouteMatch), Object(XF\Admin\Controller\Tools), NULL)
#13 src/XF/Mvc/Dispatcher.php(100): XF\Mvc\Dispatcher->dispatchFromMatch(Object(XF\Mvc\RouteMatch), Object(XF\Admin\Controller\Tools), NULL)
#14 src/XF/Mvc/Dispatcher.php(50): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#15 src/XF/App.php(2177): XF\Mvc\Dispatcher->run()
#16 src/XF.php(390): XF\App->run()
#17 admin.php(13): XF::runApp('XF\\Admin\\App')
#18 {main}

Request state

array(4) {
  ["url"] => string(31) "/forums/admin.php?tools/run-job"
  ["referrer"] => string(55) "https://www.500eboard.co/forums/admin.php?tools/run-job"
  ["_GET"] => array(1) {
    ["tools/run-job"] => string(0) ""
  }
  ["_POST"] => array(3) {
    ["_xfRedirect"] => string(65) "https://www.500eboard.co/forums/admin.php?tools/rebuild&success=1"
    ["_xfToken"] => string(8) "********"
    ["only_ids"] => string(4) "1942"
  }
}

For reference, my log is below:

 

[Xon]

Should be under /var/lib/elasticsearch or similar.

 

[webbouk]

There is an excellent add-on for viewing Elastic Search information and enabling you to easily delete indexes ....

Elasticsearch information

Description: Displays Elasticsearch information. This add-on is very useful for deleting no longer used Elasticsearch indexes. Note: this add-on only works on locally installed instances of Elasticsearch. (Example of Elasticsearch information...

www.xf2addons.com

 

[Wesker]

you ran out of space (less than 5% free) on the partition/filesystem where your ES index is stored.

This is exactly right. You have to clear your index logs. For me I just clear the httpd logs and rebuild the httpd folder via SSH.