elastisearch error is back - this time not going away

[kalvink]

I've had this before and always managed to clear it. this time it is not going away - Can someone please help - I've tried the usual tricks

Error

XFES\Elasticsearch\RequestException: Elasticsearch indexing error (queued): index [xf] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Slack trace

#0 src/addons/XFES/Elasticsearch/Api.php(331): XFES\Elasticsearch\Api->request('put', 'xf/_doc/...', '{
    "title": ...')
#1 src/addons/XFES/Elasticsearch/Api.php(349): XFES\Elasticsearch\Api->requestFromIndex('put', '_doc/thread-268...', Array)
#2 src/addons/XFES/Elasticsearch/Api.php(152): XFES\Elasticsearch\Api->requestById('put', 'thread', 268679, Array)
#3 src/addons/XFES/Search/Source/Elasticsearch.php(54): XFES\Elasticsearch\Api->index('thread', 268679, Array)
#4 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#5 src/XF/Behavior/Indexable.php(46): XF\Search\Search->index('thread', Object(XFMG\XF\Entity\Thread), false)
#6 src/XF.php(341): XF\Behavior\Indexable->XF\Behavior\{closure}()
#7 src/XF/Mvc/Dispatcher.php(156): XF::triggerRunOnce(true)
#8 src/XF/Mvc/Dispatcher.php(55): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#9 src/XF/App.php(2300): XF\Mvc\Dispatcher->run()
#10 src/XF.php(488): XF\App->run()
#11 index.php(20): XF::runApp('XF\\Pub\\App')
#12 {main}

server environment

PHP version    7.4.13
MySQL version    5.5.68 (5.5.68-MariaDB)
PHP memory_limit    512M
PHP post_max_size    128M
PHP upload_max_filesize    128M
PHP max_input_vars    1000
PHP max_execution_time    1800
cURL version    7.29.0
cURL SSL version    NSS/3.53.1
OpenSSL version    OpenSSL 1.0.2k 26 Jan 2017
Suhosin enabled    No
Imagick support    Yes
EXIF support    Yes
GZip support    Yes
mbstring support    Yes
gmp support    Yes
ZipArchive support    Yes

Xenforo V2.2.2

 

[kalvink]

Have tried:

checked service status
rebooted server
delete old index
create a new index name
Run:

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": false}'

trying to rebuild the search through cli

/opt/plesk/php/7.4/bin/php cmd.php xf-rebuild:search

 

[AndyB]

When Elasticsearch detects a node's disk is nearing being full, it sets any index that the node holds to a read only state to protect the data that it has in these indices.

https://discuss.elastic.co/t/why-am...allow-delete-response-against-my-index/244344

 

[kalvink]

alright, so this is a space allocation issue - interestingly, I have tons of space available.

 

[kalvink]

I just tried my above steps again, this time it has worked - very odd!

 

[kalvink]

My error is back, can anyone help me fix this please?

 

[kalvink]

going to try the below comands out next

curl -XPUT 'localhost:9200/_all/_settings?pretty' -H 'Content-Type: application/json' -d '{"index": {"blocks": {"read_only_allow_delete": "false"}}}'
curl -XPUT 'localhost:9200/_settings?pretty' -H 'Content-Type: application/json' -d '{"index": {"blocks": {"read_only_allow_delete": "false"}}}'

 

[kalvink]

here is the tail of /var/loig/elasticsearch/elasticsearch.log

looks like I got to increase the space allocation for elasticsearch as hinted by @AndyB

[root@forum- forums.forum.com]# tail /var/log/elasticsearch/elasticsearch.log
[2020-12-07T19:44:17,826][INFO ][o.e.c.m.MetaDataMappingService] [forum-.nh-serv.com] [elasti/XQnHNTy4TkCIti_VMIc71Q] update_mapping [_doc]
[2020-12-07T19:44:18,962][INFO ][o.e.c.m.MetaDataMappingService] [forum-.nh-serv.com] [elasti/XQnHNTy4TkCIti_VMIc71Q] update_mapping [_doc]
[2020-12-07T19:45:02,215][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.3gb[5.4%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:46:32,283][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] flood stage disk watermark [95%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3gb[4.9%], all indices on this node will be marked read-only
[2020-12-07T19:48:02,313][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.2gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:49:02,317][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.2gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:50:32,321][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.2gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:52:02,341][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.7gb[6.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:53:02,346][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.7gb[6.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:54:32,351][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.7gb[6.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[root@forum- forums.forum.com]#

 

[kalvink]

i've written to my hosts, they are super friendly and will help with any volume sizes - I will report back (I've also done some house keeping to free up space)

 

[kalvink]

ok this is fixed - my provider who is very helpful managed to free up some space -so yes this was a space issue.