elastisearch error is back - this time not going away

kalvink

Member
I've had this before and always managed to clear it. this time it is not going away - Can someone please help - I've tried the usual tricks :(

Error

Code:
XFES\Elasticsearch\RequestException: Elasticsearch indexing error (queued): index [xf] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

Slack trace

Code:
#0 src/addons/XFES/Elasticsearch/Api.php(331): XFES\Elasticsearch\Api->request('put', 'xf/_doc/...', '{
    "title": ...')
#1 src/addons/XFES/Elasticsearch/Api.php(349): XFES\Elasticsearch\Api->requestFromIndex('put', '_doc/thread-268...', Array)
#2 src/addons/XFES/Elasticsearch/Api.php(152): XFES\Elasticsearch\Api->requestById('put', 'thread', 268679, Array)
#3 src/addons/XFES/Search/Source/Elasticsearch.php(54): XFES\Elasticsearch\Api->index('thread', 268679, Array)
#4 src/XF/Search/Search.php(40): XFES\Search\Source\Elasticsearch->index(Object(XF\Search\IndexRecord))
#5 src/XF/Behavior/Indexable.php(46): XF\Search\Search->index('thread', Object(XFMG\XF\Entity\Thread), false)
#6 src/XF.php(341): XF\Behavior\Indexable->XF\Behavior\{closure}()
#7 src/XF/Mvc/Dispatcher.php(156): XF::triggerRunOnce(true)
#8 src/XF/Mvc/Dispatcher.php(55): XF\Mvc\Dispatcher->dispatchLoop(Object(XF\Mvc\RouteMatch))
#9 src/XF/App.php(2300): XF\Mvc\Dispatcher->run()
#10 src/XF.php(488): XF\App->run()
#11 index.php(20): XF::runApp('XF\\Pub\\App')
#12 {main}

server environment

Code:
PHP version    7.4.13
MySQL version    5.5.68 (5.5.68-MariaDB)
PHP memory_limit    512M
PHP post_max_size    128M
PHP upload_max_filesize    128M
PHP max_input_vars    1000
PHP max_execution_time    1800
cURL version    7.29.0
cURL SSL version    NSS/3.53.1
OpenSSL version    OpenSSL 1.0.2k 26 Jan 2017
Suhosin enabled    No
Imagick support    Yes
EXIF support    Yes
GZip support    Yes
mbstring support    Yes
gmp support    Yes
ZipArchive support    Yes

Xenforo V2.2.2
 

kalvink

Member
Have tried:

checked service status
rebooted server
delete old index
create a new index name
Run:
Code:
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": false}'
trying to rebuild the search through cli
Code:
/opt/plesk/php/7.4/bin/php cmd.php xf-rebuild:search
 

kalvink

Member
going to try the below comands out next

Code:
curl -XPUT 'localhost:9200/_all/_settings?pretty' -H 'Content-Type: application/json' -d '{"index": {"blocks": {"read_only_allow_delete": "false"}}}'
curl -XPUT 'localhost:9200/_settings?pretty' -H 'Content-Type: application/json' -d '{"index": {"blocks": {"read_only_allow_delete": "false"}}}'
 

kalvink

Member
here is the tail of /var/loig/elasticsearch/elasticsearch.log

looks like I got to increase the space allocation for elasticsearch as hinted by @AndyB

Code:
[root@forum- forums.forum.com]# tail /var/log/elasticsearch/elasticsearch.log
[2020-12-07T19:44:17,826][INFO ][o.e.c.m.MetaDataMappingService] [forum-.nh-serv.com] [elasti/XQnHNTy4TkCIti_VMIc71Q] update_mapping [_doc]
[2020-12-07T19:44:18,962][INFO ][o.e.c.m.MetaDataMappingService] [forum-.nh-serv.com] [elasti/XQnHNTy4TkCIti_VMIc71Q] update_mapping [_doc]
[2020-12-07T19:45:02,215][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.3gb[5.4%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:46:32,283][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] flood stage disk watermark [95%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3gb[4.9%], all indices on this node will be marked read-only
[2020-12-07T19:48:02,313][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.2gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:49:02,317][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.2gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:50:32,321][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.2gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:52:02,341][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.7gb[6.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:53:02,346][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.7gb[6.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2020-12-07T19:54:32,351][WARN ][o.e.c.r.a.DiskThresholdMonitor] [forum-.nh-serv.com] high disk watermark [90%] exceeded on [rGV0GZQvQRa2o1ptLcHjjw][forum-.nh-serv.com][/var/lib/elasticsearch/nodes/0] free: 3.7gb[6.1%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[root@forum- forums.forum.com]#
 

kalvink

Member
i've written to my hosts, they are super friendly and will help with any volume sizes - I will report back (I've also done some house keeping to free up space)
 
Top