Recently i have built elasticsearch with fluentd and Kibana and all is working fine, but we are facing high cpu load while performing search caused by java.
I have 60 GB Ram and 16 processors and each processor has 2 cores.
ES_HEAP_SIZE=32g
-find below more details
curl "localhost:9200/_cat/thread_pool?v&h=search.rejected"
search.rejected 387
curl "localhost:9200/_cat/thread_pool?v&h=index.rejected"
index.rejected 0
- snapshot from elasticsearch.log
[DEBUG][action.search.type ] [Hulk 2099] [logstash-2015.03.14][4], node[qxcAN3lURs65Lf1GMhB_qg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7c71025f] lastShard [true] org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@1d7c9f0f
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551) at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:228) at org.elasticsearch.action.search.type.TransportSearchCountAction$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:71) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
Now i read that i have to update the threadpool settings in ES and i have now
curl -XGET localhost:9200/_cluster/settings?pretty
{ "persistent" : { }, "transient" : { } }
i'm trying to update the settings with the below commands
curl -XPUT localhost:9200/_cluster/settings -d '{
"threadpool" : { "index": {
"type": "fixed",
"size": 32,
"queue_size": 1000 }, "bulk": {
"type": "fixed",
"size": 32,
"queue_size": 1000 }, "search": {
"type": "fixed",
"size": 96,
"queue_size": 1000 } } }'
but i keep getting
{"error":"ActionRequestValidationException[Validation Failed: 1: no settings to update;]","status":400}
What's wrong with this command? is it the right solution for my issue ?
Please advise
Thanks.
Ayman