Add non existing cluster settings in elasticsearch

Question

Recently i have built elasticsearch with fluentd and Kibana and all is working fine, but we are facing high cpu load while performing search caused by java.

I have 60 GB Ram and 16 processors and each processor has 2 cores.

ES_HEAP_SIZE=32g

-find below more details

curl "localhost:9200/_cat/thread_pool?v&h=search.rejected"

search.rejected 387

curl "localhost:9200/_cat/thread_pool?v&h=index.rejected"

index.rejected 0

snapshot from elasticsearch.log

[DEBUG][action.search.type ] [Hulk 2099] [logstash-2015.03.14][4], node[qxcAN3lURs65Lf1GMhB_qg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7c71025f] lastShard [true] org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@1d7c9f0f
    at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)

    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)

    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)

    at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)

    at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:228)

    at org.elasticsearch.action.search.type.TransportSearchCountAction$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:71)

    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)

Now i read that i have to update the threadpool settings in ES and i have now

 curl -XGET localhost:9200/_cluster/settings?pretty

{ "persistent" : { }, "transient" : { } }

i'm trying to update the settings with the below commands

curl -XPUT localhost:9200/_cluster/settings -d '{ 

"threadpool" : {    "index": { 
    "type": "fixed", 
    "size": 32, 
    "queue_size": 1000    },    "bulk": { 
    "type": "fixed", 
    "size": 32, 
    "queue_size": 1000    },    "search": { 
    "type": "fixed", 
    "size": 96, 
    "queue_size": 1000   }  }  }'

but i keep getting

{"error":"ActionRequestValidationException[Validation Failed: 1: no settings to update;]","status":400}

What's wrong with this command? is it the right solution for my issue ?

Please advise

Thanks.

Ayman

score 0 · Answer 1 · answered Mar 28 '15 at 21:55

0

You need to specify if the updates are permanent (applied across all restarts) or transient (will not survive a full cluster restart). Find more information here.

If you want to apply your settings transiently, use the command below:

curl -XPUT localhost:9200/_cluster/settings -d '{ 
    "transient": {
        "threadpool": {
            "index": {
                "type": "fixed",
                "size": 32,
                "queue_size": 1000
            },
            "bulk": {
                "type": "fixed",
                "size": 32,
                "queue_size": 1000
            },
            "search": {
                "type": "fixed",
                "size": 96,
                "queue_size": 1000
            }
        }
    }
}

To apply them permanently, replace transient with permanent. Note that you can also set individual settings transiently or permanently by grouping them accordingly in the cluster update command.

answered Mar 28 '15 at 21:55

bittusarkar

5,827
3
24
49

Should i take backup of my cluster before i apply? can i revert them back? @bsarkar – Ayman Al-Shorman Mar 28 '15 at 22:12
Well, taking a backup always helps and yes you can change the settings again. – bittusarkar Mar 28 '15 at 22:13
will try that now and will let you know @bsarkar – Ayman Al-Shorman Mar 28 '15 at 22:15
Ok it works but unfortunately my problem still persist, do you recommend anything to do? i still see **search.rejected** is increasing and i don't get the result on Kibana @bsarkar – Ayman Al-Shorman Mar 28 '15 at 22:40

score 0 · Answer 2 · answered Mar 04 '19 at 17:41

Here's an example of getting, setting, and clearing transient cluster settings...

GET /_cluster/settings?include_defaults=true

GET /_cluster/settings

PUT /_cluster/settings
{
  "transient":{ "cluster.routing.allocation.cluster_concurrent_rebalance":0 }
}

PUT /_cluster/settings
{
  // set to null to clear
  "transient":{ "cluster.routing.allocation.cluster_concurrent_rebalance":null }
}

These reference are worth a look:

Add non existing cluster settings in elasticsearch

2 Answers2

Linked