2

I'm trying to upgrade our ELK stack from 1.x > 5.x following the re-index from remote instructions. I'm not sure of how to export a list of the indices that I need to create and then import that list into the new instance. I've created a list of indices using this command, both with "pretty," and without, but I'm not sure which file format to use as well as what to next do with that file.

The create index instructions don't go into how to create more than one at a time, and the bulk instructions only refer to creating/indexing documents, not creating the indices themselves. Any assistance on how to best follow the upgrade instructions would be appreciated.

I apparently don't have enough reputation to link the "create index" and "bulk" instructions, so apologies for that.

Community
  • 1
  • 1
Travis
  • 165
  • 1
  • 1
  • 9
  • @JXG the shell solution from the below answer did not work for you? – Val Aug 08 '17 at 14:34
  • @Val "the same curl command": no. I mean, it's a good workaround, and it's the type of thing I used, but it's the opposite of what's asked for. – JXG Aug 08 '17 at 14:36
  • @JXG so you want a single curl command that will create N indices in one shot with the same index settings? – Val Aug 08 '17 at 14:38
  • @JXG Do your index names share a common pattern? If yes, one way would be to create an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) – Val Aug 08 '17 at 14:41

2 Answers2

2

With a single curl command you could create an index template that will trigger the index creation at the time the documents hit your ES 5.x cluster.

Basically, this single curl command will create an index template that will kick in for each new index created on-the-fly. You can then use the "reindex from remote" technique in order to move your documents from ES 1.x to ES 5.x and don't worry about index creation since the index template will take care of it.

curl -XPUT 'localhost:9200/_template/my_template' -H 'Content-Type: application/json' -d'
{
  "template": "*",
  "settings": {
     "index.refresh_interval" : -1,
     "index.number_of_replicas" : 0
  }
}
'
Val
  • 165,097
  • 10
  • 260
  • 279
  • This would make sense if I were dealing with hundreds of indices. If I have, say, 3, it seems like the overhead doesn't justify the template work. – JXG Aug 09 '17 at 06:11
  • what overhead? creating a template involves no overhead, it's just a way to streamline your index creation. – Val Aug 09 '17 at 06:16
  • if you only have 3 indices, I don't quite get why having to run a single curl command is that important. – Val Aug 09 '17 at 06:17
  • Well, this is in a test (in fact, it's 2 indices) so it runs often. Elastic has this elegant API for getting multiple indices, heck, even for deleting them. But there's no analogue for creation. It calls out for an explanation. Honestly, it's not that important, but it gave me some aggravation, and to be even more honest, nothing I ever do is all that important. – JXG Aug 09 '17 at 06:35
  • If it's in a test and runs often, it's even better to leverage index templates that you can create in your tear-up and delete in your tear-down processes. Getting and deleting indices don't require any payload, while creating indices does, that's probably why it is not supported to create many indices in one shot, because different indices might need different settings. – Val Aug 09 '17 at 06:57
  • So to sum it up, it is technically possible to pre-create several indices in one command using index templates (though the indices will only be created when the first document hits ES). But if you want to actually create several indices in a one-shot curl command, then it is not possible. As far as I'm concerned, this is all one and the same, the difference if truly semantic. – Val Aug 09 '17 at 11:28
  • @JXG do you have any more concerns regarding this? – Val Aug 10 '17 at 18:22
0

Was able to accomplish this with a formatted list of indices created via an index list fed through sed, then feeding that file through the following script:

#! /bin/bash

while read some_index; do
curl -XPUT "localhost:9200/$some_index?pretty" -d'
{
    "settings" : {
        "index" : {
            "refresh_interval" : -1,
            "number_of_replicas" : 0
        }
    }
}'
sleep 1
done <$1

If anyone can point me in the direction of any pre-existing mechanisms in Elasticsearch, though, please do.

Travis
  • 165
  • 1
  • 1
  • 9