Shards and replicas in Elasticsearch

Question

I am trying to understand what shard and replica is in Elasticsearch, but I didn't manage to understand it. If I download Elasticsearch and run the script, then from what I know I have started a cluster with a single node. Now this node (my PC) have 5 shards (?) and some replicas (?).

What are they, do I have 5 duplicates of the index? If so why? I could need some explanation.

Have a look here: http://stackoverflow.com/questions/12409438/when-do-you-start-additional-elasticsearch-nodes/12414123#12414123 — javanna, Mar 29 '13 at 08:21
I thought the answer you got and the linked answer above should clarify things. What's not clear then? — javanna, Mar 29 '13 at 08:55
I don't understan what a shard is and replicas. I don't get why there are many shards and replicas on one node. — LuckyLuke, Mar 29 '13 at 09:09
Every index can be split into shards to be able to distribute data. The shard is the atomic part of an index, which can be distributed over the cluster if you add more nodes. — javanna, Mar 29 '13 at 09:16
So does that mean that when I only have one node I only use one shard? When you say atomic part of an index, what do you mean? — LuckyLuke, Mar 29 '13 at 14:09
you can see a node as a machine in your cluster. In the cluster there can be multiple indexes. Every index has a certain number of shards, which are parts of an index. A node can of course hold more shards. With atomic I mean that's the part that gets distributed, eventually moved to another node depending on the shard allocation algorithm. Only an entire shard can be distributed over the cluster, not a part of it. If you have an index with a single shard, that shard can only be on a single node at a given time. — javanna, Mar 29 '13 at 14:13
So what is in a shard? Is it a duplicate of the index or what? And I still don't get the 5 shards on one node. You are answering me by using the word shard, but I don't get what it is except that it is part of an index. Why do we need them etc? — LuckyLuke, Mar 29 '13 at 14:26
I gave you a complete answer given your comments and doubts. Hope that makes things clear. — javanna, Mar 29 '13 at 15:13
the explanation on elastics site is not bad https://www.elastic.co/guide/en/elasticsearch/reference/current/_basic_concepts.html#getting-started-shards-and-replicas — Tomislav Mikulin, Jul 27 '18 at 13:10

score 1153 · Accepted Answer · edited Apr 08 '20 at 02:17

1153

I'll try to explain with a real example since the answer and replies you got don't seem to help you.

When you download elasticsearch and start it up, you create an elasticsearch node which tries to join an existing cluster if available or creates a new one. Let's say you created your own new cluster with a single node, the one that you just started up. We have no data, therefore we need to create an index.

When you create an index (an index is automatically created when you index the first document as well) you can define how many shards it will be composed of. If you don't specify a number it will have the default number of shards: 5 primaries. What does it mean?

It means that elasticsearch will create 5 primary shards that will contain your data:

 ____    ____    ____    ____    ____
| 1  |  | 2  |  | 3  |  | 4  |  | 5  |
|____|  |____|  |____|  |____|  |____|

Every time you index a document, elasticsearch will decide which primary shard is supposed to hold that document and will index it there. Primary shards are not a copy of the data, they are the data! Having multiple shards does help taking advantage of parallel processing on a single machine, but the whole point is that if we start another elasticsearch instance on the same cluster, the shards will be distributed in an even way over the cluster.

Node 1 will then hold for example only three shards:

 ____    ____    ____ 
| 1  |  | 2  |  | 3  |
|____|  |____|  |____|

Since the remaining two shards have been moved to the newly started node:

 ____    ____
| 4  |  | 5  |
|____|  |____|

Why does this happen? Because elasticsearch is a distributed search engine and this way you can make use of multiple nodes/machines to manage big amounts of data.

Every elasticsearch index is composed of at least one primary shard since that's where the data is stored. Every shard comes at a cost, though, therefore if you have a single node and no foreseeable growth, just stick with a single primary shard.

Another type of shard is a replica. The default is 1, meaning that every primary shard will be copied to another shard that will contain the same data. Replicas are used to increase search performance and for fail-over. A replica shard is never going to be allocated on the same node where the related primary is (it would pretty much be like putting a backup on the same disk as the original data).

Back to our example, with 1 replica we'll have the whole index on each node, since 2 replica shards will be allocated on the first node and they will contain exactly the same data as the primary shards on the second node:

 ____    ____    ____    ____    ____
| 1  |  | 2  |  | 3  |  | 4R |  | 5R |
|____|  |____|  |____|  |____|  |____|

Same for the second node, which will contain a copy of the primary shards on the first node:

 ____    ____    ____    ____    ____
| 1R |  | 2R |  | 3R |  | 4  |  | 5  |
|____|  |____|  |____|  |____|  |____|

With a setup like this, if a node goes down, you still have the whole index. The replica shards will automatically become primaries and the cluster will work properly despite the node failure, as follows:

 ____    ____    ____    ____    ____
| 1  |  | 2  |  | 3  |  | 4  |  | 5  |
|____|  |____|  |____|  |____|  |____|

Since you have "number_of_replicas":1, the replicas cannot be assigned anymore as they are never allocated on the same node where their primary is. That's why you'll have 5 unassigned shards, the replicas, and the cluster status will be YELLOW instead of GREEN. No data loss, but it could be better as some shards cannot be assigned.

As soon as the node that had left is backed up, it'll join the cluster again and the replicas will be assigned again. The existing shard on the second node can be loaded but they need to be synchronized with the other shards, as write operations most likely happened while the node was down. At the end of this operation, the cluster status will become GREEN.

Hope this clarifies things for you.

edited Apr 08 '20 at 02:17

Andy

3
3

answered Mar 29 '13 at 15:11

javanna

53,926
12
135
121

86

Awesome explanation, thanks for taking your time to put it together! :) – LuckyLuke Mar 29 '13 at 17:58
11

That is by far the best explanation of the shard/replica concept. Thanks a lot :) – Frank Förster May 06 '13 at 12:13
1

@javanna Great explanation, can talk a bit about multi clusters and how they work? – raffian Jul 02 '13 at 20:03
@Raffian Thanks! What do you mean exactly by multi clusters? You might want to ask a new question unless you think there's something missing regarding this specific question. – javanna Jul 03 '13 at 08:25
Won't using more shards increase search performance (latency, even if not throughput) on a multi-core machine? – Doron Yaacoby Sep 25 '13 at 07:30
@DoronYaacoby Yes it will – javanna Sep 26 '13 at 08:37
@javanna So your "just stick with a single primary shard" advice (for a single server) is not exactly accurate? – Doron Yaacoby Sep 26 '13 at 22:47
1

@DoronYaacoby there was an if in front of that sentence. That's the case if your data fits in a single shard and you don't plan to grow. But whether your docs fit or not depends on your data, your queries, your load, and what is acceptable in terms of performance. Have a look at this article too: http://blog.trifork.com/2013/09/26/maximum-shard-size-in-elasticsearch/ – javanna Sep 27 '13 at 07:59
3

May I suggest to explain further what would happen when the node that went down comes back up again? – c0dem4gnetic Nov 04 '13 at 18:31
"With a single node of course multiple shards don't make much sense" : thins one clarify some things, not so obvious for beginners, thank you for the answer. – Bax Nov 05 '13 at 08:32
@javanna - Nice explanation. One question - How many shards are required to create one index? Is it the value of the setting "number of shards"? – Andy Dufresne Nov 07 '13 at 12:49
@AndyDufresne one index is composed of at least one shard, but can have more. I thought that was clear from the answer. – javanna Nov 07 '13 at 13:25
Now, this is called Awesome. :) – Love Hasija Jan 02 '14 at 15:29
Wonderful explanation, saved my hours – Akash Kumar Apr 09 '14 at 17:55
@javanna what happens when the node1 receives a write to a primary shard (1) and goes down before it replicates to node2 replica (1R)? – tugberk Feb 14 '15 at 10:24
What should i do if the second node has been remove and another node join to cluster? It means that, I am in this situation and the replica in UNASSIGNED status. – biolinh Mar 30 '15 at 15:05
If we assign the number of replicas as 2. Then how are the copies of the shards kept in a node? – Animesh Pandey Jul 05 '15 at 20:27
1

Do you teach? Cos you totally should. You rock. But as @Animesh Pandey asked, I'm also curious to know what happens with two replicas, or 1 replica with 3 nodes. – frostymarvelous Jul 14 '15 at 21:01
Such a very great great answer! – AndrewMcLagan Feb 02 '17 at 21:58
This doesn't explain how the actual partitioning happens and what is stored in the shard. The index term is confusing because "supposedly" the index consists of Inverse Terms "table" that links terms to the documents and the actual documents. The question would be how those are distributed across multiple shards. When you say the document is routed to the shard, where does the term index is routed? If it is routed to the same shard, how does the search knows where to find each term? Or does the search always searches all shards? – Alex Pryiomka Oct 29 '17 at 18:05
@javanna Can you please look at https://stackoverflow.com/questions/57200358/issues-in-creating-and-deleting-elastic-search-index I am getting some issues, and I am not sure why is this happening? – Vivek Vardhan Jul 26 '19 at 10:10
1

There is an update for Elasticsearch v7 https://www.elastic.co/guide/en/elasticsearch/reference/current/release-highlights-7.0.0.html#_default_to_one_shard From this version there will be always one shard per index and possibility to change the amount of shards in settings – Yevhenii Herasymchuk Dec 20 '19 at 10:10
wonderful explanation! Now as you got best way to putting such answers together. I have another question for you. I have 4 nodes and created a index with 5 primary and 1 replica. Somehow my node4 have only replicas and no primary. How this happened? Second - when I request using _primary_first preference no request land on node4. Is this a expected behavior? – Manpreet Jun 25 '20 at 20:19
this explanation is amazing, I've never seen an explanation too clearly like this. thank you – fabulias Sep 13 '20 at 05:47

score 29 · Answer 2 · answered Mar 29 '13 at 05:34

29

An index is broken into shards in order to distribute them and scale.

Replicas are copies of the shards and provide reliability if a node is lost. There is often confusion in this number because replica count == 1 means the cluster must have the main and a replicated copy of the shard available to be in the green state.

In order for replicas to be created, you must have at least 2 nodes in your cluster.

You may find the definitions here easier to understand: http://www.elasticsearch.org/guide/reference/glossary/

Best Regards, Paul

answered Mar 29 '13 at 05:34

ppearcy

2,424
17
19

This is what every document says - the index is broken into shards, but what does the index actually contain? – Alex Pryiomka Oct 29 '17 at 18:06
@AlexPryiomka, Index contains data – Gaurav Jul 25 '18 at 17:49
So it is basically the same as partition-replication in the kafka world? – beatrice Apr 22 '20 at 10:11

score 24 · Answer 3 · answered Apr 18 '18 at 06:25

Shard:

Being distributed search server, ElasticSearch uses concept called Shard to distribute index documents across all nodes.
An index can potentially store a large amount of data that can exceed the hardware limits of a single node
For example, a single index of a billion documents taking up 1TB of disk space may not fit on the disk of a single node or may be too slow to serve search requests from a single node alone.
To solve this problem, Elasticsearch provides the ability to subdivide your index into multiple pieces called shards.
When you create an index, you can simply define the number of shards that you want.
Documents are stored in shards, and shards are allocated to nodes in your cluster
As your cluster grows or shrinks, Elasticsearch will automatically migrate shards between nodes so that the cluster remains balanced.
A shard can be either a primary shard or a replica shard.
Each document in your index belongs to a single primary shard, so the number of primary shards that you have determines the maximum amount of data that your index can hold
A replica shard is just a copy of a primary shard.

Replica:

Replica shard is the copy of primary Shard, to prevent data loss in case of hardware failure.
Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short.
An index can also be replicated zero (meaning no replicas) or more times.
The number of shards and replicas can be defined per index at the time the index is created.
After the index is created, you may change the number of replicas dynamically anytime but you cannot change the number of shards after-the-fact.
By default, each index in Elasticsearch is allocated 5 primary Shards and 1 replica which means that if you have at least two nodes in your cluster, your index will have 5 primary shards and another 5 replica shards (1 complete replica) for a total of 10 shards per index.

jyu · Answer 4 · 2015-06-05T19:36:22.907

21

If you really don't like to see it yellow. you can set the number of replicas to be zero:

curl -XPUT 'localhost:9200/_settings' -d '
{
    "index" : {
        "number_of_replicas" : 0
    }
}
'

Note that you should do this only on your local development box.

edited Jun 05 '15 at 19:36

answered Jan 14 '14 at 01:18

jyu

513
4
6

3

This is bad practice with a multinode cluster. Never a recommended option for 2 or more nodes. – Jinna Balu Oct 20 '20 at 16:10

score 9 · Answer 5 · answered Apr 01 '16 at 06:07

9

An index is broken into shards in order to distribute them and scale.

Replicas are copies of the shards.

A node is a running instance of elastic search which belongs to a cluster.

A cluster consists of one or more nodes which share the same cluster name. Each cluster has a single master node which is chosen automatically by the cluster and which can be replaced if the current master node fails.

answered Apr 01 '16 at 06:07

Pruthvi

189
2
5

I have three `AWS ec2` instance, each have elasticsearch installed on it. Means we have three nodes here? If all these nodes have the same `cluster.name: test` property set, will it make a Cluster name `test` which would have three nodes? – TheCoder Jul 20 '19 at 15:42

score 7 · Answer 6 · edited Feb 27 '20 at 06:56

I will explain this using a real word scenarios. Imagine you are a running a ecommerce website. As you become more popular more sellers and products add to your website. You will realize the number of products you might need to index has grown and it is too large to fit in one hard disk of one node. Even if it fits in to hard disk, performing a linear search through all the documents in one machine is extremely slow. one index on one node will not take advantage of the distributed cluster configuration on which the elasticsearch works.

So elasticsearch splits the documents in the index across multiple nodes in the cluster. Each and every split of the document is called a shard. Each node carrying a shard of a document will have only a subset of the document. suppose you have 100 products and 5 shards, each shard will have 20 products. This sharding of data is what makes low latency search possible in elasticsearch. search is conducted parallel on multiple nodes. Results are aggregated and returned. However the shards doesnot provide fault tolerance. Meaning if any node containing the shard is down, the cluster health becomes yellow. Meaning some of the data is not available.

To increase the fault tolerance replicas come in to picture. By deault elastic search creates a single replica of each shard. These replicas are always created on a other node where the primary shard is not residing. So to make the system fault tolerant, you might have to increase the number of nodes in your cluster and it also depends on number of shards of your index. The general formula to calculate the number of nodes required based on replicas and shards is "number of nodes = number of shards*(number of replicas + 1)".The standard practice is to have atleast one replica for fault tolerance.

Setting up the number of shards is a static operation, meaning you have to specify it when you are creating an index. Any change after that woulf require complete reindexing of data and will take time. But, setting up number of replicas is a dynamic operation and can be done at any time after index creation also.

you can setup the number of shards and replicas for your index with the below command.

curl -XPUT 'localhost:9200/sampleindex?pretty' -H 'Content-Type: application/json' -d '
{
  "settings":{
    "number_of_shards":2,
    "number_of_replicas":1
  }
}'

Ahmad · Answer 7 · 2020-11-26T10:42:09.720

In its simplest terms, the shard is nothing but a part of an index that stored on the disk within a separated folder:

This screenshot shows the entire Elasticsearch directory.

As you can see, all the data goes into the data directory.

By inspecting the index C-mAfLltQzuas72iMiIXNw we see that it has five shards (folders 0 to 4).

In other hand, the JH_A8PgCRj-GK0GeQ0limw index has only one shard (0 folder).

The pri shows the total number of shards.

score 6 · Answer 8 · edited Jun 20 '20 at 09:12

Not an answer but another reference for core concepts to ElasticSearch, and I think they are pretty clear as compliment to @javanna's answer.

Shards

An index can potentially store a large amount of data that can exceed the hardware limits of a single node. For example, a single index of a billion documents taking up 1TB of disk space may not fit on the disk of a single node or may be too slow to serve search requests from a single node alone.

To solve this problem, Elasticsearch provides the ability to subdivide your index into multiple pieces called shards. When you create an index, you can simply define the number of shards that you want. Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the cluster.

Sharding is important for two primary reasons:

It allows you to horizontally split/scale your content volume.

It allows you to distribute and parallelize operations across shards (potentially on multiple nodes) thus increasing performance/throughput.

Replicas

In a network/cloud environment where failures can be expected anytime, it is very useful and highly recommended to have a failover mechanism in case a shard/node somehow goes offline or disappears for whatever reason. To this end, Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short.

Replication is important for two primary reasons:

It provides high availability in case a shard/node fails. For this reason, it is important to note that a replica shard is never allocated on the same node as the original/primary shard that it was copied from.

It allows you to scale out your search volume/throughput since searches can be executed on all replicas in parallel.

score 2 · Answer 9 · answered Mar 23 '19 at 16:03

Elasticsearch is superbly scalable with all the credit goes to its distributed architecture. It is made possible due to Sharding. Now, before moving further into it, let us consider a simple and very common use case. Let us suppose, you have an index which contains a hell lot of documents, and for the sake of simplicity, consider that the size of that index is 1 TB (i.e, Sum of sizes of each and every document in that index is 1 TB). Also, assume that you have two Nodes each with 512 GB of space available for storing data. As can be seen clearly, our entire index cannot be stored in any of the two nodes available and hence we need to distribute our index among these Nodes.

In cases like this where the size of an index exceeds the hardware limits of a single node, Sharding comes to the rescue. Sharding solves this problem by dividing the indices into smaller pieces and these pieces are named as Shards.

score 1 · Answer 10 · edited May 23 '17 at 12:26

In ElasticSearch, at the top level we index the documents into indices. Each index has number of shards which internally distributes the data and inside shards exist the Lucene segments which is the core storage of the data. So if the index has 5 shards it means data has been distributed across the shards and not same data exist into the shards.

Watch out for the video which explains core of ES https://www.youtube.com/watch?v=PpX7J-G2PEo

Article on multiple indices or multiple shards Elastic search, multiple indexes vs one index and types for different data sets?

Shards and replicas in Elasticsearch

10 Answers10

Shards

Replicas

Linked