0

I've configured an elastic-search cluster on kubernetes cluster GKE v1.17.

Cluster config:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-cluster-1
spec:
  version: 7.6.1
  image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
  nodeSets:
  - name: default
    count: 3
    config:
      node.master: true
      node.data: true
      node.ingest: true
    podTemplate:
      metadata:
        labels:
          # additional labels for pods
          type: elastic-master-node
      spec:
        initContainers:
        # Increase linux map count to allow elastic to store large memory maps
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            limits:
              memory: 3.5Gi
            requests:
              cpu: 1500m
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms2g -Xmx2g"
    # Request persistent data storage for pods
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: ssd
  - name: data
    count: 2
    config:
      node.master: false
      node.data: true
      node.ingest: true
    podTemplate:
      metadata:
        labels:
          # additional labels for pods
          type: elastic-data-node
      spec:
        initContainers:
        # Increase linux map count to allow elastic to store large memory maps
        - name: sysctl
          securityContext:
            privileged: true
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        containers:
        - name: elasticsearch
          # specify resource limits and requests
          resources:
            limits:
              memory: 3.5Gi
            requests:
              cpu: 1500m
          env:
          - name: ES_JAVA_OPTS
            value: "-Xms2g -Xmx2g"
    # Request persistent data storage for pods
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: ssd
  # Google cloud storage credentials
  secureSettings:
  - secretName: "gcs-credentials"
  http:
    service:
      spec:
        # expose this cluster Service with a LoadBalancer
        type: LoadBalancer
    tls:
      certificate:
        secretName: elasticsearch-certificate

When updating the cluster, the operator shut each pod down and starting it back up. This is how it rolls updates for the cluster.

When this rolling update occurs I am seeing my index reaching red and yellow status.

My main index have 5 shards and 1 replica for each shard. My cluster have 5 pods(3 master and 2 data).

I have validated that elasticsearch distribute the shards and replicas evenly and maintaining that shard and its replica does not reach the same machine.(validated using GET _cat/shards/my-index)

Why does my index reach red and yellow state when rolling update?

Montoya
  • 2,010
  • 2
  • 21
  • 44
  • _"5 shards and 1 replica for each shard"_ Could it be that since all shards are uniformly distributed in the cluster so, if one node goes down, the index becomes unavailable. Did you try to use more than 1 replica per shard? – ashu Dec 14 '20 at 16:16
  • Didn't tried more than one replica. I don't understand why if one node goes down the index become unavailable. There is a replica in another node and elastic search should in that case be able to use the replica as a shard until the node goes back up. – Montoya Dec 15 '20 at 07:45
  • Shards != replicas. Shards are pieces that compose your index. If one of the shards becomes unavailable, your index will become unavailable. Having more replicas of each shard should resolve your issue; if one replica of a shard goes down, the index is still able to access another replica of the same shard. – ashu Dec 15 '20 at 08:20
  • But when a node goes down with a shard in it we already have a replica in another node. What you are saying implies that you need at least 2 replicas for a shard in order to mitigate its downtime, but this seems incorrect to me. Why can't the cluster use the data for this shard available in its replica? – Montoya Dec 15 '20 at 08:35
  • You're confusing shards and replicas. Maybe this can shed some light. https://stackoverflow.com/a/15705989/2410641 – ashu Dec 15 '20 at 09:29
  • This explanation solidify what I expected to happen. Lets say the node-0 that is going down holds shard-0 and replica-1. replica-0 is being held another node(lets say node-1) and thus we have a backup for this shard until it comes back up. Maybe we are not understanding each other here. If the replica-0 is available in another node it should be enough for the cluster to handle the downtime of shard-0 and the index should only move to yellow state. Its the same as the explanation you linked to. – Montoya Dec 15 '20 at 09:45
  • Yeah, seems to be communication gap. Sorry, when you sad 1 replica, I counted it as primary. My bad! – ashu Dec 15 '20 at 12:34

0 Answers0