I've configured an elastic-search
cluster on kubernetes cluster GKE v1.17.
Cluster config:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elastic-cluster-1
spec:
version: 7.6.1
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
nodeSets:
- name: default
count: 3
config:
node.master: true
node.data: true
node.ingest: true
podTemplate:
metadata:
labels:
# additional labels for pods
type: elastic-master-node
spec:
initContainers:
# Increase linux map count to allow elastic to store large memory maps
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
# specify resource limits and requests
resources:
limits:
memory: 3.5Gi
requests:
cpu: 1500m
env:
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
# Request persistent data storage for pods
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: ssd
- name: data
count: 2
config:
node.master: false
node.data: true
node.ingest: true
podTemplate:
metadata:
labels:
# additional labels for pods
type: elastic-data-node
spec:
initContainers:
# Increase linux map count to allow elastic to store large memory maps
- name: sysctl
securityContext:
privileged: true
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
# specify resource limits and requests
resources:
limits:
memory: 3.5Gi
requests:
cpu: 1500m
env:
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
# Request persistent data storage for pods
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
storageClassName: ssd
# Google cloud storage credentials
secureSettings:
- secretName: "gcs-credentials"
http:
service:
spec:
# expose this cluster Service with a LoadBalancer
type: LoadBalancer
tls:
certificate:
secretName: elasticsearch-certificate
When updating the cluster, the operator shut each pod down and starting it back up. This is how it rolls updates for the cluster.
When this rolling update occurs I am seeing my index reaching red and yellow status.
My main index have 5 shards and 1 replica for each shard. My cluster have 5 pods(3 master and 2 data).
I have validated that elasticsearch
distribute the shards and replicas evenly and maintaining that shard and its replica does not reach the same machine.(validated using GET _cat/shards/my-index)
Why does my index reach red and yellow state when rolling update?