18

The first section of the official Kubernetes tutorial states that,

A Kubernetes cluster that handles production traffic should have a minimum of three nodes.

but gives no rationale for why three is preferred. Is three desirable over two in order to avoid a split-brain scenario, to merely allow for greater availability, or to cater to something specific to the internals of Kubernetes? I would have thought a split-brain scenario would only happen with multiple Kubernetes clusters (each having distinct masters) whereas a single cluster should be able to handle at least two nodes, each, perhaps, in their own availability-zone.

rjs
  • 769
  • 1
  • 9
  • 19
  • Not really sure how to modify this to put it 'on-topic' without drastically modifying the question (afaict). I am hoping to reword it to reflect the semantic differneces between `minikube` and non-`minikube` environments, as it seems that's where my misunderstanding is, but I feel like it might morph the original question too much e.g. _Production Kubernetes recommendation of three 'nodes', master nodes singular in minikube?_ or some such. – rjs Feb 18 '18 at 01:13
  • 1
    Thanks for asking this question RJS, I think you're spot on, we need to improve the docs. Maybe [Building High-Availability Clusters ](https://kubernetes.io/docs/admin/high-availability/building/) helps though it also doesn't provide a rationale for 3 … – Michael Hausenblas Feb 18 '18 at 06:33
  • Because of leader election. Two masters cannot decide who is a leader an will constantly give errors. Three is OK but more is better for redundancy. – Terence Bor Dec 11 '19 at 04:05

1 Answers1

10

That means a minimum of 3 master nodes per cluster.

Kubernetes keeps all critical data in etcd, which uses a majority to repair when there is a fault. An instance of etcd runs on each master node. Three is the minimum number of etcds one wants backing a prod-level cluster. Hence, three is the minimum number of masters per cluster.

Jonah Benton
  • 3,188
  • 1
  • 11
  • 26
  • 1
    The full quote and the associated figure from the link I referenced seem to imply that these nodes are not master nodes but nodes that communicate directly with the master node unless nodes in the context of that doc are both masters and those communicating directly with masters? – rjs Feb 17 '18 at 22:22
  • 2
    Hmm, yeah- that quote, diagram, and the other language on that page- which is largely, though not entirely, about minikube, a single node dev version of kubernetes- looks to be inaccurate in the context of production architecture and requirements. In production, with HA considerations, kube wants sufficient master nodes to provide a quorum in the face of tolerable failures, and does leader election for the individual components. Cluster tenant workloads usually run on minion nodes. This is a better overview: https://kubernetes.io/docs/admin/high-availability/building/ – Jonah Benton Feb 17 '18 at 22:37
  • 1
    It really seems, from the tutorial OP linked to, that each cluster should have *one* master, which manages any number of "slave" nodes. They refer to it regularly as "the master", and never use the plural "masters". It's awfully vague. – Craig Otis Feb 25 '18 at 17:23
  • 1
    Yes, agreed. In the page's defense, aiming for the right balance of conceptual clarity/simplicity and technical precision in tutorials is awfully hard. Perhaps the page should direct more experienced readers to Kelsey Hightower's Kubernetes the Hard Way. – Jonah Benton Feb 25 '18 at 17:47