How to create an Elasticsearch cluster using Amazon EC2 computers?

Question

I have two computers running in Amazon EC2. I want to setup an Elasticsearch cluster between them. I have installed the EC2 Discovery Plugin and also I have included Zen Discovery configuration in their YMLs. Despite these, the cluster is not being formed. Both are starting as independent nodes.

My network and discovery settings are:

network.host: 0.0.0.0
http.port: 8200
discovery.zen.ping.unicast.hosts: ["10.0.1.2", "10.0.3.4"]
cloud.aws.access_key: "abc"
cloud.aws.secret_key: "xyz"

What other steps should I take to start the cluster?

Can you show the network configuration of both of your nodes? — Val, Jan 19 '17 at 12:34
I remember that you need to set on amazon machine level permissions for that. — Vova Bilyachat, Jan 19 '17 at 12:35
http://stackoverflow.com/questions/32978026/proper-access-policy-for-amazon-elastic-search-cluster — Vova Bilyachat, Jan 19 '17 at 12:37
@VolodymyrBilyachat I will try that and let you know. How do I see my EC2 access policy? — khateeb, Jan 19 '17 at 12:44
That's not what I meant. We need to see the `network.*` and `discovery.*` settings in your `elasticsearch.yml` files. — Val, Jan 19 '17 at 12:59

score 0 · Accepted Answer · answered Jan 19 '17 at 13:12

0

Your hosts are not seeing each other because your network.host setting doesn't use the correct IP address:

On host1 you need this config:

network.host: 10.0.1.2
http.port: 8200
discovery.zen.ping.unicast.hosts: ["10.0.3.4"]
cloud.aws.access_key: "abc"
cloud.aws.secret_key: "xyz"

On host2 you need this config:

network.host: 10.0.3.4
http.port: 8200
discovery.zen.ping.unicast.hosts: ["10.0.1.2"]
cloud.aws.access_key: "abc"
cloud.aws.secret_key: "xyz"

To simplify you can also have both hosts in the discovery.zen.ping.unicast.hosts, that will simplify automated deployments, but the above should work.

answered Jan 19 '17 at 13:12

Val

165,097
10
260
279

I shall do this tomorrow and let you know. Will this allow me to run curl commands to these servers from outside the EC2? – khateeb Jan 19 '17 at 13:20
10.* are private IP addresses, so unless you have an SSH tunnel from your local host to a host inside your EC2 environment, no you won't be able to. But it's a [good idea](https://www.elastic.co/blog/protecting-against-attacks-that-hold-your-data-for-ransom) to bind to a private IP address in order to prevent outside attackers from reaching your cluster. – Val Jan 19 '17 at 13:22
I have a DNS names for them. Can I use them in `network.host` and `discovery.zen.ping.unicast.hosts`? – khateeb Jan 19 '17 at 13:23
Yes, you can use them. – Val Jan 19 '17 at 13:24
You might also need to open the internal ports (probably 9300, unless you changed that) to each other with security groups. You should also probably set `discovery.zen.minimum_master_nodes: 2` so you don't get a split brain. – James R Jan 19 '17 at 19:46
@Val I'm getting a `BindTransportException[Failed to bind to [9300-9400]]; nested: BindException[Cannot assign requested address]`. I have opened ports from 9000-9500. – khateeb Jan 20 '17 at 12:42
@Val I had used my hostname as `network.host`. When I changed it back to `0.0.0.0` or commented it, it started normally. But then the cluster was not formed. – khateeb Jan 20 '17 at 13:22

How to create an Elasticsearch cluster using Amazon EC2 computers?

1 Answers1