Proper access policy for Amazon Elastic Search Cluster

Question

I've recently started using the new Amazon Elasticsearch Service and I can't seem to figure out the access policy I need so that I can only access the services from my EC2 instances that have a specific IAM role assigned to them.

Here's an example of the access policy I currently have assigned for the ES domain:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::[ACCOUNT_ID]:role/my_es_role",
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-east-1:[ACCOUNT_ID]:domain/[ES_DOMAIN]/*"
    }
  ]
}

But as I said, this doesn't work. I log into the EC2 instance (which has the my_es_role role attached to it) and attempt to run a simple curl call on the "https://*.es.amazonaws.com" end point, I get the following error:

{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet on resource: arn:aws:es:us-east-1:[ACCOUNT_ID]:domain/[ES_DOMAIN]/“}

Does anyone know what I have to change in the access policy in order for this to work?

Beware, ElasticSearch access policy changes take a long while to apply, unlike other IAM changes that are almost instantaneous. It's easy to just click "apply" and switch tab without noticing the "Processing..." — Cyril Duchon-Doris, May 09 '16 at 12:41

score 63 · Answer 1 · edited Aug 29 '17 at 15:52

You can lock access down to IAM-only, but how will you view Kibana in your browser? You could setup a proxy (see Gist and/or NPM module) or enable both IAM and IP-based access for viewing results.

I was able to get both IAM access IP-restricted access with the following Access Policy. Note the order is important: I could not get it working with the IP-based statement before the IAM statement.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-2:xxxxxxxxxxxx:domain/my-elasticsearch-domain/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "192.168.1.0",
            "192.168.1.1"
          ]
        }
      }
    }
  ]
}

My EC2 instance has an instance profile with the arn:aws:iam::aws:policy/AmazonESFullAccess policy. Logstash should sign requests using the logstash-output-amazon-es output plugin. Logstash running on my EC2 instance includes an output section like this:

output {
    amazon_es {
        hosts => ["ELASTICSEARCH_HOST"]
        region => "AWS_REGION"
    }
    # If you need to do some testing & debugging, uncomment this line:
    # stdout { codec => rubydebug }
}

I can access Kibana from the two IPs in the access policy (192.168.1.0 and 192.168.1.1).

Hi, you only need to use the plugin if you are using an IAM-based policy. You can use the standard elasticsearch plugin in Logstash if your access policy is based on IP addresses. You don't need an instance profile in that case either. Also, the ES service is not available in VPCs. You have to use public ip addresses to connect. Not sure if your references to 192.168 addresses are substitutions for something else, but may mislead. — Garreth McDaid, Jan 22 '16 at 16:41
The `aws:SourceIp`'s in my example are intended to be your personal workstation IP so you can use Kibana. The IAM-restricted access enables one or more EC2 instances to write to Elasticsearch without worrying about which IPs belong to a particular instance or CIDR block. — Pete, Jan 22 '16 at 20:15
It is worth noting that limiting to the _private_ IP CIDR range of your VPC does not seem to work. ES does not operate within the VPC or something. — sventechie, May 06 '16 at 20:36
Thanks for giving a sample policy in your answer; I'd been unable to get Kibana past the dreaded "User: anonymous" error until I switched `aws:SourceIp` from a scalar value to an array, as in the example you gave. (I'm CIDR notation, if that helps out anyone else.) The whole process of setting policies for AWS ES would be less frustrating if every single policy change didn't put the cluster into the mysterious "processing" state for 20 minutes as the policy is carefully inscribed on stone tablets, or whatever it is they are doing. — Robert Calhoun, May 17 '17 at 21:06

score 38 · Answer 2 · answered Oct 07 '15 at 09:43

38

According to AWS doc and as you (and I) just tested, you cannot restrict access to an AWS ES domain to a role/account/user/... and simply cURL it!

Standard clients, such as curl, cannot perform the request signing that is required of identity-based access policies. You must use an IP address-based access policy that allows anonymous access to successfully perform the instructions for this step. (http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg-search.html)

So you have basically two solutions:

change your access policy and restrict it to IP(s), I think you cannot use private IP because your ES cluster does not seems to belong to your VPC (default or not). Please use the public IP
sign your request: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html#es-managedomains-signing-service-requests

Signing your request is probably the best solution if you want to keep your access policy as is (which is more flexible than restricting to an IP), but it seems to be a bit more complex. I haven't tried so far and I cannot find any doc to help.

answered Oct 07 '15 at 09:43

Olivier

804
2
7
12

3

I used my laptop's public IP address and tried accessing the endpoint with curl/browser, but I'm still getting the User:anonymous error. – Anant Gupta Nov 29 '15 at 04:48
7

i am dealing with the same problem. and I have noticed that processing the changes by aws elasticsearch takes a loooong time. – nemo Nov 30 '15 at 22:37
Set an Access Policy with two statements: one for IAM access to write logs, the other with IP-restricted access to view KIbana. [See my answer for details](http://stackoverflow.com/a/34055389/40785) – Pete Dec 02 '15 at 23:58
2

I wondered if "loooong" meant minutes, hours, or days. Looks like its 10-15 minutes. You can see that when checking the status of your ES (green 'active' if update is complete, else, something like an orange 'preparing'. – Balmipour Jun 02 '16 at 15:01
I had the same problem and after searching I found this handy [library](https://pypi.python.org/pypi/requests-aws4auth). – gmajivu Oct 12 '16 at 12:14
Upvote for mention of private IPs not working because not in a VPC. – alph486 Oct 20 '16 at 20:51

score 6 · Answer 3 · answered Sep 28 '16 at 15:48

A bit late to the party, but I was able to deal with the exact same issue by adding signature to my requests.

If you use Python (like I do), you can use the following library to make it particularly easy to implement: https://github.com/DavidMuller/aws-requests-auth

It worked perfectly for me.

Oguz · Answer 4 · 2019-11-19T12:34:56.463

You just need to full user name in elastic search policy.

In this case, you can get your full user name from the error message itself. In my case: "arn:aws:sts::[ACCOUNT_ID]:assumed-role/[LAMBDA_POLICY_NAME]/[LAMBDA_NAME]"

    {
        "Version": "2012-10-17",
        "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": [
              "arn:aws:sts::xxxxxxxxxxxx:assumed-role/[lambda-role]/[full-lambda-name]"
            ]
          },
          "Action": "es:*",
          "Resource": "arn:aws:es:[region]:xxxxxxxxxxxxx:domain/[elasticsearch-domain-name]/*"
        }
      ]

    }

score 0 · Answer 5 · answered Apr 04 '17 at 12:26

You may either use resource based policy or identity based policy rather than IP based policy which is like hard coding the IP address.

But you need to use Signature version 4 to sign the request

For Java implementation please refer http://mytechbites.blogspot.in/2017/04/secure-amazon-elastic-search-service.html

Kunal · Answer 6 · 2020-02-07T10:33:00.417

0

Role ARN needs to be changed. it will be looks like "arn:aws:iam::[ACCOUNT_ID]:role/service-role/my_es_role"

edited Feb 07 '20 at 10:33

answered Feb 03 '20 at 14:59

Kunal

994
3
12
33

score -2 · Answer 7 · answered Oct 06 '15 at 21:01

-2

I'm also trying to do this, and I got it working using the Allow access to the domain from specific IP(s) option with the Elastic IP of my EC2 instance (could also work using the instance's private IP, but I'm not too sure)

answered Oct 06 '15 at 21:01

rkrdo

1,011
9
15

Proper access policy for Amazon Elastic Search Cluster

7 Answers7

Linked