Is there a CLI to tail logs from AWS Elastic Beanstalk

Question

Is there a CLI utility for tailing logs from Elastic Beanstalk applications. Specifically a python flask application.

You can use their eb CLI to get a snap shot ...

eb logs

But I would like to do (similar to what heroku offers)...

eb logs --tail

Has any one accomplished this?

Thanks!

Some references:

I know this is old but, would you like to mark one of the answers as correct? The one you ended up using perhaps? Thanks — Alexandre Martins, Jun 29 '15 at 14:25
check this out: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html — Alexander Mills, Nov 10 '18 at 08:17

Jorge Aranda · Answer 1 · 2018-04-10T17:16:42.947

25

UPDATE: My answer keeps getting upvotes, but things have changed since 2013. It is now easier to tail EB logs—look at posit labs's answer for a simpler way to do it :-).

Original answer:

I had been struggling with this one too. The eb CLI utility does not seem to allow for tailing your application logs currently.

However, you can tail these logs by:

Creating a key pair in the EC2 console (which should give you a .pem file)
Linking your EB instance to this key pair (in the EB console)
Finding the public DNS of your instance in the EC2 console
Connecting to your instance via ssh (`ssh -i [yourpemfile.pem] ec2-user@[your.public.dns]
Tailing your log file. For a Node.js application, that's tail -f /var/log/nodejs/nodejs.log. I don't know what's the equivalent for a Flask application.

(Thanks to Richard Soutar for pointing me in the right direction on this one.)

edited Apr 10 '18 at 17:16

answered Sep 04 '13 at 22:42

Jorge Aranda

1,948
2
19
27

tail -f /var/app/current/log/production.log for a Rails app deployed using AWS Beanstalk – Cheenu Madan Sep 29 '14 at 10:05
11

`eb ssh` allows to skip 1-4 steps and go straight to one of your running instances – Aleksey Kulikov Mar 25 '15 at 12:35
for rails with puma the following works for me: **eb ssh** and then **tail -f /var/log/puma/puma.log** – einSelbst Apr 06 '16 at 11:26
@AlekseyKulikov suggest you make that into an answer. I very nearly didn't read your comment. – toddg Oct 01 '16 at 14:15

score 12 · Answer 2 · answered Oct 25 '17 at 22:50

12

You can use the -g flag to set the log group, then use --stream to stream the results. No need to ssh into a specific machine.

eb logs -g /aws/elasticbeanstalk/yourApp-env/var/log/nodejs/nodejs.log --stream

answered Oct 25 '17 at 22:50

posit labs

7,733
4
32
57

hmmm how we get that path? I am using docker, so would it be `eb logs -g /aws/elasticbeanstalk/yourApp-env/var/log/docker/docker.log --stream` ? I hate guessing bro lol – Alexander Mills Nov 10 '18 at 08:14
I get this error: `ERROR: InvalidOptionsError - Can't retrieve instance logs for environment huru-api2-dev. Instance log streaming is disabled.` – Alexander Mills Nov 10 '18 at 08:15
I guess this has some answers: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html – Alexander Mills Nov 10 '18 at 08:16

eremite · Accepted Answer · 2021-03-25T16:29:18.110

11

2021 Update:

Now the best way to do this might be to use AWS CLI v2:

aws logs tail $log_group_name --follow

(Note: I couldn't get this working for me because of permissions errors.)

My previous answer was:

This is now as simple as:

eb logs --stream

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-logs.html

(I realize posit lab's answer already included the --stream option but I keep skimming by that answer because I want to see all the logs, not just one group. And if my memory were better I wouldn't have to keep Googling the same question and ending up at the same place.)

edited Mar 25 '21 at 16:29

answered Mar 23 '18 at 20:13

eremite

1,786
14
16

I get this error: `ERROR: InvalidOptionsError - Can't retrieve instance logs for environment huru-api2-dev. Instance log streaming is disabled.` – Alexander Mills Nov 10 '18 at 08:15
I guess this has some answers: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html – Alexander Mills Nov 10 '18 at 08:16
1

Likely need to run `eb logs --cloudwatch-logs enable` first – Jarvis Johnson Apr 05 '19 at 20:14
@JarvisJohnson I ran this and got `The following instances have not responded in the allowed command timeout time. Unsuccessful command execution on instance id(s) ''. Aborting the operation.` Somehow Beanstalk is extremely slow for any operation I tried. Anything leads to timeout. – Logan Yang Apr 27 '19 at 15:11
@LoganYang that sounds like a deeper issue - like beanstalk can't even got a response from the instances. I would try to rebuild the environment... – Jarvis Johnson Apr 27 '19 at 16:55
@JarvisJohnson I was able to run `eb local`, run the docker container and spin up the server locally with no problem, everything works on my machine. Haven't been able to figure out what's wrong after that. Every time I try to get the log, it starts updating the environment and gets stuck for several minutes, then it says log retrieved but it's nowhere to be found... – Logan Yang Apr 28 '19 at 15:05
2

Hi! I get output from calling `eb logs --stream` but it's not updating automatically when new log entries are coming in. I have to run the command again to see the new entries.. Any clue what might be wrong? – jonahe Sep 19 '19 at 12:14
Apparently this doesn't work anymore. (Or maybe it never did?!) I guess you need to [pass the log group](https://stackoverflow.com/a/46943205/167369) in with `-g` or use the [CloudWatch Logs UI](https://stackoverflow.com/a/48659489/167369). – eremite Sep 20 '19 at 20:20
I'm suffering the same issue with `jonahe`. On cloudwatch console, logs are refreshed but not in my terminal though I appended `-g` to specify the log gorup and `--stream`. – umi0410 Mar 23 '21 at 15:43
@umi0410 I don't think that method works anymore. I updated my answer to use aws cli version 2. – eremite Mar 25 '21 at 16:31

tarikakyol · Answer 4 · 2016-10-04T08:58:40.213

7

You might not need eb logs for it, try eb ssh first then

tail -f /var/log/*

edited Oct 04 '16 at 08:58

answered Jun 14 '14 at 14:41

tarikakyol

495
6
13

score 2 · Answer 5 · answered Feb 14 '16 at 15:38

So for others who are still looking for it, I wrote a small tool to tail logs for elastic beanstalk environment, you just need to specify environment name and file name, it will do the SSH and tail the file of each instance.

./eb-tail env-name

or

./eb-tail env-name -f '/path/to/file'

https://github.com/ahmednasir91/eb-tail

score 2 · Answer 6 · answered Oct 13 '17 at 18:59

This is what I do:

Grab the location of the logfile you want from eb logs

eb logs

Each log EB displays comes with a ------- preceding/prefacing it. For instance, for Passenger logs on Rails:

-------------------------------------
/var/app/support/logs/production.log
-------------------------------------

Then SSH into the server and tail the logs:

eb ssh    # may have to eb ssh --setup for first time use

# from inside the server
tail -f /var/app/support/logs/production.log

score 1 · Answer 7 · answered Mar 15 '21 at 12:06

More Detailed Answer as I was getting errors using eb logs --stream and with tail in SSH:

Enable Cloudwatch in eb CLI using this command eb logs --cloudwatch-logs enable.
Go to AWS Console and Open Cloudwatch. Now, go to Logs Groups, select the log you want to stream, web.stdout.log in my case and copy its path as highlighted in below image
Now, open the eb CLI again and type eb logs -g your_copied_path --stream. For example: eb logs -g /aws/elasticbeanstalk/yourEbsApp-env/var/log/web.stdout.log --stream. And then the log tailing will start.
(Optional) You can totally skip 3rd point and can directly stream the logs in above CloudWatch panel, just open the log group and select the log stream provided below it.

score 0 · Answer 8 · answered Oct 14 '14 at 10:59

You cannot tail the logs directly as AWS EB aggregates several logs together.assuming it's the application log that interests you, you can go to the EC2 tab and press the connect button on the actual server (you may need to define key-pair, just follow the instructions). Once logged into the server you can tail your log from there.

score 0 · Answer 9 · edited Jul 04 '19 at 02:11

0

Something I use to tail several logs from multiple servers, including aws, is the following:

for $server in server-1 server-2 server-3 server-4; do
    ssh -t $server -f "tail -f /var/log/some-process/error.log|sed -e 's/^/$server /'" &
done

Source: https://gist.github.com/iolloyd/da60ef316643d7894bdf

It allows me to monitor multiple remote instances in my local machine.

edited Jul 04 '19 at 02:11

Dave Powers

1,510
2
23
26

answered Mar 03 '15 at 21:47

Lloyd Moore

2,939
28
29

score 0 · Answer 10 · answered Feb 07 '18 at 08:52

0

This doesn't use the eb-cli, but the simplest way to stream logs would be to use eb logs -cw enable then reading the stream in the cloudwatch portal.

answered Feb 07 '18 at 08:52

jimmy

159
2
6

score 0 · Answer 11 · answered May 16 '20 at 07:30

0

For me it involved two steps.

You need to run eb logs --stream, but before this, run: eb logs --cloudwatch-logs enable

Voila!

answered May 16 '20 at 07:30

Qasim

1,328
1
11
21

score -6 · Answer 12 · answered Jun 21 '13 at 13:04

-6

I don't have an environment where i can test this but from looking at the code it looks like you can do:

eb logs tail

check out constants.py line 129

answered Jun 21 '13 at 13:04

probablyrobots

21
4

I have moved on from EB for a couple of reasons. Let me know if anyone can confirm this. – Jonathan Jun 21 '13 at 13:18
Unfortunately it doesn't look like the `eb logs tail` makes any difference over `eb logs` – Jorge Aranda Sep 04 '13 at 22:20
`ERROR: No Environment found for EnvironmentName = 'tail'.` – Jason Swett Aug 30 '15 at 19:38
1

This answer is wrong. There is no such thing in eb cli. – David Rissato Cruz Oct 18 '16 at 15:05

Is there a CLI to tail logs from AWS Elastic Beanstalk

12 Answers12