11

I try to run a shell script with crontab which runs python3 scripts. The crontab is for a user group. Now it runs the script but not the python3 scripts inside it. I try to debug it but I can't figure out what happens. It might be a permission issue or a path problem but I can't figure out. This is the line crontab

*/5 * * * * /home/group_name/path/to/script/run.sh

As I said the cron job is executed or at least thats what I think since when I run sudo grep CRON /var/log/syslog I get lines like

 Feb 16 20:35:01 ip-**-**-*-*** CRON[4947]: (group_name) CMD (/home/group_name/path/to/script/run.sh)

right below I also get a line which might have something to do with the problem

Feb 16 20:35:01 ip-**-**-*-*** CRON[4946]: (CRON) info (No MTA installed, discarding output)

and finally run.sh looks like this

#!/bin/bash

# get path to script and path to script directory
SCRIPT=$(readlink -f "$0")
SCRIPTPATH=$(dirname "$SCRIPT")

echo "set directory"
cd "$SCRIPTPATH"

echo "run first script"
/usr/bin/python3 ./first_script.py > ./log1.txt

However when the cron job executes it nothing happens, when I run it manually the cahnges to the database happen as expected. The group has the same rights as I have. The shell file can be executed by me and the group and the python files can't be executed by me so I don't know why the group would need this.

PS: I want to execute the python script in a shell since we have a lot of scripts with some times a lot of arguments and hence the crontab would become overpopulated and some scripts have to be executed in a certain order.

EDIT: Adding exec >> /tmp/output 2>&1 right after #! /bin.bash writes the echoes to /tmp/output whenever I run it manually but not when I run it in cron, not even the one before running any python script.

Running one of the python scripts directly from cron works, however even if I copy paste the exact same line as the one that works in cron, into the shell file, nothing happens.

Yannick Widmer
  • 984
  • 11
  • 25
  • the ./first_script.py should also be fully qualified... – Ivonet Feb 16 '18 at 20:53
  • 2
    Add `exec >> /tmp/output 2>&1` near the top of your shell script. Come back and tell us what the contents of `/tmp/output` reveal. Also, what do you learn from the contents of `/home/group_name/path/to/script/log1.txt` ? – Robᵩ Feb 16 '18 at 20:54
  • why not `/usr/bin/python3 "$SCRIPTPATH/first_script.py"` instead of `cd "$SCRIPTPATH" && /usr/bin/python3 ./first_script.py` ? Module search will be relative to the path of the python script no matter what is the current working directory. – Paulo Scardine Feb 16 '18 at 20:57
  • So with the `exec` command I get the echoes in `/tmp/output/` when I run it manually. I changed them to see if they will be updated next time the cron runs. My `log1.txt` file contains the output of my script but gets only updated when I run it manually. In a minute it will run again but it takes some time so I'll wait a bit and give an update. – Yannick Widmer Feb 16 '18 at 22:35
  • So I changed some echoes and `/tmp/output` was not updated eventhoug `sudo grep CRON /var/log/syslog` shows that the cron job was run. – Yannick Widmer Feb 16 '18 at 23:20
  • Does your script require a TTY? Cron doesn't have a TTY while a terminal has. Also I would compare the env of both environments. So I would add a `env` statement at the top of my bash script and compare the two environments when run through terminal – Tarun Lalwani Feb 20 '18 at 06:08
  • It sets the working directory and then runs several python scripts thats it. I will try the path varaiable tomorrow but I added the absolut python path i.e. /usr/bin/python3 to run the files but it is still not working. – Yannick Widmer Feb 20 '18 at 06:13
  • The "No MTA" message means that you have no Mail Transfer Agent (such as sendmail or Postfix) installed. If you had that, you'd receive an email with the output of the cron command. As you don't, add `>> /tmp/my_cron.log 2>&1` to the end of your cron command, and you can check the output. – Benjamin W. Feb 20 '18 at 06:22
  • Or maybe you just didn't give your shell script execute perms? (eg: were you running your shell script as `bash run.sh` manually?) – conrad Feb 26 '18 at 12:36

6 Answers6

5

There are a lot of components to this problem. I'm ignoring the MTA error because that's just about an email notification when your cron job finishes. I'll also assume you have permissions set properly and that your script runs fine when run manually in the shell.

The biggest issue is that the CRON command isn't the same as running the command from the terminal "shell". You have to specify that your script get run using bash. Change your cron job from:

*/5 * * * * /home/group_name/path/to/script/run.sh 

to:

*/5 * * * * bash /home/group_name/path/to/script/run.sh

This question has more information and additional options for solving the problem.

YPCrumble
  • 20,703
  • 15
  • 86
  • 149
  • This actually solved the problem, however it feels like a hack I had absolut paths and everything and it still didn't work. Now I have everything like I started with and it works. – Yannick Widmer Mar 03 '18 at 00:36
  • I usually start my sessions with /bin/bah, however the script was also working without doing so. So I really don't understand what happened. – Yannick Widmer Mar 03 '18 at 00:37
2

1) The message "No MTA installed" does not mean any error, it only indicates that without mail server cron cannot report any details.

Modify cron job to log its output into syslog:

*/5 * * * * /home/group_name/path/to/script/run.sh 2>&1 | /usr/bin/logger -t run.sh

Then check results via sudo tail -f /var/log/syslog (or sudo tail -f /var/log/messages on RedHat and SuSE)

Alternatively, install Postfix and configure it for "local only" delivery:

sudo apt-get install postfix

Then check mail as the group_name user.

2) The redirection > ./log1.txt in run.sh should overwrite log file upon each execution. If python script failed with exception then log1.txt would remain truncated to zero length. In that case modify the last line of run.sh:

/usr/bin/python3 ./first_script.py 2>&1 > ./log1.txt

and check results.

If log1.txt is neither truncated nor contains fresh output then the python script is not being launched at all. Refer to step 1) to debug the run.sh.

void
  • 2,629
  • 9
  • 25
  • My suggestion about `readlink` and `dirname` paths turned out to be wrong. I have edited the answer to describe possible debugging approach. – void Feb 28 '18 at 00:21
2

Change this line:

*/5 * * * * /home/group_name/path/to/script/run.sh

to:

*/5 * * * * cd /home/group_name/path/to/script && /home/group_name/path/to/script/run.sh

Regarding /var/log/syslog, when you look in /var/log/syslog, look at the timestamps to figure out if the cron job is being run.

Regarding the cron job not being able to write into log.txt, it could have to do with permissions. Try changing this line:

/usr/bin/python3 ./first_script.py > ./log1.txt

to:

/usr/bin/python3 /full/path/to/first_script.py > /tmp/log1.txt

See if there is any difference. cron should be able to write into /tmp.

hikerjobs
  • 233
  • 3
  • 11
  • No nothing is written to /tmp/log1.txt – Yannick Widmer Feb 24 '18 at 02:08
  • I changed the to: line above a bit. Use the full path to first_script.py instead of ./first_script.py and see if that helps. Also, what happens when you make the change in the cron file as suggested at the top of my answer -- cd /home/group_name... && /home/group_name/.../run.sh? – hikerjobs Feb 24 '18 at 05:33
1

The last line in your bash script contains relative paths (./) I believe this is the problem

Dror Paz
  • 357
  • 2
  • 14
  • To me, the relative path seems purposeful. That's the point of querying the path of `$0` and `cd`-ing to that location. But you may be right. – Robᵩ Feb 16 '18 at 20:57
  • I ran the shell file from different locations and it always worked. So this shouldn't be an issue. This is a minimal working example, the reason why I am `cd`-ing to that location is that I run several scripts and I want the shell scripts to be as readable as possible. – Yannick Widmer Feb 16 '18 at 23:22
  • I still tried with absolut path and get the same problem. – Yannick Widmer Feb 16 '18 at 23:39
1

There is currently a lot of guessing about the problem, and that is because your system is unable to send you the failure email to explain exactly what the problem is. I ran into a similar problem a while back, was overwhelmed by trying to set up an actual mail system, so wrote a short mail forwarding sendmail stand-in: pygeon_mail:

#!/usr/bin/python
from __future__ import with_statement
from email.mime.text import MIMEText
import email
import os
import pwd
import smtplib
import stat
import sys
import syslog
import traceback


CONFIG = '/etc/pygeon_mail.rc'
# example config file
#
# server=mail.example.com
# port=25
# domain=example.com
# host=this_pc_host_name
# root=me@example.com,you@example.com
# ethan=me@example.com
# debug=debug@example.com


def check_dangerously_writable(filename):
    "return the bits of group/other that are writable"
    mode = stat.S_IMODE(os.stat(filename)[0])       # get the mode bits
    if mode & (stat.S_IWGRP | stat.S_IWOTH):        # zero means not set
    syslog.syslog("%s must only be writable by root, aborting" % (filename, ))
    sys.exit(1)

def get_config(filename, config=None):
    "return settings from config file"
    check_dangerously_writable(filename)
    if config is None:
    config = {}
    with open(filename) as settings:
    for line in settings:
        line = line.strip()
        if line and line[:1] != '#':
        key, value = line.split('=')
        key, value = key.strip(), value.strip()
        config[key] = value
    return config

def mail(server, port, sender, receiver, subject, message):
    """sends email.message to server:port

    receiver is a list of addresses
    """
    msg = MIMEText(message.get_payload())
    for address in receiver:
    msg['To'] = address
    msg['From'] = sender
    msg['Subject'] = subject
    for header, value in message.items():
    if header in ('To','From', 'Subject'):
        continue
    msg[header] = value
    smtp = smtplib.SMTP(server, port)
    try:
    send_errs = smtp.sendmail(msg['From'], receiver, msg.as_string())
    except smtplib.SMTPRecipientsRefused as exc:
    send_errs = exc.recipients
    smtp.quit()
    if send_errs:
    errs = {}
    for user in send_errs:
        if '@' not in user:
        errs[user] = [send_errs[user]]
        continue
        server = 'mail.' + user.split('@')[1]
        smtp = smtplib.SMTP(server, 25)
        try:
        smtp.sendmail(msg['From'], [user], msg.as_string())
        except smtplib.SMTPRecipientsRefused as exc:
        if send_errs[user] != exc.recipients[user]:
            errs[user] = [send_errs[user], exc.recipients[user]]
        else:
            errs[user] = [send_errs[user]]
        smtp.quit()
    for user, errors in errs.items():
        for code, response in errors:
        syslog.syslog('%s --> %s: %s' % (user, code, response))
    return errs


if __name__ == '__main__':
    syslog.openlog('pygeon', syslog.LOG_PID)
    try:
    config = get_config(CONFIG)
    root = config.get('root')
    domain = config.get('domain', '')
    if domain:
        domain = '@' + domain
    sender = None
    receiver = []
    dot_equals_blank = False
    ignore_rest = False
    next_arg_is_subject = False
    redirect = False
    subject = ''
    for i, arg in enumerate(sys.argv[1:]):
        if next_arg_is_subject:
        subject = arg
        next_arg_is_subject = False
        sys.argv[i] = '"%s"' % (arg, )
        elif arg == '-s':
        next_arg_is_subject = True
        elif arg == '-i':
        dot_equals_blank = True
        elif arg[:2] == '-F':
        sender = arg[2:]
        elif arg[0] != '-':
        receiver.append(arg)
        else:
        pass
    command_line = ' '.join(sys.argv)
    if sender is None:
        sender = pwd.getpwuid(os.getuid()).pw_name
    sender = '%s@%s' % (sender, config['host'])
    if not receiver:
        receiver.append(pwd.getpwuid(os.getuid()).pw_name)
    limit = len(receiver)
    for i, target in enumerate(receiver):
        if i == limit:
        break
        if '@' not in target:
        receiver[i] = ''
        receiver.extend(config.get(target, target+domain).split(','))
    receiver = [r for r in receiver if r]
    server = config['server']
    port = int(config['port'])
    all_data = []
    text = []
    while True:
        data = sys.stdin.read()
        if not data:
        break
        all_data.append(data)
        if ignore_rest:
        continue
        for line in data.split('\n'):
        if line == '.':
            if dot_equals_blank:
            line = ''
            else:
            ignore_rest = True
            break
        text.append(line)
    text = '\n'.join(text)
    message = email.message_from_string(text)
    errs = mail(server, port, sender, receiver, subject, message)
    except Exception:
    exc, err, tb = sys.exc_info()
    lines = traceback.format_list(traceback.extract_tb(tb))
    syslog.syslog('Traceback (most recent call last):')
    for line in lines:
        for ln in line.rstrip().split('\n'):
        syslog.syslog(ln)
    syslog.syslog('%s: %s' % (exc.__name__, err))
    sys.exit(1)
    else:
    receiver = []
    debug_email = config.get('debug', None)
    if debug_email:
        receiver.append(debug_email)
    if errs and root not in receiver:
        receiver.append(root)
    if receiver:
        debug = [
            'command line:',
            '-------------',
            repr(command_line),
            '-' * 79,
            '',
            'sent email:',
            '-----------',
            text,
            '-' * 79,
            '',
            'raw data:',
            '---------',
            ''
            ]
        all_data = ''.join(all_data)
        while all_data:
        debug_text, all_data = repr(all_data[:79]), all_data[79:]
        debug.append(debug_text)
        debug.append('-' * 79)
        if errs:
        debug.extend([
            '',
            'errors:',
            '-------',
            ])
        for address, error in sorted(errs.items()):
            debug.append('%r: %r' % (address, error))
        debug.append('-' * 79)
        text = '\n'.join(debug)
        message = email.message_from_string(text)
        mail(server, port, 'debug@%s' % config['host'], receiver, subject+'  [pygeon_mail debugging info]', message)
    if errs:
        sys.exit(1)

It was written for Python 2.5 and should work with 2.6 and 2.7.

It needs to be copied to /usr/sbin/sendmail with permissions of 0755 and owned by root:

sudo cp pygeon_mail /usr/sbin/sendmail

sudo chown root:root /usr/sbin/sendmail

sudo chmod 0755 /usr/sbin/sendmail

You need to create an /etc/pygeon_mail.rc config file (see code for an example).

You can then test it with something like:

$ echo some useful info | sendmail myself -s "some important subject"

and you will hopefully see that email in your normal email account (which you set up in the /etc/pygeon_mail.rc file).

After that, you should be able to get the actual error, and we can actually help you

Community
  • 1
  • 1
Ethan Furman
  • 52,296
  • 16
  • 127
  • 201
0

If you log the output of the script you're invoking with cron, you can find the error pretty easy. Try something like this:

*/10 * * * * sh /bin/execute/this/script.sh >> /var/log/script_output.log 2>&1

Jan
  • 21
  • 4