Why does subprocess.call(['python', argsStr]) fail both with or without shell=True?

Question

So, I have a legacy script ts, that I need to call in a loop using several string arguments:

#User supplied
ts = '/d1/user/script.py'
adir = '/d1/user/adir'
mfile = '/d1/user/script.nc'
outdir = '/d1/user/park/'

metens = glob.glob(adir + '/*.nc')  #reads all files

for count, value in enumerate(metens):
    runcom = [ts, mfile, metens[count], outdir + os.path.basename(metens[count])]   
    runcom = " ".join(runcom) #This creates a string of CL arguments
    subprocess.call(['python2.7', runcom], shell=True)

Now, when I run it, it calls python2.7 and opens the Python shell instead of running it as python2.7 runcom.

How can I make it run as a script instead of opening the shell?

Remove the argument `shell=True`? If I run `python -c "import subprocess; subprocess.call(['python2.7', 'test.py'])"` it tells me that the file does not exist and exits 1. If I create that file and run again, it runs the script. — Jorge Leitao, Feb 20 '18 at 22:55
Possible duplicate of [Calling an external command in Python](https://stackoverflow.com/questions/89228/calling-an-external-command-in-python) — Jorge Leitao, Feb 20 '18 at 22:59
I removed the `shell=True`, And I am getting ` [Errno 2] No such file or directory`. Even though all the necessary files exists. If I just copy paste the python2.7 runcom to a command line directly, it works. So nothing is wrong with the program.. — maximusdooku, Feb 20 '18 at 23:15
If you use `shell=True` and pass a list, then the first list item is the only thing treated as a shell script (subsequent elements of the list are arguments to that script -- ie. its `$0`, `$1`, `$2`, etc), which is why they're ignored when the script doesn't refer to them. — Charles Duffy, Feb 20 '18 at 23:43
Anyhow -- if without `shell=True` you get a "No such file or directory" thrown by the Python interpreter, the plain reading of that error is that `python2.7` isn't in any directory in your PATH. Determine the location of that executable, determine what the value of `os.environ['PATH']` is at the point of execution of the code that fails, etc. — Charles Duffy, Feb 20 '18 at 23:46
Thanks for the explanation. But if I construct it inside the subprocess call, it works. — maximusdooku, Feb 20 '18 at 23:47
("plain reading" may not be the only reading -- you can get an error if, say, your Python interpreter requires a shared library and it's not present in the `rpath`, `ld.so` cached locations, or `LD_LIBRARY_PATH` directory -- but the plain reading is the place to start). — Charles Duffy, Feb 20 '18 at 23:47
...so, I think there may be some confusion here caused by the error not being included in the question as exact quoted text. The error being thrown by the *outer* interpreter and the *inner* interpreter have two completely different causes. — Charles Duffy, Feb 20 '18 at 23:49
`subprocess.call(['python2.7', runcom])` would cause the *inner* python interpreter to find a Python script named as the concatenation of all the arguments that went into `runcom`. That won't work for reasons that should be obvious (those arguments aren't actually part of the script's filename). — Charles Duffy, Feb 20 '18 at 23:50

Charles Duffy · Accepted Answer · 2018-02-21T00:04:40.817

How To Fix It

args = ['script.py', 'first argument', 'second argument']
subprocess.call(['python2.7'] + args)

Don't use shell=True
Pass each argument as a separate list item; don't concatenate them into a string.

Why It Fails (with `shell=True`)

Let's take a simple case:

args = [ 'script.py', 'first argument' 'second argument' ]
args_str = ' '.join(args)
subprocess.call(['python2.7', args_str], shell=True)

What does this actually do?

# the above is the same as running this at a shell
sh -c python2.7 'script.py first argument second argument'

And what does that actually do? It runs python2.7 with no arguments at all (as the argument list is interpreted as $0 to the sh -c instance, but the script passed in the first element of the list contains only the string python2.7 and doesn't look at its $0 at all).

Why It Fails (without `shell=True`)

Let's take a simple case:

args = [ 'script.py', 'first argument' 'second argument' ]
args_str = ' '.join(args)
subprocess.call(['python2.7', args_str])

What does this actually do?

# the above is the same as running this at a shell
python2.7 'script.py first argument second argument'

...and what does that do, even if you have a script.py in your current directory?

python2.7: can't open file 'script.py first argument second argument': [Errno 2] No such file or directory

Why did that happen? Because you made your arguments part of the script's filename, and no filename with those argument values as part of its name exists.

score 0 · Answer 2 · answered Feb 20 '18 at 23:41

0

The linked answer doesn't directly answer my question

for count, value in enumerate(metens):
    subprocess.call(['python2.7', ts, mfile, metens[count], outdir + os.path.basename(metens[count]])

The runcom, if constructed inside the subprocess, works. But if I construct it outside, I get a no file error.

answered Feb 20 '18 at 23:41

maximusdooku

4,158
7
35
73

Is this an attempt to answer the question? – Charles Duffy Feb 21 '18 at 00:02
Yes, this gives me the desired files. Though I am sure there is a better way of doing this. I am just leaving it in till something better comes along. – maximusdooku Feb 21 '18 at 00:06
No -- there isn't something better. This is the way it should be done. – Charles Duffy Feb 21 '18 at 00:06
(well -- you can construct it outside; what you need to avoid is condensing it down to a string, as opposed to keeping it as a list). – Charles Duffy Feb 21 '18 at 00:10
Thank you. I didn't know know what this was working. But I got it now. – maximusdooku Feb 21 '18 at 00:12

Why does subprocess.call(['python', argsStr]) fail both with or without shell=True?

2 Answers2

How To Fix It

Why It Fails (with shell=True)

Why It Fails (without shell=True)

Why It Fails (with `shell=True`)

Why It Fails (without `shell=True`)