Failure to run shell script via subprocess.Popen

Question

I am trying to a script run.sh using python subprocess

The script creates a csv file and a state.json file

This is the script(works fine when i run it from terminal) :

#!/bin/bash
export GOOGLE_APPLICATION_CREDENTIALS="/home/anti/Documents/bq_s3/BQ/bin/client_secret.json"

/home/anti/Documents/bq_s3/BQ/bin/tap-bigquery -c /home/anti/Documents/bq_s3/BQ/bin/tap_config.json --catalog /home/anti/Documents/bq_s3/BQ/bin/catalog.json --start_datetime '2013-01-01T00:00:00Z' --end_datetime '2013-01-02T00:00:00Z' | /home/anti/Documents/bq_s3/targetcsv/targetcsv/bin/target-csv --config /home/anti/Documents/bq_s3/targetcsv/targetcsv/bin/target_config.json > /home/anti/Documents/bq_s3/state.json

But when i try to run the same from python, it shows the correct output , but doesnt create csv file and a state.json file

This is the python code

#!/usr/bin/python3

import subprocess   
from ast import literal_eval
import io
import os
from subprocess import call

shellscript = subprocess.Popen(["/home/anti/Documents/bq_s3/run.sh"], stdout=subprocess.PIPE)
shellscript.wait()
print (shellscript.returncode)

Output

INFO Sending version information to singer.io. To disable sending anonymous usage data, set the config parameter "disable_collection" to true
INFO Running query:
    SELECT start_date,trip_seconds,trip_miles FROM `testbq-305217.test_1.tablename` WHERE 1=1 AND datetime '2013-01-01 00:00:00.000000' <= CAST(start_date as datetime) AND CAST(start_date as datetime) < datetime '2013-01-02 00:00:00.000000' ORDER BY start_date LIMIT 100
INFO METRIC: {"type": "counter", "metric": "record_count", "value": 1, "tags": {"endpoint": "test_bq"}}
INFO Syncing stream:test_bq
0

The return code is 0, the program works but the csv and json files are not being created when ran from python.

I have given necessary permission chmod u+rx run.sh to the script too.

Found the issue, the files are getting created at `home` directory and not the cwd. How do i change the output path ? — Shabari nath k, Mar 01 '21 at 08:41
You can either inside your script do an explicit `set -x`, or modify your `Popen` so that it effectively runs the command `bash -x run.sh` instead of simply `run.sh`. — user1934428, Mar 01 '21 at 08:42
I suggest that you ask this in a new question, since this is not related anymore to Python or bash, but to the correct usage of `target-csv`. As such, this question should go to [su] instead of [so]. — user1934428, Mar 01 '21 at 08:45

score 1 · Accepted Answer · edited Mar 01 '21 at 10:12

1

As you mentioned in the comments since the issue was that current working directory is not set properly pass cwd param to the subprocess.Popen. Better yet, replace subprocess.Popen() with a higher-level function, as recommended in its documentation.

#!/usr/bin/python3

import subprocess   

shellscript = subprocess.call(["/home/anti/Documents/bq_s3/run.sh"], cwd=YOUR_WORKING_DIR)
print(shellscript)

edited Mar 01 '21 at 10:12

tripleee

139,311
24
207
268

answered Mar 01 '21 at 08:48

Ritwik G

336
1
8

The issue is that the output files are getting created at home directory and not the `cwd`. The output path needs to be changed to `cwd`, anyway i can do that in the script ? – Shabari nath k Mar 01 '21 at 08:51
1

you can actually pass the `cwd` to subprocess.Popen – Ritwik G Mar 01 '21 at 09:03
https://stackoverflow.com/a/1685166/8029019 – Ritwik G Mar 01 '21 at 09:04
that worked man..thanks ..edit your answer so that i can accept it – Shabari nath k Mar 01 '21 at 09:08

Failure to run shell script via subprocess.Popen

1 Answers1