my cUrl reliquishes control to the next statement before it's finished

Question

I'm running a script of commands (OSX) as follows

curl -o yahoo_IDJG.dwn http://ichart.finance.yahoo.com/table.csv?s=IDJG.AS&a=10&b=8&c=2015&g=d&ignore=.csv && sed -i -e 's/^/yahoo,IDJG,xams,/' yahoo_IDJG.dwn > yahoo_IDJG.csv;

or

curl -o yahoo_IDJG.dwn http://ichart.finance.yahoo.com/table.csv?s=IDJG.AS&a=10&b=8&c=2015&g=d&ignore=.csv;
sed -i -e 's/^/yahoo,IDJG,xams,/' yahoo_IDJG.dwn > yahoo_IDJG.csv;

unfortunately the sed starts before the curl is even finished. In fact all commands seem to start immediately.

I'm new to this so what haven't I understood ?

Thanks for you help.

score 1 · Answer 1 · answered Feb 10 '16 at 22:45

The problem is that you miss the quotes, so :

curl -o yahoo_IDJG.dwn 'http://ichart.finance.yahoo.com/table.csv?s=IDJG.AS&a=10&b=8&c=2015&g=d&ignore=.csv'
sed -e 's/^/yahoo,IDJG,xams,/' yahoo_IDJG.dwn > yahoo_IDJG.csv

And when sed is used with -i switch, there's no output

score 1 · Accepted Answer · edited May 23 '17 at 11:45

First - note that the & tells your shell to run the given command in the background, then continue to the next line. Therefore, your initial command is really running:

1 curl -o yahoo_IDJG.dwn http://ichart.finance.yahoo.com/table.csv?s=IDJG.AS&
2 a=10&
3 b=8&
4 c=2015&
5 g=d&
6 ignore=.csv;
7 sed -i -e 's/^/yahoo,IDJG,xams,/' yahoo_IDJG.dwn > yahoo_IDJG.csv;

So what's happening is that the shell is parsing the command as a series of "background" processes, split up according to the & character. So, it

runs this in the background
assigns the shell variable $a to 10 (in the background)
assigns the shell variable $b to 8 (in the background)
assigns the shell variable $c to 2015 (in the background)
assigns the shell variable $g to d (in the background)
assigns the shell variable $ignore to .csv (not in the background, but that happens very quickly as shell variable assignment is reasonably quick).
finally, runs a sed command on the yahoo.IDJG.dwn file (though creates a backup of the file with a -e extension on the end - check the man page for sed on a mac or see the discussion on sed in-place flag that works both on Mac (BSD) and Linux) so you should have a yahoo.IDJG.dwn file (with the results of the sed command), a yahoo.IDG.dwn-e file (the original file), and (probably empty) yahoo.IDJG.csv file. The solution is to enclose the entire URL in the curl command in single-quotes, and remove the -i from the sed command.

For reference, the man page for sed on a mac has the following:

-i extension
         Edit files in-place, saving backups with the specified extension.  If a zero-length extension is given, no backup will be saved.  It is not recom-
         mended to give a zero-length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is
         exhausted, etc.

EDIT: forgot to post the real answer!: Just enclose the URL in single quotes and remove the -i flag to the sed command:

curl -o yahoo_IDJG.dwn 'http://ichart.finance.yahoo.com/table.csv?s=IDJG.AS&a=10&b=8&c=2015&g=d&ignore=.csv' && sed -e 's/^/yahoo,IDJG,xams,/' yahoo_IDJG.dwn > yahoo_IDJG.csv;

EDIT: If you don't care about the intermediate yahoo_IDJG.dwn file, you can also just pipe the curl to the sed command:

curl -s 'http://ichart.finance.yahoo.com/table.csv?s=IDJG.AS&a=10&b=8&c=2015&g=d&ignore=.csv' | sed -e 's/^/yahoo,IDJG,xams,/' > yahoo_IDJG.csv;

When I ran that on command line, I got the same output in the .csv files.

my cUrl reliquishes control to the next statement before it's finished

2 Answers2