Save JSON outputted from a URL to a file

Question

How would I save JSON outputted by an URL to a file?

e.g from the Twitter search API (this http://search.twitter.com/search.json?q=hi)

Language isn't important.

edit // How would I then append further updates to EOF?

edit 2// Great answers guys really, but I accepted the one I thought was the most elegant.

Matthew Flaschen · Accepted Answer · 2010-06-14T21:14:32.817

This is easy in any language, but the mechanism varies. With wget and a shell:

wget 'http://search.twitter.com/search.json?q=hi' -O hi.json

To append:

wget 'http://search.twitter.com/search.json?q=hi' -O - >> hi.json

With Python:

urllib.urlretrieve('http://search.twitter.com/search.json?q=hi', 'hi.json')

To append:

hi_web = urllib2.urlopen('http://search.twitter.com/search.json?q=hi');
with open('hi.json', 'ab') as hi_file:
  hi_file.write(hi_web.read())

score 8 · Answer 2 · answered Jun 14 '10 at 21:12

In PHP:

$outfile= 'result.json';
$url='http://search.twitter.com/search.json?q=hi';
$json = file_get_contents($url);
if($json) { 
    if(file_put_contents($outfile, $json, FILE_APPEND)) {
      echo "Saved JSON fetched from “{$url}” as “{$outfile}”.";
    }
    else {
      echo "Unable to save JSON to “{$outfile}”.";
    }
}
else {
   echo "Unable to fetch JSON from “{$url}”.";
}

score 6 · Answer 3 · answered Jun 14 '10 at 21:10

6

You can use CURL

curl -d "q=hi" http://search.twitter.com -o file1.txt

answered Jun 14 '10 at 21:10

Bernie Perez

12,073
13
41
54

score 4 · Answer 4 · edited May 23 '17 at 12:16

Here's the (verbose ;) ) Java variant:

InputStream input = null;
OutputStream output = null;
try {
    input = new URL("http://search.twitter.com/search.json?q=hi").openStream();
    output = new FileOutputStream("/output.json");
    byte[] buffer = new byte[1024];
    for (int length = 0; (length = input.read(buffer)) > 0;) {
        output.write(buffer, 0, length);
    }
    // Here you could append further stuff to `output` if necessary.
} finally {
    if (output != null) try { output.close(); } catch (IOException logOrIgnore) {}
    if (input != null) try { input.close(); } catch (IOException logOrIgnore) {}
}

See also:

score 3 · Answer 5 · answered Jun 14 '10 at 21:05

3

In shell:

wget -O output.json 'http://search.twitter.com/search.json?q=hi'

answered Jun 14 '10 at 21:05

Ignacio Vazquez-Abrams

699,552
132
1,235
1,283

Just output to stdout (`-`) and use the append redirection operator (`>>`). – Ignacio Vazquez-Abrams Jun 14 '10 at 21:15

antonio · Answer 6 · 2010-12-30T18:14:57.713

1

You can use Jackson:

 ObjectMapper mapper = new ObjectMapper(); 
 Map<String,Object> map = mapper.readValue(url, Map.class);
 mapper.writeValue(new File("myfile.json"), map);

edited Dec 30 '10 at 18:14

answered Dec 30 '10 at 18:05

antonio

11
2

score 1 · Answer 7 · answered Mar 03 '11 at 18:04

Here is another way of doing this with PHP and fOpen.

<?php
// Define your output file name and your search query
$output = 'result.txt';
$search = 'great';

write_twitter_to_file($output, $search);

/*
 * Writes Json responses from twitter API to a file output.
 * 
 * @param $output: The name of the file that contains the output 
 * @param $search: The search term query to use in the Twitter API
*/

function write_twitter_to_file($output, $search) {
    $search = urlencode($search);
    $url = 'http://search.twitter.com/search.json?q=' . $search;
    $handle = fopen($url, "r");

    if ($handle) {
        while (($buffer = fgets($handle, 4096)) !== false) {
            file_put_contents($output, $buffer, FILE_APPEND);
            echo "Output has been saved to file<br/>";
        }

        if (!feof($handle)) {
            echo "Error: unexpected fgets() fail\n";
        }

        fclose($handle);
    }

}
?>

Save JSON outputted from a URL to a file

7 Answers7

Linked