0

I want to download a zip file from a url and parse its contents in node. I do not want to save the file on disk. The zip file is a directory of csv files that I want to process. How can I approach this? The only package that has an option for unzipping from a URL is unzipper but it does not work for me. Every other package lets you unzip a file on disk by providing the path to the file but not a url.

I am downloading the file like so:

const res = await this.get(test)

But what can I do now? There are packages like AdmZip that can extract zip files but need a path as a string to a file on disk. Is there a way I can pass/ stream my res object above to the below?

var AdmZip = require('adm-zip');

    // reading archives
    var zip = new AdmZip("./my_file.zip");
    var zipEntries = zip.getEntries(); // an array of ZipEntry records

    zipEntries.forEach(function(zipEntry) {
        console.log(zipEntry.toString()); // outputs zip entries information
        if (zipEntry.entryName == "my_file.txt") {
             console.log(zipEntry.getData().toString('utf8')); 
        }
    });
Oamar Kanji
  • 1,152
  • 2
  • 13
  • 25
  • i think you can't unzip from url. You need complete zip file to unzip a complete file, so you will need to download the file first. – AnonyMouze Apr 19 '21 at 11:50
  • Thanks, see edit above for more clarity. I am downloading it but not saving it to a file – Oamar Kanji Apr 19 '21 at 11:55
  • 1
    i see. you can put content in buffer and pass it to adm since it accepts buffer. You can refer to the is link . https://stackoverflow.com/questions/10359485/how-to-download-and-unzip-a-zip-file-in-memory-in-nodejs – AnonyMouze Apr 19 '21 at 12:02

1 Answers1

1

Here's a simple example of downloading a .zip file and unzipping using adm-zip. As @anonymouze points out, you can pass a buffer to the AdmZip constructor.

const axios = require("axios");
const AdmZip = require('adm-zip');

async function get(url) {
    const options =  { 
        method: 'GET',
        url: url,
        responseType: "arraybuffer"
    };
    const { data } = await axios(options);
    return data;
}

async function getAndUnZip(url) {
    const zipFileBuffer = await get(url);
    const zip = new AdmZip(zipFileBuffer);
    const entries = zip.getEntries();
    for(let entry of entries) {
        const buffer = entry.getData();
        console.log("File: " + entry.entryName + ", length (bytes): " + buffer.length + ", contents: " + buffer.toString("utf-8"));
    }
}

getAndUnZip('https://www.learningcontainer.com/wp-content/uploads/2020/05/sample-zip-file.zip');

In this case I'm simply using axios to download a zip file buffer, then parsing it with AdmZip.

Each entry's data can be accessed with entry.getData(), which will return a buffer.

In this case we'll see an output something like this:

File: sample.txt, length (bytes): 282, contents: I would love to try or hear the sample audio your app can produce. I do...

Here's another example, this time using node-fetch:

const fetch = require('node-fetch');
const AdmZip = require('adm-zip');

async function get(url) {
    return fetch(url).then(res => res.buffer());
}

async function getAndUnZip(url) {
    const zipFileBuffer = await get(url);
    const zip = new AdmZip(zipFileBuffer);
    const entries = zip.getEntries();
    for(let entry of entries) {
        const buffer = entry.getData();
        console.log("File: " + entry.entryName + ", length (bytes): " + buffer.length + ", contents: " + buffer.toString("utf-8"));
    }
}

getAndUnZip('https://www.learningcontainer.com/wp-content/uploads/2020/05/sample-zip-file.zip');
Terry Lennox
  • 17,423
  • 2
  • 18
  • 28