LAMP: How to create .Zip from remote URLs/files and stream it to the client on the fly

Question

With external services like S3 for object storage, the old problem of providing custom zip archives of stored files gets a bit more complicated. One way would be for a web server to copy all the resources to temporary local files, compile them into a .zip, then return the zip to the user, but this is slow and resource intensive.

Can this be done similarly to the solution for local files? e.g. can you curl the files, pipe them into zip in streaming mode, then out to the user on the fly?

Answer 1

Yes — you can have zip operate on a set of fifos rather than a set of files. Here’s the code:

$sh = 'mkfifo a.jpg b.jpg ; '.                     // 1
  'curl -s some_service.com/a.jpg > a.jpg & ' .    // 2
  'curl -s some_service.com/b.jpg > b.jpg & ' .    // 
  'zip -FI - a.jpg b.jpg | cat > out.fifo';        // 3

// Open output fifo & curl-zip task pipe
posix_mkfifo('out.fifo', 0777);                    // 4
$pipe_output = popen('cat out.fifo', 'r');         //
$pipe_curlzip = popen($sh, 'r');                   //

It works as follows:

1. We have two remote files: some_service.com/a.jpg and some_service.com/b.jpg – we create a fifo for each, each fifo having the name of the corresponding file as we wish it to appear in our zip file. So for some_service.com/a.jpg, we create a fifo called a.jpg.
2. Kick off two curl tasks to feed the remote files a.jpg and b.jpg into each’s respective fifo. These curl tasks will hang until something reads from their output fifos, so we run them in the background (&).
3. Start our zip task, giving it our two fifos as input. We cat its streaming output into a third fifo – out.fifo – to allow our PHP process to stream the file to the user.
4. Kick things off.

It's then just a matter of reading what comes out of out.fifo and sending it to the user:

header('Content-Type: application/octet-stream');
header('Content-Disposition: attachment; filename="Download.zip"');

while (!feof($pipe_output)) {
    echo fread($pipe_output, 128);   // Your magic number here
}