1

In a deployment bash script, I have two hosts:

  • localhost, which is a machine, that typically builds docker images.
  • $REMOTE_HOST, which is believed to be some production web server.

And I need to transfer locally built docker image to $REMOTE_HOST, in most efficient way (fast, reliable, private, storage-friendly). Up to day, I have following command in my streaming script:

docker save $IMAGE_NAME :latest | ssh -i $KEY_FILE -C $REMOTE_HOST docker load

This has following PROS:

  • Utilizes "compression-on-the-fly"
  • Does not stores intermediate files on both source and destination
  • Does direct transfer (images may be private), that also reduces upload time and stays "green", in another broader terms.

However, the CONS are also on checkerboard: When you are involved in transferring larger images, you dont know operation progress. So you have to wait unknown, but sensible time, that you cant estimate. I heard that progress can be tracked with kinda rsync --progress command

But rsync seems to transfer files, and is not working good with my ol'UNIX-style commands. Of couse you can docker load from some file, but how to avoid it?

How can I utilize piping, to preserve above advantages? (Or is there another special tool do copy build image to remote docker host, which shows progress?)

xakepp35
  • 1,949
  • 2
  • 18
  • 36
  • Can you set up a private Docker registry, or use a cloud-hosted registry? That’s almost always better than `docker save`/`load`. – David Maze Sep 05 '19 at 15:56
  • @DavidMaze As my friend loves to say, "almost doesn't count" :-) I need to do direct transfers, and just wanted to add a nice progress bar to existing structure. Is that possible, while leaving all other working almost "as it previously was"? – xakepp35 Sep 05 '19 at 17:27
  • 1
    I think the `docker save` path is fully streaming, and you don’t actually know how much data there will be until you’ve sent it all. That makes it impossible to have a progress bar without landing the file on disk somewhere. – David Maze Sep 05 '19 at 17:36

1 Answers1

3

You could invoke pv as part of your pipeline:

docker save $1:latest | pv [options...] | ssh -i $3 -C $2 docker load

pv works like cat, in that it reads from its standard input and writes to its standard output. Except that, like the documentation says,

pv allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA.

pv has a number of options to control what kind of progress information it prints. You should read the documentation and choose the output that you want. In order to display a percentage complete or an ETA, you will probably need to supply an expected size for the data transfer using the -s option.

Kenster
  • 18,710
  • 21
  • 68
  • 90
  • "Need to get 48,3 kB of archives..." ahaha, that's it! – xakepp35 Sep 05 '19 at 23:08
  • Could you do a bit of more work? I know image size up to single byte, but `ssh -C` does compression, so I dont know number of bytes.. So, the-follow-up question would be: *"what to pass to `-s` from a deployment bash script?"* – xakepp35 Sep 05 '19 at 23:11
  • As a stub, If we believe that size of stream, compressed by `-C`, would be less that actual size => our best bet could be [`stat --printf="%s" /path/to/local/docker/image`](https://unix.stackexchange.com/questions/16640/how-can-i-get-the-size-of-a-file-in-a-bash-script) – xakepp35 Sep 05 '19 at 23:15