0

This is a 2 part question.

Ive made a bash script that logs into a remote server makes a list.txt and saves that locally.

#!/bin/bash
sshpass -p "xxxx" ssh user@pass ls /path/to/files/ | grep "^.*iso" > list.txt

It then starts a for loop using the list.txt

for f in $(cat list.txt); do

The next command splits the target file and saves it locally

sshpass -p "xxxx" ssh user@pass tar --no-same-owner -czf - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part"

Question 1

I need help understanding the above command, why is it saving the *part files locally? Even though that is what I intend to do I would like to understand it better, How would I do this the other way round, tar and split files saving output to remote directory (flip around what is happening in the above command using the same tools sshpass is a requirement)

Question 2

When running the above command even though I have made it not verbose it still prints this message

tar: Removing leading `/' from member names

How do I get rid of it as I have my own echo output as part of the script I have tried the following after searching online but I think me piping a few commands together confuses tar and breaks the operation.

I have tried these with no luck

sshpass -p "xxxx" ssh user@pass tar --no-same-owner -czfP - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part

sshpass -p "xxxx" ssh user@pass tar --no-same-owner -czf -C /path/to/files/$f | split -b 10M - "$f.tar.bz2.part

sshpass -p "xxxx" ssh user@pass tar --no-same-owner -czf - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part > /dev/null 2>&1

sshpass -p "xxxx" ssh user@pass tar --no-same-owner -czf - /path/to/files/$f  > /dev/null 2>&1 | split -b 10M - "$f.tar.bz2.part

All of the above break the operation and I would like it to not display any messages at all. I suspect it has something to do with regex and how the pipe passes through arguments. Any input is appreciated.

Anyways this is just part of the script the other part uploads the processed file after tar and splitting it but Ive had to break it up into a few commands a 'tar | split' locally, then uploading via rclone. It would be way more efficient if I could pipe the output of split and save it remotely via ssh.

Hani Umer
  • 11
  • 3

1 Answers1

0

First and foremost, you must consider the security vulnerabilities when using sshpass.

About question 1:

Using tar with -f - option will create the tar on the fly and will send to stdout. The | separates the commands.

  1. sshpass -p "xxxx" ssh user@pass tar --no-same-owner -czf - /path/to/files/$f - Runs remotely
  2. split -b 10M - "$f.tar.bz2.part" - Runs in local shell

The second command reads the stdin from the first command (the tar output) and it creates the file locally. If you want to perform all the operations in the remote machine, you could enclose the rest of the commands in quotes like this (read other sources about qouting).

sshpass -p "xxxx" ssh user@pass 'tar --no-same-owner -czf - /path/to/files/$f | split -b 10M - "$f.tar.bz2.part"'

About question 2.

tar: Removing leading '/' from member names is generated by tar command which sends errors/warnings to STDERR which in the terminal, STDERR defaults to the user's screen.

So you can suppress tar errors by adding 2>/dev/null:

sshpass -p "xxxx" ssh user@pass tar --no-same-owner -czf - /path/to/files/$f 2 > /dev/null | split -b 10M - "$f.tar.bz2.part

Ardit
  • 1,294
  • 16
  • 21
  • Thanks for your answer if you clarify one last thing I will accept the answer. How would I do this the other way round. create tars on the fly split and send the output over ssh to store remotely? So the opposite of the first command. – Hani Umer Oct 02 '20 at 22:53
  • Enclosing the commands in quotes (in your case in single qoutes) will run everything in the remote machine. Use the command suggested in the posted answer. Additionally if you want to send multiple commands you can use a `here document` as explained in this thread https://stackoverflow.com/questions/4412238/what-is-the-cleanest-way-to-ssh-and-run-multiple-commands-in-bash – Ardit Oct 03 '20 at 16:20