How can I loop over comma-separated lists inside each line of a file?

Question

Need to write some status checker at bash-script:

Have file with strings like that:

domain.com; 111.111.111.111,222.222.222.222; /link/to/somefile.js,/link/to/somefile2.js
domain2.com; 122.122.111.111,211.211.222.222; /link/to/somefile2.js,/link/to/somefile3.js

Need to execute such commands at total:

curl -s -I -H 'Host: domain.com' http://111.111.111.111/link/to/somefile.js
curl -s -I -H 'Host: domain.com' http://222.222.222.222/link/to/somefile.js

curl -s -I -H 'Host: domain.com' http://111.111.111.111/link/to/somefile2.js
curl -s -I -H 'Host: domain.com' http://222.222.222.222/link/to/somefile2.js

curl -s -I -H 'Host: domain2.com' http://122.122.111.111/link/to/somefile2.js
curl -s -I -H 'Host: domain2.com' http://211.211.222.222/link/to/somefile2.js

curl -s -I -H 'Host: domain2.com' http://122.122.111.111/link/to/somefile3.js
curl -s -I -H 'Host: domain2.com' http://211.211.222.222/link/to/somefile3.js

The question is:

what tool do I need to use to have such result at total?
Maybe xargs with some arguments/flags can do that or gnu parallel?
Can you, please, show examples?

I can to separate lines and set result to different variables that's isn't problem at all:

domain=$(cut -d';' -f1 file| xargs -I N -d "," echo curl -H) \'N\'
ip=$(cut -d';' -f2 file| xargs -I N -d "," echo curl -H) \'N\'
and else

But question at other :) : how after delimiting and separating strings to variables, I can execute curl with different variables at that case - the number of arguments for different variables will be different ?

The answer's that get Barmar doesn't cover task problem at all, cause it has greater than two list's. The problem is not at ignorance of bash, but of way I can resolve issue

Is it accurate that spaces can only appear concatenated to semicolons? — Charles Duffy, Sep 20 '18 at 15:02
You can use a `while read` loop, after setting `IFS` so that it splits on `;` and space. Then split up the IP and pathname parts on space, and use a nested loop for that part. — Barmar, Sep 20 '18 at 15:02
StackOverflow is not a free coding service. SO expects you to [try to solve your own problem first](https://meta.stackoverflow.com/questions/261592). Please update your question to show what you have already tried in a [mcve]. For further information, please see [ask], and take the [tour] :) — Barmar, Sep 20 '18 at 15:02
[BashFAQ #1](http://mywiki.wooledge.org/BashFAQ/001) is a good place to start. — Charles Duffy, Sep 20 '18 at 15:04
I believe this problem can be broken down into three separate technical challenges: Iterating over a file line-by-line (reading each column into a different variable); splitting a string on a delimiter into an array; and iterating over distinct arrays (the ones you split the IP and file fields into) in lockstep. We already have answered questions about how to do all of those things. If you're getting stuck somewhere else, please [edit] the question to show how you tried to apply those answers, and exactly which problem you still have. — Charles Duffy, Sep 20 '18 at 15:10
I can to separate line with and set result to different variables. domain=(cut -d';' -f1 file| xargs -I N -d "," echo curl -H) \'N\', ip=(cut -d';' -f2 file| xargs -I N -d "," echo curl -H) \'N\' — Kein, Sep 20 '18 at 15:32
But question at other :) how after delimiting and separating strings to variables I can execute curl with different variables at that case - the number of arguments for different variables will be different — Kein, Sep 20 '18 at 15:37
The answer's that get Barmar doesn't cover task problem at all, cause it has greater than two list's — Kein, Sep 20 '18 at 15:41
The code in your question doesn't appear to reflect any attempt to use or build on the answers to the linked duplicates, or the FAQ I linked earlier. (`xargs` isn't the right tool here; neither is `cut`, for that matter). — Charles Duffy, Sep 20 '18 at 16:25
Yes, there are inner lists, but if you combine the linked answers you get what you need: Iterate over the lines, reading each of the columns into a variable; then split the columns that have inner lists into arrays; then iterate over the arrays. All the answers you need are there -- and if you showed code that indicated that you were trying to apply them and just ran into a problem in the attempt, you'd get a much friendlier reception than showing no effort or intent to listen to the advice you're given. — Charles Duffy, Sep 20 '18 at 16:32
"xargs isn't the right tool here; neither is cut" because.. :) can you please explain your position? I resolve to problem's in one line if i add "while readline" around, there will leave one unresolved question: how to get curl works with this as you say "inner array's". The answer I get resolve work with two similar arrays but not even more — Kein, Sep 20 '18 at 16:42
`cut` is expensive: Because it's not part of bash, every time you start it up, you're doing a fork and exec pair, plus all the FIFO overhead in the command substitution to capture its output. Bash has its own built-in string manipulation primitives -- see [BashFAQ #100](http://mywiki.wooledge.org/BashFAQ/100) -- and sticking to built-in functionality is way faster (pertinent if you're running in an inner loop over a lot of inputs). — Charles Duffy, Sep 20 '18 at 16:59
Moreover, generating shell commands *as strings* has serious pitfalls -- see [BashFAQ #48](http://mywiki.wooledge.org/BashFAQ/048) if you try to evaluate them with `eval`, or [BashFAQ #50](http://mywiki.wooledge.org/BashFAQ/050) if you don't. — Charles Duffy, Sep 20 '18 at 16:59

Charles Duffy · Accepted Answer · 2018-09-20T16:51:34.593

#!/usr/bin/env bash
#              ^^^^- IMPORTANT: not /bin/sh

# print command instead of running it, so people can test their answers without real URLs
log_command() { printf '%q ' "$@"; printf '\n'; }

while IFS='; ' read -r domain addrs_str files_str; do
  IFS=, read -a addrs <<<"$addrs_str"
  IFS=, read -a files <<<"$files_str"
  for file in "${files[@]}"; do
    for addr in "${addrs[@]}"; do
      log_command curl -s -I -H "Host: $domain" "http://$addr/$file"
    done
  done
done

...emits as output (as the list of commands if it would run if the log_command prefix were removed):

curl -s -I -H Host:\ domain.com http://111.111.111.111//link/to/somefile.js 
curl -s -I -H Host:\ domain.com http://222.222.222.222//link/to/somefile.js 
curl -s -I -H Host:\ domain.com http://111.111.111.111//link/to/somefile2.js 
curl -s -I -H Host:\ domain.com http://222.222.222.222//link/to/somefile2.js 
curl -s -I -H Host:\ domain2.com http://122.122.111.111//link/to/somefile2.js 
curl -s -I -H Host:\ domain2.com http://211.211.222.222//link/to/somefile2.js 
curl -s -I -H Host:\ domain2.com http://122.122.111.111//link/to/somefile3.js 
curl -s -I -H Host:\ domain2.com http://211.211.222.222//link/to/somefile3.js

...as you can see at https://ideone.com/dTC8q8

Now how does this work?

Step 1: Read each line into domain, addrs_str and files_str, split on semicolons and spaces.

That's what's done by the line IFS='; ' read -r domain addrs_str files_str, which operates as described in BashFAQ #1, and in How to read variables from file, with multiple variables per line?
Step 2: For addrs_str and files_str, split them on commas into separate arrays. This is described in How do I split a string on a delimiter in Bash?
Step 3: Iterate over those arrays, and call curl for each combination. If you wanted to call the first IP with only the first file, and the second IP with the second file, you could use Iterate over two arrays simultaneously in bash; otherwise, it's a plain nested loop.

score 0 · Answer 2 · answered Sep 21 '18 at 09:50

With GNU Parallel it would look like this

doit() {
  domain="$1"
  ips="$2"
  paths="$3"
  parallel --dry-run -d ',' -q curl -s -I -H Host:\ "$domain" http://{1}/{2} ::: "$ips" ::: "$paths"
}
export -f doit
parallel --colsep ';' doit :::: input.file

Remove --dry-run when you are convinced it works.

How can I loop over comma-separated lists *inside* each line of a file?

2 Answers2

How can I loop over comma-separated lists inside each line of a file?