Use sed (or similar) to remove anything between repeating patterns

Question

I'm essentially trying to "tidy" a lot of data in a CSV. I don't need any of the information that's in "quotes".

Tried sed 's/".*"/""/' but it removes the commas if there's more than one section together.

I would like to get from this:

1,2,"a",4,"b","c",5

To this:

1,2,,4,,,5

Is there a sed wizard who can help? :)

see https://stackoverflow.com/questions/5319840/greedy-vs-reluctant-vs-possessive-quantifiers wrt why your attempt failed — Sundeep, Apr 16 '20 at 14:59
Please take a look at: [What should I do when someone answers my question?](http://stackoverflow.com/help/someone-answers) — Cyrus, Apr 17 '20 at 12:10
All 3 solutions work - thank you all. Just found a curveball that in some lines, there are line breaks - just need to remove those and I'm sorted :) — materangai, Apr 22 '20 at 08:13

score 3 · Accepted Answer · answered Apr 16 '20 at 14:34

You may use

sed 's/"[^"]*"//g' file > newfile

See online sed demo:

s='1,2,"a",4,"b","c",5'
sed 's/"[^"]*"//g' <<< "$s"
# => 1,2,,4,,,5

Details

The "[^"]*" pattern matches ", then 0 or more characters other than ", and then ". The matches are removed since RHS is empty. g flag makes it match all occurrences on each line.

RavinderSingh13 · Answer 2 · 2020-04-16T14:47:35.197

Could you please try following.

awk -v s1="\"" 'BEGIN{FS=OFS=","} {for(i=1;i<=NF;i++){if($i~s1){$i=""}}} 1' Input_file

Non-one liner form of solution is:

awk -v s1="\"" '
BEGIN{
  FS=OFS=","
}
{
  for(i=1;i<=NF;i++){
    if($i~s1){
      $i=""
    }
  }
}
1
'  Input_file

Detailed explanation:

awk -v s1="\"" '         ##Starting awk program from here and mentioning variable s1 whose value is "
BEGIN{                   ##Starting BEGIN section of this code here.
  FS=OFS=","             ##Setting field separator and output field separator as comma(,) here.
}
{
  for(i=1;i<=NF;i++){    ##Starting a for loop which traverse through all fields of current line.
    if($i~s1){           ##Checking if current field has " in it if yes then do following.
      $i=""              ##Nullifying current field value here.
    }
  }
}
1                        ##Mentioning 1 will print edited/non-edited line here.
'  Input_file            ##Mentioning Input_file name here.

score 2 · Answer 3 · answered Apr 16 '20 at 14:59

2

With Perl:

perl -p -e 's/".*?"//g' file

? forces * to be non-greedy.

Output:

1,2,,4,,,5

answered Apr 16 '20 at 14:59

Cyrus

69,405
13
65
117

Use sed (or similar) to remove anything between repeating patterns

3 Answers3