Is there a way to read a multi-line csv file in Apache Beam using the ReadFromText transform (Python)?

Question

Is there a way to read a multi-line csv file using the ReadFromText transform in Python? I have a file that contains one line I am trying to make Apache Beam read the input as one line, but cannot get it to work.

def print_each_line(line):
    print line

path = './input/testfile.csv'
# Here are the contents of testfile.csv
# foo,bar,"blah blah
# more blah blah",baz

p = apache_beam.Pipeline()

(p
 | 'ReadFromFile' >> apache_beam.io.ReadFromText(path)
 | 'PrintEachLine' >> apache_beam.FlatMap(lambda line: print_each_line(line))
 )

# Here is the output:
# foo,bar,"blah blah
# more blah blah",baz

The above code parses the input as two lines even though the standard for multi-line csv files is to wrap multi-line elements within double-quotes.

You need a PCollection with only one line in it. Am I right? — Arjun Kay, Apr 20 '18 at 04:14
@ArjunKay Yes, currently the input I have is one line, but beam treats it as two — Brandon, Apr 20 '18 at 17:22

score 2 · Answer 1 · answered Apr 20 '18 at 23:21

Beam doesn't support parsing CSV files. You can however use Python's csv.reader. Here's an example:

import apache_beam
import csv

def print_each_line(line):
  print line

p = apache_beam.Pipeline()

(p 
 | apache_beam.Create(["test.csv"])
 | apache_beam.FlatMap(lambda filename:
     csv.reader(apache_beam.io.filesystems.FileSystems.open(filename)))
 | apache_beam.FlatMap(print_each_line))

p.run()

Output:

['foo', 'bar', 'blah blah\nmore blah blah', 'baz']

Arjun Kay · Answer 2 · 2018-04-23T04:41:03.767

0

ReadFromText parses a text file as newline-delimited elements. So ReadFromText treats two lines as two elements. If you would like to have the contents of the file as a single element, you could do the following:

contents = []
contents.append(open(path).read()) 
p = apache_beam.Pipeline()
p | beam.Create(contents)

edited Apr 23 '18 at 04:41

answered Apr 22 '18 at 05:24

Arjun Kay

268
1
11

score 0 · Answer 3 · answered Nov 08 '20 at 05:17

None of the answers worked for me but this did

(
  p
  | beam.Create(['data/test.csv'])
  | beam.FlatMap(lambda filename:
    csv.reader(io.TextIOWrapper(beam.io.filesystems.FileSystems.open(known_args.input)))
  | "Take only name" >> beam.Map(lambda x: x[0])
  | WriteToText(known_args.output)
)

Is there a way to read a multi-line csv file in Apache Beam using the ReadFromText transform (Python)?

3 Answers3