How to process dataflow two batch files simultaneously on GCP

Question

I want to process two files from gcp to dataflow at the same time simultaneously.

I think it will be possible if one more file comes in side-input.

However, in this case, I think it will be processed every time, not just once.

e.g) How to read and process file1 and file2 at the same time (do I have to put two files in one file and just follow the path?)

I'd appreciate it if you could give me a good example or advice.

Thank you.

Do you know the 2 files name before the pipeline start? – guillaume blaquiere Jan 27 '21 at 08:14 — guillaume blaquiere, Jan 27 '21 at 08:14
@guillaumeblaquiere yes, i know 2 files name. – 꽥꽥꽥 Jan 28 '21 at 08:31 — 꽥꽥꽥, Jan 28 '21 at 08:31

score 1 · Answer 1 · answered Jan 28 '21 at 11:03

If you know the 2 files from the beginning you can simply have a pipeline with 2 entry (fileIO)

I don't know your language, but by design you can do this

     PCollection1                        PCollection2
          |                                   |
   FileIO(readFile1)                   FileIO(readFile2)
          |                                   |
    Transform file                       Transform file
          |                                   |
    WriteIO(sink)                        WriteIO(sink)

You can imagine side input, flatten, group by,... all depends on your needs.

How to process dataflow two batch files simultaneously on GCP

1 Answers1