I have a pipeline with one gcs file as input and generate two gcs output files.
One output file contains error info and another contains normal info.
And I have a cloud function with the gcs trigger of the two output files.
I want to do something with the normal info file only when the error info file is 0 byte.
So I must let the error info file be generated earlier than the normal info file to check the size of the error info file.
Now I use 2 TextIO.Write to generate the two files.
But I can not control which one is generated first.
In the cloud functions, I let the normal info file check the size of the error info file with a retry.
But the cloud functions as a timeout limit of 540s so I can not retry until the error info file is generated.
How can I handle this in Cloud Dataflow?
Can I generate the error info file before the normal info file programmatically?