I am making a dataflow (java api) which will write protobuf object to gcs. Can anyone please help me with how to do this. A sample of code or something would be helpful.
Asked
Active
Viewed 330 times
1 Answers
1
https://issues.apache.org/jira/browse/BEAM-221 has some thoughts on this.
Wrap AvroIO in a Composite PTransform:
- Avro schema is just a single byte[].
- On read, convert from byte[] to the user's proto.
- On write, convert from proto to byte[].
In this way, ProtoIO would just be a simple wrapper for AvroIO.Read+ParDo (read) or ParDo+AvroIO.Write.
![](../../users/profiles/4392693.webp)
Sam McVeety
- 3,094
- 1
- 10
- 34