-1

I am making a dataflow (java api) which will write protobuf object to gcs. Can anyone please help me with how to do this. A sample of code or something would be helpful.

Mikhail Berlyant
  • 117,385
  • 6
  • 77
  • 139
Raman
  • 172
  • 14

1 Answers1

1

https://issues.apache.org/jira/browse/BEAM-221 has some thoughts on this.

Wrap AvroIO in a Composite PTransform:

  • Avro schema is just a single byte[].
  • On read, convert from byte[] to the user's proto.
  • On write, convert from proto to byte[].

In this way, ProtoIO would just be a simple wrapper for AvroIO.Read+ParDo (read) or ParDo+AvroIO.Write.

Sam McVeety
  • 3,094
  • 1
  • 10
  • 34