4

I am using Kinesis Analytics to read in JSON from Kinesis Firehose. I am successfully filtering out some of the records and writing a subset of the JSON properties to another Firehose.

I wanted to execute an Athena query on the data being written to S3 via the destination Firehose. However, the JSON records written to the files in S3 do not have any newlines. Consequently, when I query the data using Athena, it only returns the first record in each file.

When I write records to the source Firehose, I manually insert a newline between records, but Analytics doesn't seem to do this when writing to the destination.

Is there a way to get Analytics to write out a separator or newline between records, so Athena can see all of the records?

John Rotenstein
  • 165,783
  • 13
  • 223
  • 298
Jed
  • 143
  • 1
  • 9
  • Looks like for now the only way is to add a lambda function to the firehose :( – ibizaman Jan 29 '18 at 21:43
  • 1
    you need to check the answer here. https://stackoverflow.com/questions/48226472/kinesis-firehose-putting-json-objects-in-s3-without-seperator-comma/51273983#51273983 – xichen Jul 10 '18 at 21:33

0 Answers0