-1

I have Amazon Kinesis Streams, which has all the click stream data and we want to write an API to query Kinesis Streams.

My plan is to create a API gateway which calls an AWS Lambda function that would query Kinesis Streams and return it back.

Is it possible to use Lambda to query Kinesis Streams or should we use Kinesis Analytics and Lambda?

John Rotenstein
  • 165,783
  • 13
  • 223
  • 298
ABHILASH SOMA
  • 41
  • 3
  • i am currently using Firehose, but it spits out data to S3 at least after 1 min and i want near real time results. so i want to query streams to get real time data. – ABHILASH SOMA Apr 17 '17 at 18:09

3 Answers3

2

Your approach is not the typical way to use Amazon Kinesis Streams.

The normal situation is that data is sent to a Kinesis stream, which holds the data for 24 hours (but can be configured for up to 7 days). Your app can then extract the data from the stream and processes it -- either updating something in real-time or storing information in a database. In fact, Amazon Kinesis Firehose can automatically store incoming data into Amazon S3, Amazon Redshift or Amazon Elasticsearch Service.

Thus, streams are processed rather than queried. Just think of them as replayable queues (rather than databases).

Alternatively, you can have Amazon Kinesis Streams trigger an AWS Lambda function whenever a new message arrives in the stream. This allows near-realtime processing of incoming information.

Amazon Kinesis Streams triggering an AWS Lambda function

John Rotenstein
  • 165,783
  • 13
  • 223
  • 298
  • i am currently using Firehose, but it spits out data to S3 after 1 min and i want near real time results. so i want to query streams to get real time data. – ABHILASH SOMA Apr 17 '17 at 18:07
0

You can use AWS Kinesis with Lambda via the Stream-based model. AWS Lambda will itself poll the kinesis stream and fetch the number of records(configured by you) from the Kinesis.

Here's more information: http://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html

Vipin Jain
  • 1
  • 1
0

Use something like Firehose and then send it to Elasticsearch which you can then use to query the data. A Kinesis stream is like a temporary buffer to retain data in until you can do something else with it such as load it into a data store.

See my comment below you can use Kinesis Analytics for what you would like to do.

OpenBSDNinja
  • 714
  • 5
  • 15
  • i am currently using Firehose, but it spits out data to S3 after 1 min and i want near real time results. so i want to query streams to get real time data. – ABHILASH SOMA Apr 17 '17 at 18:08
  • Look into Kinesis Analytics it does what you want to do. See: https://www.slideshare.net/AmazonWebServices/introduction-to-amazon-kinesis-analytics and https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-analyzing-streaming-data-in-realtime-with-amazon-kinesis-analytics-bdm304 – OpenBSDNinja Apr 21 '17 at 06:10