0

I do not care much about the order of events but I would like the message to be processed exactly once. The lambda listening to SQS messages will store it in DynamoDB so throughput is pretty important as I have multiple microservices (as producers) writing messages to this SQS that will be read by a single microservice.

About processing messages exactly once, that is something that FIFO queue supports but is said to have not a good throughput.

Is the throughput of the FIFO queue the same as the Standard queue if each message has a unique groupId?

If not, my next option is probably to use "attribute_not_exists" in DynamoDB while storing the message.

Which of these should work better?

S Khurana
  • 3,003
  • 8
  • 35
  • 87
  • How many messages do you expect per second? FIFO can handle 3000/s. – Marcin Mar 30 '21 at 00:02
  • 1
    I would say that implementing dedupe logic at the destination (DynamoDB in your case) is the better choice In terms of cost, performance and reliability. – Mike Dinescu Mar 31 '21 at 00:28

2 Answers2

1

FIFO SQS queues have different rate limits than a regular SQS queue regardless of the use of message group ids

SQS supports 3000 Transactions Per Second (TPS) for each API method (send/receive/delete)

FIFO SQS supports 300 TPS for each API method

Look at the quota docs here

Also, AWS has a new feature for higher throughput FIFO SQS queue which might interest you

With batching of maximum 10 messages per API call you can handle 3000 messages per second with FIFO queue

Regarding making sure you don't handle the same message twice - have you had a look at FIFO de-duplication ID? I am not sure if that's exactly what you need but it sounds pretty similar to your requirement

Erez Rabih
  • 14,472
  • 3
  • 37
  • 59
0

SQS delivery guarantee is at least once. Your application must be designed to handle processing duplicate messages.

I'd strongly recommend building your application this way.

If you must process some type of data exactly once, you need a strongly consistent system. Consider using dynamodb and conditional updates

Aaron Stuyvenberg
  • 1,563
  • 3
  • 13