0

I have the following use cases:

  1. Assume you have two micro-services one AccountManagement and ActivityReporting that processes event U.
  2. When a user registers, event U containing the user information will published into a broker for the two micro-services to process.
  3. AccountManagement, and ActivityReporting microservice are replicated across two instances each for performance and scalability reasons.
  4. Each microservice instance has a consumer listening on the broker topic. The choice of topic is so that both AccountManagement, and ActivityReporting can process U concurrently.

However, I want only one instance of AccountManagement to process event U, and one instance of ActivityReporting to process event U.

Please share your experience implementing a Consume Once per Application Group, broker system.

As this would effectively solve this problem.

F.O.O
  • 4,024
  • 4
  • 20
  • 31

2 Answers2

0

If all your consumer listeners even from different instances have the same group.id property then only one of them will receive the message. You need to set this property when you initialise the consumer. So in your case you will need one group.id for AccountManagement and another for ActivityReporting.

Katya Gorshkova
  • 1,050
  • 5
  • 12
  • Is this for kafka? Also, do you know if the producer also need to indicate if the message is consume once. As I imagine there would be case where you want all instances to get the message. Thanks – F.O.O Jun 11 '19 at 12:47
  • Yes, this is for Kafka. Producer does not affect the policy of the consuming of the messages. If you need all the instances consume the messages, each of them must have a different group.id – Katya Gorshkova Jun 11 '19 at 12:59
0

I would recommend Cadence Workflow which is much more powerful solution for microservice orchestration.

It offers a lot of advantages over using queues for your use case.

  • Built it exponential retries with unlimited expiration interval
  • Failure handling. For example it allows to execute a task that notifies another service if both updates couldn't succeed during a configured interval.
  • Support for long running heartbeating operations
  • Ability to implement complex task dependencies. For example to implement chaining of calls or compensation logic in case of unrecoverble failures (SAGA)
  • Gives complete visibility into current state of the update. For example when using queues all you know if there are some messages in a queue and you need additional DB to track the overall progress. With Cadence every event is recorded.
  • Ability to cancel an update in flight.

See the presentation that goes over Cadence programming model.

Maxim Fateev
  • 4,568
  • 2
  • 16
  • 29