6

I'm developing a website with the microservice architecture, and each of the service owns a database. The database stores the data which the microservice needs.

Post, Video services need the user information, so both of the services subscribed to the NEW_USER_EVENT.

The NEW_USER_EVENT will be triggered when there's a new user registered.

enter image description here

Once the services received the NEW_USER_EVENT, they put the incoming user information to each of their own database. So they can do things without asking the User service.

enter image description here

So far so good. But here comes the question:

  • What if I'm going to create a new service? How do I get the registered user informations and put them in the new service?

Maybe I can get the informations from the existing services. But the events are pushed by the messaging queue (NSQ).

If I'm going to copy the data from one of the microservices, how do I make sure which service has the latest user informations? (Because some services haven't received the latest event)

enter image description here

Read More:

The Hardest Part About Microservices: Your Data

Intro to Microservices, Part 4: Dependencies and Data Sharing

Yami Odymel
  • 1,495
  • 1
  • 18
  • 35
  • Why do "posts" and "videos" and other services need to know about user creation? If a request arrives related to a user who those services never heard of, they have no need to validate that the user exists... they can simply assume that the user must indeed exist, else they wouldn't have received a request related to the user. – Michael - sqlbot Jan 03 '17 at 23:41
  • @Michael The services received the `username` and they will convert it to the `user_id` without asking the `User` service for loose coupling. When the new user registered, the services will store the `username-userId` map in each of their database so they can convert it in their own service. – Yami Odymel Jan 04 '17 at 10:34
  • I know that this problem can be solved by using JWT, but I still have similar things not just about the user needs to be convert. – Yami Odymel Jan 04 '17 at 10:36
  • I don't think there's a need to have the user data in the new service - when you say there's a new piece of music (for instance), you'll post a "user ID_HERE created some music", and just store that ID. The service doesn't need to know anything other than the ID, does it? It could respond to a "User Disabled" event and archive them all potentially - though even that may not be required. Basically, am not sure why this service cares about usernames at all – Kieren Johnstone Jan 05 '17 at 07:55
  • Great diagrams by the way, what tool did you use Yami? – Kieren Johnstone Jan 05 '17 at 07:55
  • @Kieren Johnstone I use the `username` because that would be easier to read on the URL (*and people can remember it easily*). If I don't store anything about the user then I cannot tell if the user does exist or not in the service. – Yami Odymel Jan 05 '17 at 08:16
  • The diagrams were drew by using Sketch on macOS, you can get the same result with https://draw.io I guess. – Yami Odymel Jan 05 '17 at 08:16
  • @Yami - but in a Microservices architecture, wouldn't the Music service only deal with music, not building URLs, serving pages etc - as I understand, uServices are almost never exposed to the outside world, there's a public service layer sits atop them to build such things? In either case, this seems a very good use-case for such a wrapping service - the "public music service", which combines the user service and music service. uServices should probably be Bounded Contexts. The ID tells you the last part - if they exist or not. If the service is called with an ID, the user exists (1/2) – Kieren Johnstone Jan 05 '17 at 09:25
  • (2/2) - this is because the service is called by your own wrapping service which knows the user exists, or some public API service which validates the ID. If the user is deleted, the services would then be notified individually by whichever API service initiates the delete. Another idea - have a "URL generation service" which in turn uses the User Service to get the username. – Kieren Johnstone Jan 05 '17 at 09:27
  • @Kieren Johnstone I put the services behind the gateway, and the gateway **doesn't** chain the services (though the gateway has an `authorization layer`). If I send the `POST /music` request to the gateway, the gateway will call to the `Music` service, and that's it, no other services will be called. If I chain the services it would be complex and highly coupling. (https://groups.google.com/forum/#!topic/microservices/Ra84t2Xmyj4) – Yami Odymel Jan 05 '17 at 11:07
  • I don't think it would be - a software component can use many (lots!) of components without it being "coupling". Check out some talks about Monzo (then Mondo) and Microservices - one API call can orchestrate tens or hundreds of microservice requests or operations. Yes, it's complex - that is a (necessary) disadvantage of adhering to the microservices ideals! – Kieren Johnstone Jan 05 '17 at 11:44
  • (See here for a good example - http://microservices.io/patterns/apigateway.html) – Kieren Johnstone Jan 05 '17 at 11:45

1 Answers1

3

What if I'm going to create a new service? How do I get the registered user informations and put them in the new service?

You have to replay all events to which this new service is subscribed from the beginning of time (you should have an "event store" that keeps all events already happened in your application). Also, you can put a bit smarter logic when replaying events by starting from the most recent ones and going back in time. In this way, you will be able to restore most valuable data first. Just be careful to handle interdependent events correctly.

Data source: The events are pushed by the messaging queue(NSQ), If I'm going to copy the data from one of the microservices, how do I make sure the copy source has the latest user informations?

You are not talking about doing backups, right?

Aside from backups, in the event-driven systems people usually don't copy the data in a classical way, row by row. Instead, they just replaying events from event store from the beginning of time and feeding those events to the event handlers for the new service (or new instance). As a result, new service eventually becomes consistent with the other parts of the system.

IlliakaillI
  • 1,242
  • 11
  • 22
  • Just to be clear, you're talking about Event Sourcing I believe, which isn't necessarily tied to a microservices architecture? – Kieren Johnstone Jan 05 '17 at 07:57
  • You are correct, Yami pointed out that he is using NEW_USER_EVENT to synchronize data between services which implies usage of event sourcing techniques. – IlliakaillI Jan 05 '17 at 08:03
  • I might used the wrong way to described my last question, but your answer still helps me understand some extra things. I was talking about: Maybe I can get the registered user informations from the existing services, but since I'm using `Messaging Queue` to sync the user informations, I cannot make sure which service has the latest user informations (*Because some services haven't received the `NEW_USER_EVENT` event yet*). – Yami Odymel Jan 05 '17 at 08:05
  • If I'm going to create a event store, I still cannot make sure the event store has the latest events (*If `Event store = A microservice which subscribed to all the events`*). Or am I just overthinking about this, because `Messaging Queue` isn't that **slow**? – Yami Odymel Jan 05 '17 at 08:08
  • 1
    With Event Store it is guaranteed that it has the latest version. The reason is that all services should save events to event store directly and synchroniously. After that, some background worker may publish events to a bus if required. Some event stores have pub/sub built-in mechanisms already, so you don't even need a queue. This is actually a preferred way of doing ES. This one is pretty good: https://geteventstore.com/ – IlliakaillI Jan 05 '17 at 08:18
  • An event-driven architecture does not imply the usage of event-sourcing techniques. A service-bus is not (And should not) be used as an event store. – Orestis P. Jan 28 '19 at 13:17