NSQ Re-Queue half Message

Question

I have an array of Ids of type int64 And this is my Nsq Message that I am trying to publish.

nsqMsg := st{
    Action  :     "insert",
    Ids     :     Ids
    GID     :     Gids
}

msg, err := json.Marshal(nsqMsg)
if err != nil {
    log.Println(err)
    return err
}
err = nsqProducer.Publish(TOPIC-NAME, msg)
if err != nil {
    log.Println(err)
    return err
}

While in my consumer I am taking each Id one by one and fetching an info based on my Id from my datastore.

So while fetching there can be a case if my CreateObject method returns an error so I handle that case by requeue the msg (which is giving the error) and so it can be retried.

for i := 0; i < len(data.Ids); i++ {

    Object, err := X.CreateObject(data.Ids[i)
        if err != nil {
            requeueMsgData = append(requeueMsgData, data.Ids[i])
            continue
        }
        DataList = append(DataList, Object)
    }

    if len(requeueMsgData) > 0 {
        msg, err := json.Marshal(requeueMsgData)
        if err != nil {
            log.Println(err)
            return err
        }
        message.Body = msg
        message.Requeue(60 * time.Second)
        log.Println("error while creating Object", err)
        return n
}

So, is this the right way of doing this? Is their any drawback of this case? Is it better to publish it again?

score 1 · Answer 1 · answered Sep 08 '18 at 19:10

Some queues (like Kafka) support acknowledgement where items that are dequeued are not removed from the queue until the consumer has actually acknowledged successful receipt of the item.

The advantage of this model is that if the consumer dies after consumption but before acknowledgement, the item will be automatically re-queued. The downside of your model is that the item might be lost in that case.

The risk of an acknowledgment model is that items could now be double consumed. Where a consumer attempts consumption that has side-effects (like incrementing a counter or mutating a database) but doesn't acknowledge so retries might not create the desired result. (note that reading through the nsq docs, retries are not guaranteed to happen even if you don't re-enqueue the data so your code will likely have to be defensive against this anyway).

You should look into the topic of "Exactly Once" vs. "At Most Once" processing if you want to understand this deeper.

Reading through the nsq docs, it doesn't look like acknowledgement is supported so this might be the best option you have if you are obligated to use nsq.

dm03514 · Answer 2 · 2018-09-08T21:43:02.020

Along the lines with what dolan was saying there are a couple of cases that you could encounter:

main message heartbeat/lease times out and you receive ALL ids again (from the original message). NSQ provides "at least once" semantics.
Requeue of any single message times out and is never complete (fallback to the main IDS)

Because nsq can (and most def will :p) deliver messages more than once CreateObjects could/should be idempotent in order to handle this case.

Additionally the redelivery is an important safety mechanism, The original message shouldn’t be fin’d until all individual ids or confirmed created or successfully requeued, which ensures that no data is lost.

IMO the way you are handling it looks perfectly good, but the most important considerations IMO are handling correctness/data integrity in an environment where duplicate messages will be received.

Another option could be to batch the Requeue so that it attempts to produce a single output message of failed ids, which could cut back on the # of messages in the queue at any given time:

Consider a message with 3 ids:

message ids: [id1, id2, id3]

id1 succeeds creation and id2 and id3 fail:

the program could attempt all operations and emit a single requeue message, with id2, id3.

But trade offs with this too.

NSQ Re-Queue half Message

2 Answers2