1

trying to learn nsq, and following the examples from here golang example and here nsqjs. I am sending messages in server side doing w/ a for loop and go routines

var wg sync.WaitGroup
for i := 0; i < 100; i++ {
    wg.Add(1)
    go func(x int) {
        defer wg.Done()

        chanName := fmt.Sprintf("import_progress_587e6442ff74889098498f6e")
        m := map[string]interface{}{
            "body": map[string]interface{}{
                "progress": x,
            },
        }
        msg, _ := json.Marshal(m)

        req := NSQPubReq{
            Topic: chanName,
            Body:  msg,
        }
        if err := producer.Publish(req.Topic, req.Body); err != nil {
        }
        utils.Info(fmt.Sprintf("sent msg=%v", string(msg)))

    }(i)
}

wg.Wait()

but the problem is, on the clientside. 

// channel = 'import_progress_587e6442ff74889098498f6e'
let reader = new nsq.Reader(channel, channel, {
    //lookupdHTTPAddresses: '<<IP>>:4161',
    maxInFlight: 10000,
    snappy: true
})
reader.connect()

reader.on('message', (msg) => {
    var msgData = {
            id:     msg.id,
            body:   msg.body.toString(),
            chan:   channel
    }
    io.emit(channel, msgData)
    msg.finish()
})

the message don't come up immediately to the client. i will wait for a couple of seconds until the message come to the nodejs client. is there any settings that I need to do? thank you!

Hokutosei
  • 1,773
  • 4
  • 19
  • 40
  • What's the `maxInFlight` set to? – Oliver Jan 18 '17 at 17:52
  • Also, be aware that if you're exposing the IP of your `nsqd` host and it's open to the public, then anyone can connect to it and e.g. delete topics and channels. – Oliver Jan 18 '17 at 17:53
  • @Oliver where should I set maxinflight? I tried setting it in server side using `config.Set` w/ value of 10000.. thank you. Yeah I forgot to delete the IP, thank you for the edit – Hokutosei Jan 19 '17 at 00:08
  • Try setting it to `1`. – Oliver Jan 19 '17 at 03:16

1 Answers1

4

There are a couple of reasons why a nsqjs client will be slow to receive a message just published:

  1. If the topic is new and the discovery of topics is via nsqlookupd, then by default, the nsqjs Reader will attempt to discover new topics every 30 seconds.

    From the example above, it looks like you are creating new topics for every import. I believe that if you start publishing messages first from the Golang client and then start the nsqjs client, then you shouldn't see the delay.

  2. If you have multiple nsqds with a max-in-flight set too low, then it puts the nsqjs Reader in a starvation mode where it moves the RDY count between nsqds for a set period of time.

    I'm not sure that's what's going on here since I can't tell anything about the nsq topology. As long as your max-in-flight is set higher than the number of nsqd instances that you have, then you'll be in good shape.

Community
  • 1
  • 1
boxcarr
  • 41
  • 2
  • thank you for the explanation. I have tried connecting directly to nsqds as I am planning to deploy it to k8s, i can just use services instead of `nslookupd`. for development purposes, on a 4core 32gb ram, both client and server `maxInFlight` set to more than 10k and has 1 nsqd as example. the delay improved a bit, but not as quick as realtime.. I will update my post.. – Hokutosei Jan 19 '17 at 17:04
  • As an addendum: you really don't want to create a topic (or channel) for "each import" - you want a mostly static set of topics and channels. If a consumer is already connected to a topic/channel on an nsqd, it will receive new messages very quickly. Try using a single topic/channel for all of these messages e.g. topic="import_progress". A consumer can filter out messages it is not interested in reasonably efficiently. – Pierce Jan 25 '17 at 11:40