Stack Overflow
Stack Overflow

Questions tagged [nsq]

NSQ is a realtime message processing system designed to operate at bitly’s scale, handling billions of messages per day.

NSQ is a realtime message processing system designed to operate at bitly’s scale, handling billions of messages per day.

It promotes distributed and decentralized topologies without single points of failure, enabling fault tolerance and high availability coupled with a reliable message delivery guarantee.

Operationally, NSQ is easy to configure and deploy (all parameters are specified on the command line and compiled binaries have no runtime dependencies). For maximum flexibility, it is agnostic to data format (messages can be JSON, MsgPack, Protocol Buffers, or anything else). Go and Python libraries are available out of the box.

Questions can also be posted to the nsq-users Google group.

Bugs can be filed on GitHub.

55 questions
7
votes
1 answer

Distributed Systems NSQ topology pattern on Docker containers

Is it possible to replicate "NSQ realtime distributed messaging platform" described in the last example of "Topology Patterns" with Docker? Does anybody have a dockerfile or image example?
Luca G. Soave
  • 11,195
  • 10
  • 55
  • 105
6
votes
1 answer

How to sync the database with the microservices (and the new one)?

I'm developing a website with the microservice architecture, and each of the service owns a database. The database stores the data which the microservice needs. Post, Video services need the user information, so both of the services subscribed to…
Yami Odymel
  • 1,495
  • 1
  • 18
  • 35
4
votes
1 answer

Golang and NSQ (bitly)

So kind of a noob here but I can't seem to find any examples or help anywhere. I have a working instance of NSQ, can register tasks, consumers, etc. Unfortunately I did not originally set the system up. What I am trying to do is figure out how to…
Mark Hayden
  • 485
  • 4
  • 13
3
votes
2 answers

NSQ Docker Swarm

I'm trying to use NSQ in Docker Swarm without success mhlg/rpi-nsq is a Docker image built for the Raspberry Pi ARM7 board and I can confirm is working correctly if run as a normal Docker container Running NSQ in Docker (OK) # crete a bridged…
Gavello
  • 1,069
  • 2
  • 11
  • 21
3
votes
2 answers

What's the max size of a NSQ message?

I'm evaluating using NSQ, http://nsq.io/, for a specific project. The idea is to setup a data pipeline where each step is a job, and where the state ideally will be located in the message body. Which got me to think about a potential maximal message…
Niklas9
  • 7,378
  • 7
  • 31
  • 54
3
votes
0 answers

NSQ trendrr : Is it possible to get the queue size with client?

I'm using this classes at the moment: com.trendrr.nsq.NSQProducer com.trendrr.nsq.NSQConsumer My service needs to know how many messages are queued at the moment, because there is some "when it will be done" estimation method. Does someone knows…
heaphach
  • 1,432
  • 1
  • 18
  • 41
3
votes
1 answer

How to use nsq as a broker for the ELK stack (ElasticSearch+LogStash+Kibana)

I am trying to use nsq as a broker for the ELK stack. I was thinking of doing this by having nsq send data to elasticsearch via the REST api. According to that doc, I can make an HTTP PUT request to add data to elastic search. But, according the…
rexposadas
  • 2,689
  • 3
  • 24
  • 46
2
votes
0 answers

NSQ cluster in Kubernetes

I'm trying to set up an NSQ cluster in Kubernetes and having issues. Basically, I want to scale out NSQ and NSQ Lookup. I have a stateful set(2 nodes) definition for both of them. To not post the whole YAML file, I'll post only part of it for…
Andrej Kovalksy
  • 185
  • 1
  • 1
  • 12
2
votes
0 answers

NSQ cannot get the message

I look this:nsq cannot consume message by connecting to nsqlookupd But he doesn't apply to me,All sorts of ways have been tried.It could be the environment. system: VMware CentOS + Docker-compose NSQ Version: all latest docker-compose.yml: version:…
Jiayu.Wang
  • 21
  • 2
2
votes
1 answer

nsq cannot consume message by connecting to nsqlookupd

I tried to use docker-compose to run nsq, the docker-compose.yml as below: version: '3' services: nsqlookupd: image: nsqio/nsq command: /nsqlookupd ports: - "4160:4160" - "4161:4161" nsqd: image: nsqio/nsq …
pangpang
  • 7,603
  • 8
  • 46
  • 90
2
votes
1 answer

Should all independent tasks be processed outside the http request goroutine?

Since golang handles incoming requests in separate goroutines, it's unclear to me which types of tasks should be deferred for processing by a message queue e.g. NSQ consumers and which should be handled within the http request goroutine.
paulkon
  • 1,587
  • 2
  • 18
  • 32
1
vote
1 answer

Inject dynamically commands into a deployment

When you have a StatefulSet or a Deployment you usually have means to provide commands that will be fed into a pod's container on its creation. Is it possible, more arguments to be injected into the commands array as the deployment autoscales with…
KDX2
  • 727
  • 2
  • 7
  • 25
1
vote
1 answer

How to get docker generated ip port from docker-compose

docker networking seems not working, docker supposed to know the address and port when specifying the container name, or what am I missing? Would it be possible to get the generated ip port like: docker-compose.yml command: nsqd …
swoopy
  • 99
  • 12
1
vote
0 answers

How to connect/listen to nsqd from a webserver

So I have a simple flask server and wanted to listen to nsqd,(My nsqd and nsqlookupd dockers are running fine.) I'm using gnsq from python as a tool to achieve that server.py: from flask import Flask import gnsq consumer =…
swoopy
  • 99
  • 12
1
vote
0 answers

How to write an Integration test for API with Message Queue?

Consider that I have an API end-point http://localhost/add-page-view. When we send a post request with data {url: "google.com"} it will be sent to a message queue, then later will be inserted to the DB by the queue subscriber I need to write a test…
Gijo Varghese
  • 8,506
  • 17
  • 61
  • 104
1
2 3 4