Cygnus-ngsi installed with Docker-compose does not save data in MongoDB

Question

I have Orion, MongoDB and Cygnus-ngsi installed with docker-compose on an Ubuntu 18.04 machine. The images I used were: fiware/orion:latest, fiware/cygnus-ngsi:latest and mongo:3.6.

They were installed with the command: docker-compose -f reflexwaterDocker.yaml up

I had no installation problems. All containers are UP. Orion saving to MongoDB. The Cygnus API working.

I created a subscription to an Orion entity, all ok. But when I update the entity, the orion performs on the mongo, but Cygnus does not persist the historical data.

The agent.conf file in opt/apache-flume/conf is configured correctly. I am using the default setup for the docker-compose installation.

I followed the process described in: https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/cygnus-ngsi/installation_and_administration_guide/install_with_docker.md

All ports are correct and have been tested.

I've used Cygnus with Orion and MongoDB manually installing, and I had no problem. But also, there I set up an agent_conf and a cygnus_instance.conf. Using the docker, following the step by step in the documentation it is not necessary to configure cygnus_instance.conf. It doesn't even exist inside the opt/apache-flume/conf folder.

Does anyone have any idea what might be happening to Cygnus not persisting the data? Or have you been through this and managed to solve it?

My docker-compose looks like this for Cygnus-ngsi:

    image: fiware/cygnus-ngsi:latest
    hostname: cygnus
    container_name: cygnus
    networks:
      - default
    depends_on:
      - mongo
    expose:
      - "5050"
      - "5080"
    ports:
      - "5050:5050"
      - "5080:5080" 
    environment:
      - CYGNUS_SERVICE_PORT=5050
      - CYGNUS_AGENT_NAME=cygnus-ngsi
      - CYGNUS_MONGO_SERVICE_PORT=5050
      - CYGNUS_DEFAULT_SERVICE=def_serv 
      - CYGNUS_DEFAULT_SERVICE_PATH=reflexWater
      - CYGNUS_MONGO_HOSTS=localhost:27017
      - CYGNUS_MONGO_USER=""
      - CYGNUS_MONGO_PASS=""
      - CYGNUS_MONGO_ENABLE_ENCODING=false
      - CYGNUS_MONGO_ENABLE_GROUPING=false
      - CYGNUS_MONGO_ENABLE_NAME_MAPPINGS=false
      - CYGNUS_MONGO_DATA_MODEL=dm-by-entity
      - CYGNUS_MONGO_ATTR_PERSISTENCE=column
      - CYGNUS_MONGO_DB_PREFIX=db_
      - CYGNUS_MONGO_COLLECTION_PREFIX=col_   
      - CYGNUS_MONGO_ENABLE_LOWERCASE=false
      - CYGNUS_MONGO_BATCH_TIMEOUT=30
      - CYGNUS_MONGO_BATCH_TTL=10
      - CYGNUS_MONGO_DATA_EXPIRATION=0
      - CYGNUS_MONGO_COLLECTIONS_SIZE=0
      - CYGNUS_LOG_LEVEL=DEBUG
      - CYGNUS_SKIP_CONF_GENERATION=false

I ran the command: docker logs cygnus

Cygnus tries to persist the data as per the log: time=2020-05-03T01:18:27.683Z | lvl=INFO | corr=fda0c4fc-8cdb-11ea-ad11-0242ac120003 | trans=f1674faf-aa5a-4ca8-a62a-2136378f6d08 | srv=default |subsrv=/ | comp=cygnus-ngsi | op=persistAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSIMongoSink[235] : [mongo-sink] Persisting data at NGSIMongoSink. Database: db_default, Collection: col_/_Room1_Room, Data: [Document{{temperature=279, recvTime=Sun May 03 01:18:27 UTC 2020}}]

But I also noticed the following error:

time=2020-05-03T01:18:28.553Z| lvl=WARN|corr=fda0c4fc-8cdb-11ea-ad11-0242ac120003 |trans=f1674faf-aa5a-4ca8-a62a-2136378f6d08 | srv=default| subsrv=/ | comp=cygnus-ngsi | op=createCollection|msg=com.telefonica.iot.cygnus.backends.mongo.MongoBackendImpl[192] : Error in collection col_/_Room1_Room creating index ex=Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName='""', source='db_default', password=<hidden>, mechanismProperties=<hidden>}

As I understand it, it doesn't persist because of MongoDB's credentials.. But I am not using Username and Password for the MongoDB bank.

So, what can it be? I didn't understand the error and I couldn't even find a plausible justification for the error.

Someone who has already used cygnus on the docker and who can give me some guidance on how to solve this.

Thank you

You say "Using the docker, following the step by step in the documentation...". To which documentation do you refer? It would be great if you could edit your question post and include a link to such documentation. Thanks! — fgalan, May 05 '20 at 20:38

score 2 · Accepted Answer · answered May 13 '20 at 12:12

Taking into account this

I've used Cygnus with Orion and MongoDB manually installing, and I had no problem

I'd say that the problem is somehow related to the docker deployment. Moreover, looking to the trace:

Exception authenticating MongoCredential

maybe the CYGNUS_MONGO_USER="" and CYGNUS_MONGO_PASS="" env vars are not correctly processed (by entrypoint.sh or similar script, I guess) and this is causing the problem.

In order to debug this, I'd suggest you to compare the Cygnus agents.conf/cygnus_instance.conf files in working case (manually installing) and in the failing case (docker). Maybe the differences could provide some insight on the problem. In the case of docker, maybe the files aren't at opt/apache-flume/conf but they should be in someplace (either internally in the container or mounted as volume in the hosting system).

Another related question: the Cygnus version you make to work manually is exactly the same you one in the docker case? Try to ensure you are using exactly the same software.

Hi fgalan, I made the comparison and the error was precisely in my environment variables ```CYGNUS_MONGO_USER = ""``` and ```CYGNUS_MONGO_PASS = ""```. The correct one was: ```CYGNUS_MONGO_USER =``` and ```CYGNUS_MONGO_PASS =``` I removed the double quotes and it worked. Thank you! — Rosiberto, May 15 '20 at 23:34
I'm happy to know you finally found the solution :) So, if you find my answer correct, please upvote it and mark it as correct. This is not to feed my ego ;) but to help other users, letting them to know that the same answer can be useful for them. Thanks! — fgalan, May 17 '20 at 13:04

Cygnus-ngsi installed with Docker-compose does not save data in MongoDB

1 Answers1