3

So i have read all the documentation and followed the tutorial on MySQL persistence but i can't still presist any kind of data on MySQL database.

Even though i'm puting the presistence mode = row it doesn't create any database nor table.

What am i doing wrong?

My Subscription: 

python2.7 SetSubscription.py bustest4 http://localhost:5050/notify

Output:

* Asking to http://localhost:1026/v1/subscribeContext
* Headers: {'Fiware-Service': 'fiwaretestapi', 'content-type': 'application/json                                                                                        ', 'accept': 'application/json', 'X-Auth-Token': 'NULL'}
* Sending PAYLOAD:
{
    "reference": "http://localhost:5050/notify",
    "throttling": "PT5S",
    "entities": [
        {
            "type": "",
            "id": "bustest4",
            "isPattern": "false"
        }
    ],
    "attributes": [
        "temperature"
    ],
    "duration": "P1M",
    "notifyConditions": [
        {
            "condValues": [
                "temperature"
            ],
            "type": "ONCHANGE"
        }
    ]
}

...

* Status Code: 200

{
  "subscribeResponse" : {
    "subscriptionId" : "5567332298add18cc3e183ac",
    "duration" : "P1M",
    "throttling" : "PT5S"
  }
}

My agent_a1.conf:

#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel mysql-channel ckan-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler = es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default service (service semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service = fiwaretestapi
# Default service path (service path semantic depends on the persistence sink)
cygnusagent.sources.http-source.handler.default_service_path = /
# Number of channel re-injection retries before a Flume event is definitely discarded (-1 means infinite retries)
cygnusagent.sources.http-source.handler.events_ttl = 10
# Source interceptors, do not change
cygnusagent.sources.http-source.interceptors = ts de
# Timestamp interceptor, do not change
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
# Destination extractor interceptor, do not change
cygnusagent.sources.http-source.interceptors.de.type = es.tid.fiware.fiwareconnectors.cygnus.interceptors.DestinationExtractor$Builder
# Matching table for the destination extractor interceptor, put the right absolute path to the file if necessary
# See the doc/design/interceptors document for more details
cygnusagent.sources.http-source.interceptors.de.matching_table = /usr/cygnus/conf/matching_table.conf

# ============================================
# OrionMySQLSink configuration
# channel name from where to read notification events
cygnusagent.sinks.mysql-sink.channel = mysql-channel
# sink class, must not be changed
cygnusagent.sinks.mysql-sink.type = es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink
# the FQDN/IP address where the MySQL server runs
cygnusagent.sinks.mysql-sink.mysql_host = localhost
# the port where the MySQL server listes for incomming connections
cygnusagent.sinks.mysql-sink.mysql_port = 3306
# a valid user in the MySQL server
cygnusagent.sinks.mysql-sink.mysql_username = root
# password for the user above
cygnusagent.sinks.mysql-sink.mysql_password = ***********
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.mysql-sink.attr_persistence = row

#=============================================
# mysql-channel configuration
# channel type (must not be changed)
cygnusagent.channels.mysql-channel.type = memory
# capacity of the channel
cygnusagent.channels.mysql-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.mysql-channel.transactionCapacity = 100

My cygnus_instance_c1.conf:

# Who to run cygnus as. Note that you may need to use root if you want
# to run cygnus in a privileged port (<1024)
CYGNUS_USER=root

# Where is the config folder
CONFIG_FOLDER=/usr/cygnus/conf

# Which is the config file
CONFIG_FILE=/usr/cygnus/conf/agent_a1.conf

# Name of the agent. The name of the agent is not trivial, since it is the base for the Flume parameters
# naming conventions, e.g. it appears in .sources.http-source.channels=...
AGENT_NAME=cygnususer

# Name of the logfile located at /var/log/cygnus. It is important to put the extension '.log' in order to the log rotation works properly
LOGFILE_NAME=cygnus.log

# Administration port. Must be unique per instance
ADMIN_PORT=8081

My cygnus.log:

Info: Sourcing environment configuration script /usr/cygnus/conf/flume-env.sh
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.log.file=cygnus.log -cp '/usr/cygnus/conf:/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/libext/*' -Djava.library.path= es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication -p 8081 -f /usr/cygnus/conf/agent_a1.conf -n root
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.7.1-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

EDIT:

So after some changes i got the log file to work and i found out that the 8081 port was already in use. Can you explain me what the ADMIN_PORT is used for, what port is it recommended to be?

LOG FILE:

02 Jun 2015 05:16:40,680 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/usr/cygnus/conf/agent_a1.conf
02 Jun 2015 05:16:40,685 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,686 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: mysql-sink Agent: cygnusagent
02 Jun 2015 05:16:40,686 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,687 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:mysql-sink
02 Jun 2015 05:16:40,692 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [cygnusagent]
02 Jun 2015 05:16:40,692 WARN  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:138)  - No configuration found for this host:root
02 Jun 2015 05:16:40,693 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101)  - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,693 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,694 INFO  [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85)  - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,695 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.stopAllComponents:101)  - Shutting down configuration: { sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
02 Jun 2015 05:16:40,695 INFO  [conf-file-poller-0] (es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusApplication.startManagementInterface:85)  - Starting a Jetty server listening on port 8081 (Management Interface)
02 Jun 2015 05:16:40,696 INFO  [Thread-26] (org.mortbay.log.Slf4jLog.info:67)  - jetty-6.1.26
02 Jun 2015 05:16:40,704 WARN  [Thread-26] (org.mortbay.log.Slf4jLog.warn:76)  - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 WARN  [Thread-26] (org.mortbay.log.Slf4jLog.warn:76)  - failed Server@4f1b95f: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,704 FATAL [Thread-26] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63)  - Fatal error running the Management Interface. Details=Address already in use
02 Jun 2015 05:16:40,705 INFO  [Thread-27] (org.mortbay.log.Slf4jLog.info:67)  - jetty-6.1.26
02 Jun 2015 05:16:40,709 WARN  [Thread-27] (org.mortbay.log.Slf4jLog.warn:76)  - failed SocketConnector@0.0.0.0:8081: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 WARN  [Thread-27] (org.mortbay.log.Slf4jLog.warn:76)  - failed Server@ed4c222: java.net.BindException: Address already in use
02 Jun 2015 05:16:40,709 FATAL [Thread-27] (es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer.run:63)  - Fatal error running the Management Interface. Details=Address already in use

EDIT 2:

My script that updates entity on Context Broker:

BASE_URL = 'http://localhost:1026'
UPDATE_URL = BASE_URL+'/ngsi10/updateContext'

HEADERS = {
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'Fiware-Service' : 'fiwaretestapi',
    'Fiware-ServicePath': '/'
}


UPDATE_EXAMPLE = {
    "contextElements": [
        {
            "type": "",
            "isPattern": "false",
            "id": "bustest4",
            "attributes": [
            {
                "name": "temperature",
                "type": "int",
                "value": "99"
            }
            ]
        }
    ],
    "updateAction": "APPEND"
}

def post(url, data):
    """"""
    req = urllib2.Request(url, data, HEADERS)
    f = urllib2.urlopen(req)
    result = json.loads(f.read())
    f.close()
    return result

if __name__ == "__main__":
    print post(UPDATE_URL, json.dumps(UPDATE_EXAMPLE))

EDIT3:

Even though i set the admin port to be 8085 on the cygnus agent configuration it tries to bind to the 8081, is that normal?

Here are the logs from the cygnus:

time=2015-06-12T05:56:39.820EDT | lvl=INFO | trans= | function=start | comp=Cygnu                                                                                       s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty                                                                                       pe: CHANNEL, name: mysql-channel started
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents                                                                                        | comp=Cygnus | msg=org.apache.flume.node.Application[173] : Starting Sink mysql                                                                                       -sink
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startAllComponents                                                                                        | comp=Cygnus | msg=org.apache.flume.node.Application[184] : Starting Source htt                                                                                       p-source
time=2015-06-12T05:56:39.821EDT | lvl=INFO | trans= | function=startManagementInt                                                                                       erface | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.nodes.CygnusAppl                                                                                       ication[85] : Starting a Jetty server listening on port 8081 (Management Interfac                                                                                       e)
time=2015-06-12T05:56:39.822EDT | lvl=INFO | trans= | function=start | comp=Cygnu                                                                                       s | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionMySQLSink[151] : [mysql-                                                                                       sink] Startup completed
time=2015-06-12T05:56:39.823EDT | lvl=INFO | trans= | function=info | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.824EDT | lvl=INFO | trans= | function=info | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[67] : jetty-6.1.26
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=info | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[67] : Started SocketConnector@0.0.0.0:5050
time=2015-06-12T05:56:39.825EDT | lvl=INFO | trans= | function=start | comp=Cygnu                                                                                       s | msg=org.apache.flume.instrumentation.MonitoredCounterGroup[94] : Component ty                                                                                       pe: SOURCE, name: http-source started
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[76] : failed SocketConnector@0.0.0.0:8081: java.n                                                                                       et.BindException: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=WARN | trans= | function=warn | comp=Cygnus                                                                                        | msg=org.mortbay.log.Slf4jLog[76] : failed Server@1c9c5521: java.net.BindExcept                                                                                       ion: Address already in use
time=2015-06-12T05:56:39.827EDT | lvl=FATAL | trans= | function=run | comp=Cygnus                                                                                        | msg=es.tid.fiware.fiwareconnectors.cygnus.http.JettyServer[63] : Fatal error r                                                                                       unning the Management Interface. Details=Address already in use

Log when i make a subscription:

time=2015-06-12T06:03:56.529EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle                                                                                       rs.OrionRestHandler[153] : Starting transaction (1434103313-190-0000000000)
time=2015-06-12T06:03:56.535EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle                                                                                       rs.OrionRestHandler[239] : Received data ({  "subscriptionId" : "557aae8c98add18c                                                                                       c3e183b6",  "originator" : "localhost",  "contextResponses" : [    {      "contex                                                                                       tElement" : {        "type" : "thing",        "isPattern" : "false",        "id"                                                                                        : "autocarro1",        "attributes" : [          {            "name" : "temperatu                                                                                       re",            "type" : "int",            "value" : "95",            "metadatas"                                                                                        : [              {                "name" : "TimeInstant",                "type"                                                                                        : "ISO8601",                "value" : "2015-06-03T09:17:44.046583Z"                                                                                                     }            ]          }        ]      },      "statusCode" : {        "code" :                                                                                        "200",        "reasonPhrase" : "OK"      }    }  ]})
time=2015-06-12T06:03:56.540EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=getEvents | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.handle                                                                                       rs.OrionRestHandler[261] : Event put in the channel (id=1983722072, ttl=10)
time=2015-06-12T06:03:56.724EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or                                                                                       ionSink[126] : Event got from the channel (id=1983722072, headers={timestamp=1434                                                                                       103436542, content-type=application/json, transactionId=1434103313-190-0000000000                                                                                       , fiware-service=fiwaretestapi, fiware-servicepath=, ttl=10, destination=autocarr                                                                                       o1_thing}, bodyLength=657)
time=2015-06-12T06:03:57.260EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=persist | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or                                                                                       ionMySQLSink[227] : [mysql-sink] Persisting data at OrionMySQLSink. Database: fiw                                                                                       aretestapi, Table: autocarro1_thing, Data: 1434103436,2015-06-12T06:03:56.542,aut                                                                                       ocarro1,thing,temperature,thing,95,[{"name":"TimeInstant","type":"ISO8601","value                                                                                       ":"2015-06-03T09:17:44.046583Z"}]
time=2015-06-12T06:03:57.270EDT | lvl=INFO | trans=1434103313-190-0000000000 | fu                                                                                       nction=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.Or                                                                                       ionSink[187] : Finishing transaction (1434103313-190-0000000000)
frb
  • 3,592
  • 2
  • 17
  • 48
Diogo Nunes
  • 309
  • 3
  • 16

1 Answers1

1

As I can see in the log, most probably Cygnus is running but not starting any Flume component (any source, channel or sink). This is due to some configuration errors.

Regarding agent_a1.conf file:

  • It is missing the list of sources, channels and sinks: cygnusagent.sources = http-source cygnusagent.sinks = mysql-sink cygnusagent.channels = mysql-channel
  • cygnusagent.sources.http-source.channels value should be mysql-channel

Regarding cygnus_instance_c1.conf:

  • AGENT_NAME value must be cygnusagent

Which version have you installed? Are you reunning Cygnus as a service or as an standalone process?

In adition, you could try to start Cygnus in DEBUG mode? Simply edit the /usr/cygnus/conf/log4j.properties file.

Do the proposed changes and see how the log evolves! :)

EDIT 1

Such a "fatal error" is not so fatal. It was a bug appearing in Cygnus 0.7.1, currently fixed. Anyway, even in 0.7.1 it did not affect the normal behaviour of Cygnus since the management port is only used for retrieving the version, nothing important.

Did you try to send some update context to Orion in order a notification is received by Cygnus? Even by simulating the notification? Please, see the Cygnus Quick Start Guide for an example about how to make such a simulation.

EDIT 2

Cygnus packages names es.tid.fiware.fiwareconnectors.cygnus... were replaced by com.telefonica.iot.cygnus... from release 0.8.0 (or maybe 0.9.0).

frb
  • 3,592
  • 2
  • 17
  • 48
  • Hi, regarding the agent_a1.conf file i forgot to add those 3 lines refering to the sources, sinks and channels.. In addition, are you sure the cygnusagent.sources.http-source.channels value should be mysql-sink and not mysql-channel? I don't know what i did but the log file just disappeared from /var/log/cygnus directory :-( – Diogo Nunes May 29 '15 at 09:28
  • You are right about `cygnusagent.sources.http-source.channels`, I'll edit it. Regarding my questions, which version have you installed? Are you reunning Cygnus as a service or as an standalone process? – frb May 29 '15 at 09:43
  • I forgot to tell you i'm sorry. I'm running cygnus as a service and the version installed is 0.7.1. – Diogo Nunes May 29 '15 at 09:55
  • Did you implemented the changes? How was it? If some progress was made, please, edit the question and print the log traces. – frb Jun 01 '15 at 10:54
  • Yes, i implemented the changes but the log file is still missing.. I even changed back to the way it was before DEBUG mode and it still doesn't write anything on the log. – Diogo Nunes Jun 01 '15 at 11:17
  • Doing the comand /etc/init.d/cygnus status i get: Cygnus c1 status... cygnus-flume-ng (pid 27002) is running... So that means that cygnus is running. I don't know what's missing... – Diogo Nunes Jun 01 '15 at 15:31
  • I made some improvements! I edited the question to include de log file :-) – Diogo Nunes Jun 02 '15 at 09:30
  • I edited my answer with updated info regarding your log. – frb Jun 02 '15 at 10:00
  • Hi! Thank you for providing me the link to the quick start guide, i did not noticed that one existed! So i've tryed sending updateContext to Orion before but it did not made anything but now it does. It had something to do with me not starting Cygnus from the command line maybe.. Well thank you very very veryyyy much for your support! :D – Diogo Nunes Jun 02 '15 at 11:11
  • There are 2 ways of starting Cygnus: as a service, or as a standalone application from the command line. The first one is preferred, but it is up to you :) So, everything is OK right now? – frb Jun 03 '15 at 09:06
  • Well it's parttialy ok. I start cygnus as a Service but it only works if i do the item number 2 on the quick start guide that says how to run Cygnus from the command line. If i do service cygnus start; it says it starts but the entities are not presisted when updated value after cygnus subscription. Is that normal? Do i need to start cygnus that way everytime i want to presist the changed value on the entity? – Diogo Nunes Jun 03 '15 at 09:26
  • I noticed something, when i make the subscription the value that is currently on the context broker is stored in mysql database but when i update that value, it is not stored in any table. – Diogo Nunes Jun 11 '15 at 16:21
  • Do you make an "APPEND" as updateAction always? I mean, AFAIK, you make an APPEND in order to create the entity, but once created you must perform "UPDATE" as updateAction (I'm going to double check it with some Orion expert). – frb Jun 12 '15 at 06:35
  • Once checked with an Orion expert, the recommende behaviour should be an initial APPEND, and then UPDATEs. Nevetheless, if you always use APPEND, it is automatically converted into an UPDATE if the entity already exists. Anyway, you say you ony receive data with the initial APPEND... just in case, are you changing the updated value? I mean, the subscription is of type ONCHANGE, which means a notification is sent from Orion to Cygnus if the attribute's value changes. If "99" is always updated, there is no real update. – frb Jun 12 '15 at 07:06
  • Hi, what i say is: i create the entity using APPEND, i asign the value 99 for example. Then i make the subscription. I check the mysql database and i see that the value has been presisted. Then i update the value (tested using UPDATE and APPEND) to 19 for example and i go see if the value is presisted in the database. It's not. Then i make another subscription to that entity. I go see if the value is presisted (without any update) and i see that it is. Therefore i say that the value is only presisted once i make a subscription and the value is already in the Context Broker. – Diogo Nunes Jun 12 '15 at 09:23
  • I see... can you update the post with Cygnus logs? Just to see if the notification arrives to Cygnus or it is a problem with Orion and/or the subscription. I understand you did it for testing purposes, but let me remind you (and other possible readers :)) the subscription must be made just once. – frb Jun 12 '15 at 09:47
  • Yes, i'm doing it only for testing purposes :) i edited the question. I'm sorry but i don't know why do the logs are printed that way in stackoverflow :( – Diogo Nunes Jun 12 '15 at 10:09