3

I am trying to persist my Orion data into the public cosmos.lab.fi-ware.org instance using Cygnus. Cygnus is up and running and the HDFSSink part of my /usr/cygnus/conf/agent_1.conf looks like this:

# OrionHDFSSink configuration cygnusagent.sinks.hdfs-sink.channel = hdfs-channel cygnusagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.OrionHDFSSink cygnusagent.sinks.hdfs-sink.enable_grouping = false cygnusagent.sinks.hdfs-sink.backend_impl = rest cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fi-ware.org cygnusagent.sinks.hdfs-sink.hdfs_port = 14000 cygnusagent.sinks.hdfs-sink.hdfs_username = myUsernameInCosmosLabInstance cygnusagent.sinks.hdfs-sink.hdfs_password = myPasswordInCosmosLabInstance cygnusagent.sinks.hdfs-sink.oauth2_token = myTokenForCosmosLabInstance cygnusagent.sinks.hdfs-sink.hive = true cygnusagent.sinks.hdfs-sink.hive.server_version = 2 cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lablfi-ware.org cygnusagent.sinks.hdfs-sink.hive.port = 10000 cygnusagent.sinks.hdfs-sink.hive.db_type = default-db

I add a new subscription with Cygnus as the reference endpoint and I send an update to previously created NGSIEntity, but nothing appears in my cosmos.lab.fi-ware.org instance. When looking at /var/log/cygnus/cygnus.log I cant find nothing useful, and I find some Java errors.

I am using Orion v. 0.28 and Cygnus v. 0.13.

José Castillo Lema
  • 307
  • 1
  • 2
  • 16

2 Answers2

1

As the log is saying:

Could not open connection to jdbc:hive2://cosmos.lablfi-ware.org:10000/default: java.net.UnknownHostException: cosmos.lablfi-ware.org

You must configure the right Hive endpoint:

cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lab.fiware.org

Instead of:

cygnusagent.sinks.hdfs-sink.hive.host = cosmos.lablfi-ware.org

NOTE: Youy may have noticed I've used cosmos.lab.fiware.org. Both cosmos.lab.fiware.org and cosmos.lab.fi-ware.org are valid, bit the first one is preferred.

frb
  • 3,592
  • 2
  • 17
  • 48
  • Thanks @frb! The data is successfully created as a table in hive in the cosmos.lab.fi-ware.org instance. I was wondering, shouldn´t it be created also as a file? I see this line in the log, but no file is created in my user´s folder: `function=persistAggregation | Persisting data at OrionHDFSSink. HDFS file (def_serv/def_servpath/Room1_Room/Room1_Room.txt)` and I can´t find this **Room1_Room.txt** file anywhere. – José Castillo Lema Mar 06 '16 at 21:07
  • 1
    Such a file should be under your Cosmos user space, i.e. under `hdfs://user/josecastillolema/`. In fact, is is there (I've seen it as super user): `# hadoop fs -ls /user/josecastillolema/def_serv/def_servpath/Room1_Room/Room1_Room.txt Found 1 items -rw-r--r-- 3 josecastillolema josecastillolema 6098 2016-03-06 21:51 /user/josecastillolema/def_serv/def_servpath/Room1_Room/Room1_Room.txt` – frb Mar 07 '16 at 11:47
0

To find the data that Orion was persisting in my Cosmos global instance:

  1. From Hadoop:

# hive hive> select * from myUsernameInCosmosLabInstance_def_serv_def_servpath_room1_room_column;

  1. Alternative method:

# hadoop fs -ls /user/myUsernameInCosmosInstance/def_serv/def_servpath/Room1_Room/Room1_Room.txt

José Castillo Lema
  • 307
  • 1
  • 2
  • 16