4

I've got an Orion instance with Cygnus, subcription and notify work fine but I can not send files to cosmos.lab.fi-ware.org from my instance.

[ERROR - es.tid.fiware.orionco nnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] Connection to http://130.206.80.46:14000 refused

My cygnus.conf :

# APACHE_FLUME_HOME/conf/cygnus.conf
# The next tree fields set the sources, sinks and channels used by Cygnus. You could use different names than the
# ones suggested below, but in that case make sure you keep coherence in properties names along the configuration file.
# Regarding sinks, you can use multiple ones at the same time; the only requirement is to provide a channel for each
# one of them (this example shows how to configure 3 sinks at the same time).
cygnusagent.sources = http-source
cygnusagent.sinks = hdfs-sink
cygnusagent.channels = hdfs-channel
#=============================================
# source configuration
# channel name where to write the notification events
cygnusagent.sources.http-source.channels = hdfs-channel
# source class, must not be changed
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
# listening port the Flume source will use for receiving incoming notifications
cygnusagent.sources.http-source.port = 5050
# Flume handler that will parse the notifications, must not be changed
cygnusagent.sources.http-source.handler =
es.tid.fiware.fiwareconnectors.cygnus.handlers.OrionRestHandler
# URL target
cygnusagent.sources.http-source.handler.notification_target = /notify
# Default organization (organization semantic depend on the persistence sink)
cygnusagent.sources.http-source.handler.default_organization = org42
# ============================================
# OrionHDFSSink configuration
# channel name from where to read notification events
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
# sink class, must not be changed
cygnusagent.sinks.hdfs-sink.type =
es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionHDFSSink
# The FQDN/IP address of the Cosmos deployment where the notification events will be persisted
cygnusagent.sinks.hdfs-sink.cosmos_host = 130.206.80.46
# port of the Cosmos service listening for persistence operations; 14000 for httpfs, 50070 for webhdfs and free choice for inifinty
cygnusagent.sinks.hdfs-sink.cosmos_port = 14000
# default username allowed to write in HDFS
cygnusagent.sinks.hdfs-sink.cosmos_default_username = myUsername
# default password for the default username
cygnusagent.sinks.hdfs-sink.cosmos_default_password = **********
# HDFS backend type (webhdfs, httpfs or infinity)
cygnusagent.sinks.hdfs-sink.hdfs_api = httpfs
# how the attributes are stored, either per row either per column (row, column)
cygnusagent.sinks.hdfs-sink.attr_persistence = column
# prefix for the database and table names, empty if no prefix is desired
cygnusagent.sinks.hdfs-sink.naming_prefix =
# Hive port for Hive external table provisioning
cygnusagent.sinks.hdfs-sink.hive_port = 10000
#=============================================
# hdfs-channel configuration
# channel type (must not be changed)
cygnusagent.channels.hdfs-channel.type = memory
# capacity of the channel
cygnusagent.channels.hdfs-channel.capacity = 1000
# amount of bytes that can be sent per transaction
cygnusagent.channels.hdfs-channel.transactionCapacity = 100

Error log :

2015-02-04 22:52:28,627 (lifecycleSupervisor-1-1) 
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.createDir(HttpFSBackend.java:68)] 
HttpFS operation: PUT 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/?op=mkdirs&user.name=maxime.mularz HTTP/1.1
2015-02-04 22:53:31,690 (lifecycleSupervisor-1-1) 
[ERROR -.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.start(OrionHDFSSink.java:108)] 
Connection to http://130.206.80.46:14000 refused
2015-02-04 22:56:02,182 (SinkRunner-PollingRunner-DefaultSinkProcessor) 
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.OrionHDFSSink.persist(OrionHDFSSink.java:212)] 
Persisting data. File: Room1-Room-temperature-float.txt, Data: 2015- 02-04T22:56:02.182|1423086962|Room1|Room|temperature|float|90)
2015-02-04 22:56:02,183 (SinkRunner-PollingRunner-DefaultSinkProcessor) 
[INFO - es.tid.fiware.orionconnectors.cosmosinjector.hdfs.HttpFSBackend.exists(HttpFSBackend.java:158)] 
HttpFS operation: GET 130.206.80.46:14000/webhdfs/v1/user/maxime.mularz/4planet/Room1-Room-temperature-float.txt?op=getfilestatus&user.name=maxime.mularz HTTP/1.1

Thanks in advance.

fgalan
  • 10,250
  • 6
  • 36
  • 70
Maxime
  • 68
  • 6
  • Yesterday the Namenode of Cosmos entered in safe mode due to the HDD was running out of space. It should be solved now. Try again, please, if not fixed tell us. – frb Feb 04 '15 at 06:24
  • It still does not work.2015-02-04 14:07:47,001 (lifecycleSupervisor-1-1) [ERROR – Maxime Feb 04 '15 at 13:15
  • Ok, let's see... Which version are you running? Can you edit the question by adding the error logs? Is the `cygnusagent.sinks.hdfs-sink.cosmos_default_username = myUsername` parameter deliberately ofuscated? – frb Feb 04 '15 at 15:46
  • Cygnus Version : cygnus-0.5.1 and cygnusagent.sinks.hdfs-sink.cosmos_default_username parameter was deliberately ofuscated. – Maxime Feb 04 '15 at 21:51
  • Are you running Cygnus in a FIWARE LAB virtual machine or a Liunx machine of your own? Because accesses to Cosmos are limited to FIWARE LAB virtual machines. – frb Feb 05 '15 at 08:02
  • Cygnus is on Fiware Lab machine, orion-psb-image-R4.1. – Maxime Feb 05 '15 at 08:20
  • Thanks for your help, I've created a new instance in Spain and it works like a charm. The previous instance was created in Lannion, France. – Maxime Feb 05 '15 at 22:06
  • @frb, it's look like the cosmos.lab.fi-ware.org server doesn't allow access from other country. What can I allow to keep my Lannion instance and access to cosmos.lab.fi-ware.org ? – Maxime Feb 09 '15 at 08:47
  • Let me check internally. Anyway, if it is a matter of connectivity, it can be solved. – frb Feb 09 '15 at 10:49
  • Hi Maxime, as far as I know, the access is now open to Lanion node. You should be able to connect. – frb Feb 26 '15 at 14:35
  • Thank you @frb, it works like a charm – Maxime Mar 02 '15 at 09:32

1 Answers1

1

Fixed once the access from the Lanion node to the spanish node has been enabled.

frb
  • 3,592
  • 2
  • 17
  • 48