fiware - 天鹅座宇宙整合

Question

我们成功整合了 idas-cygnus-orion 流程，一切工作正常。 (感谢弗朗西斯科的帮助)

现在我们想要将 Cosmos 添加到集成中，我已经使用您发布的答案配置了 Cygnus here (以 * 开头的部分是为 Cosmos 配置添加的)

当我启动 Cygnus 时，我们收到此错误消息

        [root@idas-new centos]# tail -500f nohup.out
Warning: No configuration directory set! Use --conf <dir> to override.
Warning: JAVA_HOME is not set!
+ exec /usr/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/usr/cygnus/lib/*:/usr/cygnus/plugins.d/cygnus/lib/*:/usr/cygnus/agent_test.conf -n cygnusagent
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/cygnus/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/cygnus/plugins.d/cygnus/lib/cygnus-0.12.0_SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
16/05/18 09:26:46 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
16/05/18 09:26:46 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/usr/cygnus/conf/agent_test.con
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Added sinks: hdfs-sink Agent: cygnusagent
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Processing:hdfs-sink
16/05/18 09:26:46 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [cygnusagent]
16/05/18 09:26:46 INFO node.AbstractConfigurationProvider: Creating channels
16/05/18 09:26:46 INFO channel.DefaultChannelFactory: Creating instance of channel hdfs-channel type memory
16/05/18 09:26:46 INFO node.AbstractConfigurationProvider: Created channel hdfs-channel
16/05/18 09:26:46 INFO source.DefaultSourceFactory: Creating instance of source http-source, type org.apache.flume.source.http.HTTPS
16/05/18 09:26:46 INFO handlers.OrionRestHandler: Cygnus version (0.12.0_SNAPSHOT.d6ee0e4548ca6bf9116c1dee4c4cbcc7da8b1752)
16/05/18 09:26:46 INFO handlers.OrionRestHandler: Startup completed
16/05/18 09:26:46 INFO sink.DefaultSinkFactory: Creating instance of sink: hdfs-sink, type: com.telefonica.iot.cygnus.sinks.OrionHDF
16/05/18 09:26:46 ERROR node.AbstractConfigurationProvider: Sink hdfs-sink has been removed due to an error during configuration
java.lang.NullPointerException
        at com.telefonica.iot.cygnus.sinks.OrionHDFSSink.configure(OrionHDFSSink.java:339)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurati
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
16/05/18 09:26:46 INFO node.AbstractConfigurationProvider: Channel hdfs-channel connected to [http-source]
16/05/18 09:26:46 INFO node.Application: Starting new configuration:{ sourceRunners:{http-source=EventDrivenSourceRunner: { source:oyChannel{name: hdfs-channel}} }
16/05/18 09:26:46 INFO node.Application: Starting Channel hdfs-channel
16/05/18 09:26:46 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: hdfs-channel, regist
16/05/18 09:26:46 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: hdfs-channel started
16/05/18 09:26:46 INFO node.Application: Starting Source http-source
16/05/18 09:26:46 INFO interceptors.GroupingRules: No grouping rules have been read
Exception in thread "Thread-1" java.lang.NullPointerException
        at java.io.File.<init>(Unknown Source)
        at com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$ConfigurationReader.run(GroupingInterceptor.java:244)
16/05/18 09:26:46 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
16/05/18 09:26:46 INFO mortbay.log: jetty-6.1.26
16/05/18 09:26:47 INFO mortbay.log: Started <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="60330f030b0514230f0e0e0503140f1220504e504e504e50" rel="noreferrer noopener nofollow">[email protected]</a>:5050

我们的路径中有这个 grouping_rules.conf 文件，但在里面，所有内容都被注释了(因为我们不需要这个文件来进行 CKAN 集成)我们需要为 Cosmos 编辑这个文件吗？(如果是，如何编辑？)

您能帮助我们解决这个问题吗？

非常感谢 奥马尔

cygnusagent.sources = http-source
cygnusagent.sinks = ckan-sink hdfs-sink
cygnusagent.channels = ckan-channel hdfs-channel

cygnusagent.sources.http-source.channels = ckan-channel hdfs-channel
cygnusagent.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnusagent.sources.http-source.port = 5050
cygnusagent.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.OrionRestHandler
cygnusagent.sources.http-source.handler.notification_target = /notify
cygnusagent.sources.http-source.handler.events_ttl = 2
cygnusagent.sources.http-source.interceptors = ts gi
cygnusagent.sources.http-source.interceptors.ts.type = timestamp
cygnusagent.sources.http-source.interceptors.gi.type = com.telefonica.iot.cygnus.interceptors.GroupingInterceptor$Builder
cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /etc/cygnus/conf/grouping_rules.conf

cygnusagent.channels.ckan-channel.type = memory
cygnusagent.channels.ckan-channel.capacity = 1000
cygnusagent.channels.ckan-channel.transactionCapacity = 100

cygnusagent.sinks.ckan-sink.channel = ckan-channel
cygnusagent.sinks.ckan-sink.type = com.telefonica.iot.cygnus.sinks.OrionCKANSink
cygnusagent.sinks.ckan-sink.enable_grouping = false
cygnusagent.sinks.ckan-sink.api_key = XXXXXXXXXX
cygnusagent.sinks.ckan-sink.ckan_host = data.lab.fiware.org
cygnusagent.sinks.ckan-sink.ckan_port = 443
cygnusagent.sinks.ckan-sink.orion_url = localhost:1026
cygnusagent.sinks.ckan-sink.attr_persistence = row
cygnusagent.sinks.ckan-sink.ssl = true
cygnusagent.sinks.ckan-sink.batch_size = 1
cygnusagent.sinks.ckan-sink.batch_timeout = 10

cygnusagent.sinks.hdfs-sink.type = com.telefonica.iot.cygnus.sinks.OrionHDFSSink
cygnusagent.sinks.hdfs-sink.channel = hdfs-channel
cygnusagent.sinks.hdfs-sink.enable_grouping = false
cygnusagent.sinks.hdfs-sink.hdfs_host = cosmos.lab.fiware.org
cygnusagent.sinks.hdfs-sink.hdfs_port = 14000
cygnusagent.sinks.hdfs-sink.hdfs_username = XXXXX
cygnusagent.sinks.hdfs-sink.hdfs_password = XXX
cygnusagent.sinks.hdfs-sink.oauth2_token = XXXXXXXXX
cygnusagent.sinks.hdfs-sink.file_format = json-column


cygnusagent.channels.hdfs-channel.type = memory
cygnusagent.channels.hdfs-channel.capacity = 1000
cygnusagent.channels.hdfs-channel.transactionCapacity = 100

Answer 1

配置似乎一切正常，除了:

cygnusagent.sources.http-source.interceptors.gi.gropuing_rules_conf_file = /etc/cygnus/conf/grouping_rules.
conf

• gropuing -> grouping(这可能来自《快速入门指南》，在我们修复之前此拼写错误一直存在)。
• conf 文件名的一部分似乎在第二行... Flume 配置不喜欢这样，最好所有参数都在同一行中。

尝试此更改并让我们看看。