java - Apache Storm : storm-kafka-monitor script throws exception

Question

我已在本地设置了一个 Apache Storm 集群，并提交了一个拓扑，该拓扑在 Storm UI 中可见。一旦一切启动并运行，我就会在 ui.log 文件中收到以下异常。该错误与 storm-kafka-monitor 有点相关，它是一个 bash 脚本，我在 Windows 上运行它。
任何帮助将不胜感激。预先感谢:)

2018-06-05 11:26:20.293 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception thrown while getting lag for spout id: kafka-spout and spout class: org.apache.storm.kafka.KafkaSpout
2018-06-05 11:26:20.294 o.a.s.u.TopologySpoutLag qtp1185188034-26 [WARN] Exception message:Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
java.io.IOException: Cannot run program "E:\apache-storm-1.2.1\bin\storm-kafka-monitor": CreateProcess error=193, %1 is not a valid Win32 application
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_162]
    at org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:180) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.run(ShellUtils.java:161) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils$ShellCommandExecutor.execute(ShellUtils.java:371) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:461) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:444) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.getLagResultForKafka(TopologySpoutLag.java:163) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.getLagResultForOldKafkaSpout(TopologySpoutLag.java:193) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.utils.TopologySpoutLag.lag(TopologySpoutLag.java:59) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$topology_lag.invoke(core.clj:805) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$fn__9586.invoke(core.clj:1165) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$make_route$fn__5979.invoke(core.clj:100) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$if_route$fn__5967.invoke(core.clj:46) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$if_method$fn__5960.invoke(core.clj:31) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.compojure.core$routing$fn__5985.invoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
    at clojure.core$some.invoke(core.clj:2570) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113) ~[storm-core-1.2.1.jar:1.2.1]
    at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[clojure-1.7.0.jar:?]
    at clojure.core$apply.invoke(core.clj:632) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.shade.compojure.core$routes$fn__5989.invoke(core.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.cors$wrap_cors$fn__8894.invoke(cors.clj:149) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__8841.invoke(json.clj:56) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__7904.invoke(reload.clj:22) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.helpers$requests_middleware$fn__6874.invoke(helpers.clj:52) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.core$catch_errors$fn__9761.invoke(core.clj:1428) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__6541.invoke(keyword_params.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__6584.invoke(nested_params.clj:84) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.params$wrap_params$fn__6513.invoke(params.clj:64) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__6621.invoke(multipart_params.clj:118) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__6836.invoke(flash.clj:35) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.middleware.session$wrap_session$fn__6822.invoke(session.clj:98) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet$make_service_method$fn__6371.invoke(servlet.clj:127) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet$servlet$fn__6375.invoke(servlet.clj:136) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162]
    at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
    at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
    at org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__6967.invoke(helpers.clj:192) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown Source) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ~[storm-core-1.2.1.jar:1.2.1]
    at org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) ~[storm-core-1.2.1.jar:1.2.1]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.io.IOException: CreateProcess error=193, %1 is not a valid Win32 application
    at java.lang.ProcessImpl.create(Native Method) ~[?:1.8.0_162]
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) ~[?:1.8.0_162]
    at java.lang.ProcessImpl.start(ProcessImpl.java:137) ~[?:1.8.0_162]
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_162]
    ... 68 more

Answer 1

正如您所注意到的，该脚本是一个 bash 文件。该功能(在 Storm UI 中显示喷口落后日志端偏移多远)当前不适用于 Windows。这有点不方便，但它不应该妨碍您的拓扑正常工作。

如果您想知道 spout 落后日志端偏移量多远，您可以在其中一台 Kafka 机器上使用 kafka-consumer-groups.bat 脚本。如果您使用的是 storm-kafka-client spout，则 spout 滞后也是该组件提供的指标的一部分。

我希望我们能够在某个时候摆脱 storm-kafka-monitor (它还有一些其他问题 https://issues.apache.org/jira/browse/STORM-2600 )，但在那之前，如果您想尝试提供Windows 兼容性，您需要移植 https://github.com/apache/storm/blob/master/bin/storm-kafka-monitor到cmd/powershell，修改https://github.com/apache/storm/blob/4137328b75c06771f84414c3c2113e2d1c757c08/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L146因此它会根据操作系统调用 Windows 或 bash 脚本。