java - 使用Docker构建一个Maven项目并提交为Flink作业

标签 java docker maven docker-compose apache-flink

我正在尝试使用docker构建Maven项目并在flink中提交生成的jar。但是最后，发生了一些异常。任何人都可以帮助我解决该异常。
Docker文件

FROM maven:3.6-jdk-8-slim AS builder

COPY src /home/app/src
COPY pom.xml /home/app

RUN mvn -f /home/app/pom.xml clean package

FROM flink:1.10.0-scala_2.12
COPY --from=builder /home/app/target/demo-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/local/lib/demo.jar

ENV FLINK_HOME=/opt/flink
ENV PATH=$FLINK_HOME/bin:$PATH

WORKDIR $FLINK_HOME

EXPOSE 8081

ENTRYPOINT ["./bin/flink", "run", "-c", "com.demo.app.App", "/usr/local/lib/demo.jar"]
docker-compose.yml
version: "3"
services:
  demo:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: demoapp
    expose:
      - "6121"
      - "6122"
      - "6123"
    ports:
      - "8081:8081"
异常(exception)
------------------------------------------------------------
Building demo
Step 1/10 : FROM maven:3.6-jdk-8-slim AS builder
 ---> addee4586ff4
Step 2/10 : COPY src /home/app/src
 ---> Using cache
 ---> 24e9837c9bc5
Step 3/10 : COPY pom.xml /home/app
 ---> Using cache
 ---> 13965367c201
Step 4/10 : RUN mvn -f /home/app/pom.xml clean package
 ---> Using cache
 ---> 0a6dec38b357
Step 5/10 : FROM flink:1.10.0-scala_2.12
 ---> d373f7bf2f75
Step 6/10 : COPY --from=builder /home/app/target/demo-1.0-SNAPSHOT-jar-with-dependencies.jar /usr/local/lib/demo.jar
 ---> 431c3080d639
Step 7/10 : ENV FLINK_HOME=/opt/flink
 ---> Running in bbae40bc5d84
Removing intermediate container bbae40bc5d84
 ---> 59171a4fd11e
Step 8/10 : ENV PATH=$FLINK_HOME/bin:$PATH
 ---> Running in 45e30a54e742
Removing intermediate container 45e30a54e742
 ---> 4e986527391a
Step 9/10 : WORKDIR $FLINK_HOME
 ---> Running in 1f0ca2067595
Removing intermediate container 1f0ca2067595
 ---> aa817f1dee2d
Step 10/10 : ENTRYPOINT ["./bin/flink", "run", "-c", "com.demo.app.App", "/usr/local/lib/demo.jar"]
 ---> Running in f984981dfbc8
Removing intermediate container f984981dfbc8
 ---> 1cdc22379a90
Successfully built 1cdc22379a90
Successfully tagged demo:latest
Creating demo ... 
Creating demo ... done
Attaching to demo
demo | 
demo | ------------------------------------------------------------
demo |  The program finished with the following exception:
demo | 
demo | org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
demo |  at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:335)
demo |  at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:205)
demo |  at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:138)
demo |  at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:664)
demo |  at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
demo |  at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:895)
demo |  at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:968)
demo |  at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
demo |  at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:968)
demo | Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
demo |  at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:199)
demo |  at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1741)
demo |  at org.apache.flink.streaming.api.environment.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:94)
demo |  at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:63)
demo |  at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1620)
demo |  at com.demo.app.App.main(App.java:105)
demo |  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
demo |  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
demo |  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
demo |  at java.lang.reflect.Method.invoke(Method.java:498)
demo |  at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:321)
demo |  ... 8 more
demo | Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
demo |  at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
demo |  at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
demo |  at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1736)
demo |  ... 17 more
demo | Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed to submit JobGraph.
demo |  at org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:359)
demo |  at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
demo |  at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
demo |  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
demo |  at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
demo |  at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:287)
demo |  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
demo |  at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
demo |  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
demo |  at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
demo |  at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:342)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
demo |  at java.lang.Thread.run(Thread.java:748)
demo | Caused by: org.apache.flink.runtime.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
demo |  at org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:284)
demo |  ... 21 more
demo | Caused by: java.util.concurrent.CompletionException: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8081
demo |  at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
demo |  at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
demo |  at java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
demo |  at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
demo |  ... 19 more
demo | Caused by: org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8081
demo | Caused by: java.net.ConnectException: Connection refused
demo |  at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
demo |  at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:714)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:336)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:549)
demo |  at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:511)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:918)
demo |  at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
demo |  at java.lang.Thread.run(Thread.java:748)
另外，检查正在运行的容器，但不运行任何容器。
我使用的构建命令
sudo docker-compose up --build
还尝试使用此命令进行构建
docker run --rm -it demo:latest
sudo docker build -t demo .
但是结果一样异常

最佳答案

这不是使用docker运行Flink的方法。您需要安排一个Job Manager和至少一个Task Manager启动-您必须先创建要运行的作业的集群才能使用./bin/flink run。
学习这个主题的一个好方法是遍历Flink Operations Playground，它包括一个完整的基于docker的基础结构，您可以将其用作自己项目的起点。
还有documentation on how to run Flink on Docker。

关于java - 使用Docker构建一个Maven项目并提交为Flink作业，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/63307957/

上一篇：csv - 无法在 hive 中转换列

下一篇：docker - 无法更改在Docker中运行的数据库

相关文章：

java - 当我的 JUnit 测试从 Ant 运行时 getPackage() 返回 null

docker - 拒绝 127.0.0.1 :5672 Rabbit-mq with docker compose

运行 Rust 应用程序的 Azure 应用服务 Docker 容器发出出站 HTTP 请求 "unable to get local issuer certificate"

java - 如何修改我的 eclipse.ini 设置以构建大型 java 项目？

java - 在Java中打印排序数组

docker - docker-compose up-使用云docker-machine构建变得非常慢

maven - 如何为本地 .m2 存储库设置 archiveBaseName

java - Maven 命令 mvn 从终端运行没有错误，但从 python 没有错误

java - 执行 maven 3 - 何时使用 maven enforcer 插件？什么时候使用 pom 先决条件元素？

java - 接受 5 个名字并打印最长的名字？

©2024 IT工具网  联系我们