mesos - Spark shell 连接到 Mesos 挂起 : No credentials provided. 尝试在没有身份验证的情况下注册

Question

我使用 Mesosphere 中的以下说明在 OpenStack 环境中安装了 Mesos:https://open.mesosphere.com/getting-started/datacenter/install/ .我按照描述运行了验证测试，它成功了。 Mesos 和 Marathon 的 UI 都按预期工作。

当我从笔记本电脑运行 Spark shell 时，我无法连接。 shell 与下面的输出一起挂起。我在 Mesos 主站或从站日志中没有看到任何指示错误的内容，因此我不确定接下来要调查什么。

任何帮助，将不胜感激。

TOMWATER-M-60SN:bin tomwater$ ./spark-shell --master mesos://zk://10.93.193.78:2181,10.93.193.79:2181,10.93.193.80:2181/mesos
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/06 15:39:02 INFO SecurityManager: Changing view acls to: tomwater
15/08/06 15:39:02 INFO SecurityManager: Changing modify acls to: tomwater
15/08/06 15:39:02 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tomwater); users with modify permissions: Set(tomwater)
15/08/06 15:39:02 INFO HttpServer: Starting HTTP Server
15/08/06 15:39:02 INFO Utils: Successfully started service 'HTTP class server' on port 63056.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.4.1
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_51)
Type in expressions to have them evaluated.
Type :help for more information.
15/08/06 15:39:05 INFO SparkContext: Running Spark version 1.4.1
15/08/06 15:39:05 INFO SecurityManager: Changing view acls to: tomwater
15/08/06 15:39:05 INFO SecurityManager: Changing modify acls to: tomwater
15/08/06 15:39:05 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(tomwater); users with modify permissions: Set(tomwater)
15/08/06 15:39:05 INFO Slf4jLogger: Slf4jLogger started
15/08/06 15:39:05 INFO Remoting: Starting remoting
15/08/06 15:39:05 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@10.93.235.120:63057]
15/08/06 15:39:05 INFO Utils: Successfully started service 'sparkDriver' on port 63057.
15/08/06 15:39:05 INFO SparkEnv: Registering MapOutputTracker
15/08/06 15:39:05 INFO SparkEnv: Registering BlockManagerMaster
15/08/06 15:39:05 INFO DiskBlockManager: Created local directory at /private/var/folders/7g/p1nw5zg94yx5cck_6c4jgwh80000gp/T/spark-74145a91-396f-4989-b2c0-5902e32e9e16/blockmgr-511d3fdf-f84a-40dc-b6e5-daace4d3f786
15/08/06 15:39:05 INFO MemoryStore: MemoryStore started with capacity 265.1 MB
15/08/06 15:39:05 INFO HttpFileServer: HTTP File server directory is /private/var/folders/7g/p1nw5zg94yx5cck_6c4jgwh80000gp/T/spark-74145a91-396f-4989-b2c0-5902e32e9e16/httpd-4ce76073-5636-4656-9fba-633fbc1c16f4
15/08/06 15:39:05 INFO HttpServer: Starting HTTP Server
15/08/06 15:39:05 INFO Utils: Successfully started service 'HTTP file server' on port 63058.
15/08/06 15:39:05 INFO SparkEnv: Registering OutputCommitCoordinator
15/08/06 15:39:05 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/08/06 15:39:05 INFO SparkUI: Started SparkUI at http://10.93.235.120:4040
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@716: Client environment:host.name=TOMWATER-M-60SN
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@723: Client environment:os.name=Darwin
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@724: Client environment:os.arch=14.4.0
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@725: Client environment:os.version=Darwin Kernel Version 14.4.0: Thu May 28 11:35:04 PDT 2015; root:xnu-2782.30.5~1/RELEASE_X86_64
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@733: Client environment:user.name=tomwater
I0806 15:39:06.235976 547205120 sched.cpp:157] Version: 0.23.0
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@741: Client environment:user.home=/Users/tomwater
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@log_env@753: Client environment:user.dir=/Users/tomwater/development/tools/spark-1.4.1-bin-hadoop2.6/bin
2015-08-06 15:39:06,236:30782(0x1210e7000):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=10.93.193.78:2181,10.93.193.79:2181,10.93.193.80:2181 sessionTimeout=10000 watcher=0x11eca0d00 sessionId=0 sessionPasswd=<null> context=0x7f8f7cffbaf0 flags=0
2015-08-06 15:39:06,333:30782(0x12147c000):ZOO_INFO@check_events@1703: initiated connection to server [10.93.193.78:2181]
2015-08-06 15:39:06,705:30782(0x12147c000):ZOO_INFO@check_events@1750: session establishment complete on server [10.93.193.78:2181], sessionId=0x14f0502209a0006, negotiated timeout=10000
I0806 15:39:06.707475 544960512 group.cpp:313] Group process (group(1)@10.93.235.120:63059) connected to ZooKeeper
I0806 15:39:06.707785 544960512 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0806 15:39:06.707952 544960512 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0806 15:39:06.712241 547741696 detector.cpp:138] Detected a new leader: (id='126')
I0806 15:39:06.712530 555130880 group.cpp:656] Trying to get '/mesos/info_0000000126' in ZooKeeper
W0806 15:39:06.714071 544960512 detector.cpp:444] Leading master master@192.168.1.69:5050 is using a Protobuf binary format when registering with ZooKeeper (info): this will be deprecated as of Mesos 0.24 (see MESOS-2340)
I0806 15:39:06.714269 544960512 detector.cpp:481] A new leading master (UPID=master@192.168.1.69:5050) is detected
I0806 15:39:06.714498 544960512 sched.cpp:254] New master detected at master@192.168.1.69:5050
I0806 15:39:06.714643 544960512 sched.cpp:264] No credentials provided. Attempting to register without authentication

Answer 1

我刚刚有了这个 - 显然要检查您是否可以与主 mesos 节点通信(通常在端口 5050 上)。但是，您还需要允许 mesos master 与您的客户端对话(令人讨厌的是，它是一个临时端口)。

如果你跟踪它，你可以看到发生了什么。

strace -e trace=network -f -s 16384 -o /tmp/strace.log pyspark

查看 strace.log - 首先我们请求一个随机套接字并监听它:

28462 socket(PF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 254
28462 setsockopt(254, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
28462 bind(254, {sa_family=AF_INET, sin_port=htons(0), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
28462 getsockname(254, {sa_family=AF_INET, sin_port=htons(46975), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
28462 listen(254, 500000)               = 0

现在是有趣的部分 - 我们与 mesos master (10.1.201.191:5050) 交谈，然后告诉它我们的 IP 和我们打开的端口 (10.1.200.212:46975)

然后它会回复我们(accept()):

28507 connect(258, {sa_family=AF_INET, sin_port=htons(5050), sin_addr=inet_addr("10.1.201.191")}, 16) = -1 EINPROGRESS (Operation now in progress)
28510 getsockopt(258, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
28510 sendto(258, "POST /master/mesos.scheduler.Call HTTP/1.1\r\nUser-Agent: libprocess/scheduler-52db362e-d5dd-4109-97d3-e28e80f2391b@10.1.200.212:46975\r\nLibproce
ss-From: scheduler-52db362e-d5dd-4109-97d3-e28e80f2391b@10.1.200.212:46975\r\nConnection: Keep-Alive\r\nHost: \r\nTransfer-Encoding: chunked\r\n\r\n54\r\n\20\1\32P\n
N\n\6ubuntu\22\fPySparkShell:\34ip-10-1-200-212.ec2.internalJ\30http://10.1.200.212:4040\r\n0\r\n\r\n", 375, MSG_NOSIGNAL, NULL, 0) = 375
28510 accept(254, {sa_family=AF_INET, sin_port=htons(33743), sin_addr=inet_addr("10.1.201.191")}, [16]) = 259