azure - Windows 从站处于挂起状态 - 使用 Azure VM 代理插件

Question

在我重新启动从属 Azure VM(维护)后，我的 Windows 从属服务器被挂起，并且当从属服务器备份并运行并成功连接到 Jenkins 时，挂起状态并未被删除，因此出现了一些奇怪的行为。

我已经将其复制了 3 次 - 对于每个虚拟机，我都手动安装了额外的软件，这需要重新启动，但重新启动后，我安排的任何构建作业都会挂起，并显示消息“正在等待 Windows 上的下一个可用执行程序” ，从属节点在 Jenkins 中显示为在线，但旁边有“(已暂停)”，然后最终 Azure VM 代理插件只是创建一个新 VM 并在其上运行构建。

从日志中提取:

May 25, 2018 3:47:29 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel slavew823810 java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) Caused by: java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3154) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862) at java.io.ObjectInputStream.(ObjectInputStream.java:358) at hudson.remoting.ObjectInputStreamEx.(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)

May 25, 2018 3:47:34 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Azure VM Agents Clean Task May 25, 2018 3:47:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: start May 25, 2018 3:47:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: Running clean with 5 minute timeout May 25, 2018 3:47:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs INFO: AzureVMAgentCleanUpTask: cleanVMs: node slavew56f680 blocked to cleanup May 25, 2018 3:47:34 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists INFO: AzureVMManagementServiceDelegate: virtualMachineExists: check for slavew823810 May 25, 2018 3:47:34 PM com.microsoft.rest.interceptors.LoggingInterceptor log INFO: <-- HTTP FAILED: java.net.SocketException: Connection reset May 25, 2018 3:47:34 PM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists INFO: AzureVMManagementServiceDelegate: virtualMachineExists: slavew823810 exists May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments INFO: AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments INFO: AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask$2 call INFO: AzureVMAgentCleanUpTask: cleanVMs: shutting down slavew823810 May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMAgent shutdown INFO: AzureVMAgent: shutdown: Add suspended status for node slavew823810 May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMAgent shutdown INFO: AzureVMAgent: shutdown: shutting down agent slavew823810 May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate shutdownVirtualMachine INFO: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for slavew823810 May 25, 2018 3:47:35 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: end May 25, 2018 3:47:35 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Azure VM Agents Clean Task. 389 ms May 25, 2018 3:47:35 PM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? May 25, 2018 3:49:09 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Azure VM Maintainer Pool Size May 25, 2018 3:49:09 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Azure VM Maintainer Pool Size. 1 ms May 25, 2018 3:51:04 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Azure VM Verification Task May 25, 2018 3:51:04 PM com.microsoft.azure.vmagent.AzureVMCloudVerificationTask getVirtualMachineCount INFO: AzureVMCloudVerificationTask: getVirtualMachineCount: start May 25, 2018 3:51:04 PM com.microsoft.azure.vmagent.AzureVMCloudVerificationTask getVirtualMachineCount INFO: AzureVMCloudVerificationTask: getVirtualMachineCount: end, cloud Jenkins Azure Cloud has currently 2 vms May 25, 2018 3:51:04 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Azure VM Verification Task. 39 ms May 25, 2018 3:52:34 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Azure VM Agents Clean Task May 25, 2018 3:52:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: start May 25, 2018 3:52:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: Running clean with 5 minute timeout May 25, 2018 3:52:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs INFO: AzureVMAgentCleanUpTask: cleanVMs: node slavew56f680 blocked to cleanup May 25, 2018 3:52:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs INFO: AzureVMAgentCleanUpTask: cleanVMs: node slavew823810 blocked to cleanup May 25, 2018 3:52:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments INFO: AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments May 25, 2018 3:52:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments INFO: AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments May 25, 2018 3:52:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: end May 25, 2018 3:52:34 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Azure VM Agents Clean Task. 21 ms May 25, 2018 3:53:00 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making slavew56f680 offline because it’s not responding May 25, 2018 3:53:09 PM hudson.node_monitors.ResponseTimeMonitor$1 monitor WARNING: Making slavew56f680 offline because it’s not responding May 25, 2018 3:54:09 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Azure VM Maintainer Pool Size May 25, 2018 3:54:09 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Azure VM Maintainer Pool Size. 1 ms May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: launch method called for agent slavew56f680 May 25, 2018 3:54:56 PM com.microsoft.rest.interceptors.LoggingInterceptor log INFO: <-- HTTP FAILED: java.net.SocketException: Connection reset May 25, 2018 3:54:56 PM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate isVMAliveOrHealthy INFO: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/running May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh INFO: AzureVMAgentSSHLauncher: connectToSsh: start May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession INFO: AzureVMAgentSSHLauncher: getRemoteSession: getting remote session for user slavew to host slavew56f680.ukwest.cloudapp.azure.com:22 May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession INFO: AzureVMAgentSSHLauncher: getRemoteSession: Got remote session for user slavew to host slavew56f680.ukwest.cloudapp.azure.com:22 May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh INFO: AzureVMAgentSSHLauncher: connectToSsh: Got remote connection May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: starting dir C:.azure-agent-init May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: checking for java runtime May 25, 2018 3:54:56 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: starting java -fullversion May 25, 2018 3:54:57 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully May 25, 2018 3:54:57 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: java runtime present, copying slave.jar to remote May 25, 2018 3:54:57 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote INFO: AzureVMAgentSSHLauncher: copyFileToRemote: Initiating file transfer to slave.jar May 25, 2018 3:55:07 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote INFO: AzureVMAgentSSHLauncher: copyFileToRemote: copied file Successfully to slave.jar May 25, 2018 3:55:07 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: launching agent: java -jar slave.jar May 25, 2018 3:55:07 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: Connected successfully May 25, 2018 3:55:13 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: launched agent successfully May 25, 2018 3:57:13 PM com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform getSelectedProtocol INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? May 25, 2018 3:57:18 PM com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform getSelectedProtocol INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? May 25, 2018 3:57:34 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Azure VM Agents Clean Task May 25, 2018 3:57:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: start May 25, 2018 3:57:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: Running clean with 5 minute timeout May 25, 2018 3:57:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs INFO: AzureVMAgentCleanUpTask: cleanVMs: node slavew823810 blocked to cleanup May 25, 2018 3:57:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments INFO: AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments May 25, 2018 3:57:34 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanDeployments INFO: AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments May 25, 2018 3:57:34 PM com.microsoft.rest.interceptors.LoggingInterceptor log INFO: <-- HTTP FAILED: java.net.SocketException: Connection reset May 25, 2018 3:57:34 PM okhttp3.internal.platform.Platform log INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? May 25, 2018 3:57:35 PM com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask execute INFO: AzureVMAgentCleanUpTask: execute: end May 25, 2018 3:57:35 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Azure VM Agents Clean Task. 240 ms May 25, 2018 3:59:09 PM hudson.model.AsyncPeriodicWork$1 run INFO: Started Azure VM Maintainer Pool Size May 25, 2018 3:59:09 PM hudson.model.AsyncPeriodicWork$1 run INFO: Finished Azure VM Maintainer Pool Size. 0 ms May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMCloud provision INFO: AzureVMCloud: provision: start for label windows workLoad 1 May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMCloud provision INFO: AzureVMCloud: provision: checking for node reuse options May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMCloud provision INFO: AzureVMCloud: provision: agent computer eligible for reuse slavew823810 May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists INFO: AzureVMManagementServiceDelegate: virtualMachineExists: check for slavew823810 May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists INFO: AzureVMManagementServiceDelegate: virtualMachineExists: slavew823810 exists May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMCloud provision INFO: AzureVMCloud: provision: asynchronous provision finished, returning 1 planned node(s) May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMCloud$2 call INFO: Found existing node, starting VM slavew823810 May 25, 2018 3:59:15 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate startVirtualMachine INFO: AzureVMManagementServiceDelegate: startVirtualMachine: slavew823810 May 25, 2018 3:59:15 PM org.jenkinsci.plugins.cloudstats.CloudStatistics logTypeNotSupported INFO: No support for cloud-stats plugin by class hudson.slaves.NodeProvisioner$PlannedNode May 25, 2018 3:59:15 PM hudson.slaves.NodeProvisioner$StandardStrategyImpl apply INFO: Started provisioning slavew823810 from Jenkins Azure Cloud with 1 executors. Remaining excess workload: -0.282 May 25, 2018 3:59:24 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate setVirtualMachineDetails INFO: Azure agent details: nodeNameslavew823810 adminUserName=3b22afb5-97a8-4da1-9436-18a770b5c131 shutdownOnIdle=true retentionTimeInMin=0 labels=windows May 25, 2018 3:59:24 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: launch method called for agent slavew823810 May 25, 2018 3:59:48 PM com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate isVMAliveOrHealthy INFO: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/running May 25, 2018 3:59:48 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh INFO: AzureVMAgentSSHLauncher: connectToSsh: start May 25, 2018 3:59:48 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession INFO: AzureVMAgentSSHLauncher: getRemoteSession: getting remote session for user slavew to host slavew823810.ukwest.cloudapp.azure.com:22 May 25, 2018 3:59:50 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession INFO: AzureVMAgentSSHLauncher: getRemoteSession: Got remote session for user slavew to host slavew823810.ukwest.cloudapp.azure.com:22 May 25, 2018 3:59:50 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh INFO: AzureVMAgentSSHLauncher: connectToSsh: Got remote connection May 25, 2018 3:59:50 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: starting dir C:.azure-agent-init May 25, 2018 3:59:51 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully May 25, 2018 3:59:51 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: checking for java runtime May 25, 2018 3:59:51 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: starting java -fullversion May 25, 2018 3:59:52 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand INFO: AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully May 25, 2018 3:59:52 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: java runtime present, copying slave.jar to remote May 25, 2018 3:59:52 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote INFO: AzureVMAgentSSHLauncher: copyFileToRemote: Initiating file transfer to slave.jar May 25, 2018 4:00:02 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote INFO: AzureVMAgentSSHLauncher: copyFileToRemote: copied file Successfully to slave.jar May 25, 2018 4:00:02 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: launching agent: java -jar slave.jar May 25, 2018 4:00:02 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: Connected successfully May 25, 2018 4:00:13 PM com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch INFO: AzureVMAgentSSHLauncher: launch: launched agent successfully May 25, 2018 4:00:13 PM com.microsoft.azure.vmagent.AzureVMCloud$2 call INFO: Remove suspended status for node: slavew823810 May 25, 2018 4:00:15 PM hudson.slaves.NodeProvisioner$2 run INFO: slavew823810 provisioning successfully completed. We have now 3 computer(s) May 25, 2018 4:00:31 PM com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform getSelectedProtocol INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path? May 25, 2018 4:00:44 PM org.jenkinsci.plugins.workflow.job.WorkflowRun finish INFO: tokio-utp/master #29 completed: FAILURE

从日志中可以看到，在 3:47 我手动重新启动了从节点“slavew56f680”。

到 3:54，从属节点已备份并运行，因此我单击启动代理，该代理连接成功(但在 UI 中仍然显示已暂停)。

在 3:57 点，我点击构建了一个作业，希望解除挂起状态并在该从属设备上运行该作业。

在 3:59，它转移到另一个从站(“slavew823810”)上，将其放到网上并在其上运行作业。

在这个特定的示例中，slavew823810 是以前发生过这种情况的 Windows 从属设备，我只是从未删除它，并且它启动了它备份。如果我删除了它，那么它就会创建一个新的 Windows 从属虚拟机。

有人见过这个吗？知道如何阻止这种情况发生或取消暂停状态吗？

Jenkins 版本是 2.107.3 我的主服务器 (Linux) 和从服务器 (Windows) 都在 Azure 上。

提前致谢，斯蒂芬。

更新

在周末关闭我的从机后，我在 Azure 中启动了它们，然后一旦连接到 Jenkins(但说已暂停)，我就开始了构建作业 - 结果是，经过长时间的延迟后，它创建了另一台 Windows 从机来运行该作业在。我现在有 3 个 Windows 从机。我希望它能使用之前两台机器中的一台，这样我就不会被阻止，因为它们已经更新了我需要的额外软件和补丁。

Answer 1

尝试系统 groovy 命令(或类似的 Azure VM):

Jenkins.instance.getNode('Node-Name').getComputer().setAcceptingTasks(true)

似乎没有其他方便的方法来取消暂停。

当我们将节点设置为:

时，我们遇到了类似的困境:

可用性:“根据时间表让该代理上线

然后将它们翻转回“尽可能让这个从属设备在线” 当他们仍处于停机窗口时。节点在线但已暂停。

找不到任何有关此问题的文档，也找不到为什么没有 GUI 切换选项(例如脱机或断开连接/启动)。