c# - DebugDiag 消息线程似乎没有等待远程服务器响应

Question

我们有一个 C# Windows 服务，它运行一个分为多个任务的进程。大多数任务使用 WCF 联系 Web 服务来对数据库执行工作。该服务的任务在多个线程中运行。

一位客户向我们提出了一个支持案例，称 Windows 服务偶尔无法响应，需要重新启动。我从 Windows 服务获取了内存转储。我跑了DebugDiag 2.0分析转储文件。

DebugDiag 报告的摘要中有一个有趣的条目:

The following threads in WindowsService.DMP are attempting to make an HttpWebRequest, however they do not appear to be waiting on the remote server to respond (eg. not 'on the wire'). One or more of these requests are using at least half of its maximum number of available connections.

( 17 18 27 31 32 33 42 ) 12.07% of threads blocked (7 threads)

If many threads are in this state, it is often an indication that a throttling limit (i.e. the 'maxconnection' setting) has been exhausted. Click on any thread in the list to the left to review the throttling details for the WebRequest on which it is waiting.

If necessary, you can increase the number of connections available by either modifying the 'maxconnection' parameter in the application configuration file (see <connectionManagement> Element), or by modifying the appropriate ConnectionLimit property programmatically (see Managing Connections).

我跳到线程 17 并看到了这个:

Thread 17 - System ID 4612

Entry point mscorwks!Thread::intermediateThreadProc Create time 9/10/2015 10:13:14 AM Time spent in user mode 0 Days 00:00:00.000 Time spent in kernel mode 0 Days 00:00:00.000

This thread is attempting to make an HttpWebRequest, however they do not appear to be waiting on the remote server to respond (eg. not 'on the wire'). One or more of these requests are using at least half of its maximum number of available connections.

Warning, at least half of the availabe connections are being used

HttpRequest URI:http://WebServer/MyWebSite/SubDir/MyService.svc ServicePoint - ConnectionLimit:48 CurrentConnections:44

The HttpWebRequest object is a loopback address but the connection limit still applies to this webrequest object because the connection limit is defined (either through autoconfig set to true in the processModel section or by adding a * entry inside connectionManagement section

.NET 调用堆栈如下:

Function

[[HelperMethodFrame_1OBJ] (System.Threading.WaitHandle.WaitOneNative)] System.Threading.WaitHandle.WaitOneNative(Microsoft.Win32.SafeHandles.SafeWaitHandle, UInt32, Boolean, Boolean) 
mscorlib_ni!System.Threading.WaitHandle.WaitOne(Int64, Boolean)+2f 
mscorlib_ni!System.Threading.WaitHandle.WaitOne(Int32, Boolean)+25 
System_ni!System.Net.LazyAsyncResult.WaitForCompletion(Boolean)+d3 
System_ni!System.Net.Connection.SubmitRequest(System.Net.HttpWebRequest)+2b7 
System_ni!System.Net.ServicePoint.SubmitRequest(System.Net.HttpWebRequest, System.String)+7c 
System_ni!System.Net.HttpWebRequest.SubmitRequest(System.Net.ServicePoint)+f9 
System_ni!System.Net.HttpWebRequest.GetRequestStream(System.Net.TransportContext ByRef)+1d3 
System_ni!System.Net.HttpWebRequest.GetRequestStream()+e 
System_ServiceModel_ni!System.ServiceModel.Channels.HttpOutput+WebRequestHttpOutput.GetOutputStream()+45 
System_ServiceModel_ni!System.ServiceModel.Channels.HttpOutput.Send(System.TimeSpan)+f6 
System_ServiceModel_ni!System.ServiceModel.Channels.HttpChannelFactory+HttpRequestChannel+HttpChannelRequest.SendRequest(System.ServiceModel.Channels.Message, System.TimeSpan)+121 
System_ServiceModel_ni!System.ServiceModel.Channels.RequestChannel.Request(System.ServiceModel.Channels.Message, System.TimeSpan)+cb 
System_ServiceModel_ni!System.ServiceModel.Dispatcher.RequestChannelBinder.Request(System.ServiceModel.Channels.Message, System.TimeSpan)+17 
System_ServiceModel_ni!System.ServiceModel.Channels.ServiceChannel.Call(System.String, Boolean, System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.Object[], System.Object[], System.TimeSpan)+1a2 
System_ServiceModel_ni!System.ServiceModel.Channels.ServiceChannel.Call(System.String, Boolean, System.ServiceModel.Dispatcher.ProxyOperationRuntime, System.Object[], System.Object[])+33 
System_ServiceModel_ni!System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage, System.ServiceModel.Dispatcher.ProxyOperationRuntime)+43 
System_ServiceModel_ni!System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage)+65 
mscorlib_ni!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)+bd
[[TPMethodFrame] (IMyWebService.GetDataSet)]
IMyWebService.GetDataSet(System.Guid, System.String, System.Data.DataSet)
<service code snipped>
mscorlib_ni!System.Threading.ThreadHelper.ThreadStart_Context(System.Object)+66
mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+51 
[[HelperMethodFrame_PROTECTOBJ] (System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup)]
System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode, CleanupCode, System.Object) 
mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+67 
mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+45 
mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+44 
[[GCFrame]] 

我看到了它提出的建议并开始研究它们。我的问题是:

为什么 DebugDiag 说线程没有似乎在等待服务器响应？

查看.NET Reference Source ，请求似乎已成功提交，并且服务似乎正在等待响应。

更新

闯入正常调用时，我确实看到调用堆栈在 ws2_2 处等待，如下面 Puneet Gupta 所建议的:

ntdll.dll!_NtWaitForSingleObject@12()
mswsock.dll!_SockWaitForSingleObject@16()
mswsock.dll!_WSPRecv@36()
***ws2_32.dll!_recv@16()***
System.ni.dll!6c084a13()
[Managed to Native Transition]  
System.dll!System.Net.Sockets.Socket.Receive(byte[] buffer, int offset, int size, System.Net.Sockets.SocketFlags socketFlags, out System.Net.Sockets.SocketError errorCode)
System.dll!System.Net.Sockets.Socket.Receive(byte[] buffer, int offset, int size, System.Net.Sockets.SocketFlags socketFlags)
System.dll!System.Net.Sockets.NetworkStream.Read(byte[] buffer, int offset, int size)
System.dll!System.Net.PooledStream.Read(byte[] buffer, int offset, int size)
System.dll!System.Net.Connection.SyncRead(System.Net.HttpWebRequest request, bool userRetrievedStream, bool probeRead)
System.dll!System.Net.ConnectStream.ProcessWriteCallDone(System.Net.ConnectionReturnResult returnResult)
System.dll!System.Net.HttpWebRequest.WriteCallDone(System.Net.ConnectStream stream, System.Net.ConnectionReturnResult returnResult)
System.dll!System.Net.ConnectStream.CallDone(System.Net.ConnectionReturnResult returnResult)
System.dll!System.Net.ConnectStream.ResubmitWrite(System.Net.ConnectStream oldStream, bool suppressWrite)
System.dll!System.Net.HttpWebRequest.EndWriteHeaders_Part2()
System.dll!System.Net.HttpWebRequest.EndWriteHeaders(bool async)
System.dll!System.Net.HttpWebRequest.WriteHeadersCallback(System.Net.WebExceptionStatus errorStatus, System.Net.ConnectStream stream, bool async)
System.dll!System.Net.ConnectStream.WriteHeaders(bool async)
System.dll!System.Net.HttpWebRequest.EndSubmitRequest()
System.dll!System.Net.HttpWebRequest.CheckDeferredCallDone(System.Net.ConnectStream stream)
System.dll!System.Net.HttpWebRequest.GetResponse()
System.ServiceModel.dll!System.ServiceModel.Channels.HttpChannelFactory<System.ServiceModel.Channels.IRequestChannel>.HttpRequestChannel.HttpChannelRequest.WaitForReply(System.TimeSpan timeout)
System.ServiceModel.dll!System.ServiceModel.Channels.RequestChannel.Request(System.ServiceModel.Channels.Message message, System.TimeSpan timeout)
System.ServiceModel.dll!System.ServiceModel.Dispatcher.RequestChannelBinder.Request(System.ServiceModel.Channels.Message message, System.TimeSpan timeout)  Unknown
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannel.Call(string action, bool oneway, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation, object[] ins, object[] outs, System.TimeSpan timeout)
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(System.Runtime.Remoting.Messaging.IMethodCallMessage methodCall, System.ServiceModel.Dispatcher.ProxyOperationRuntime operation)
System.ServiceModel.dll!System.ServiceModel.Channels.ServiceChannelProxy.Invoke(System.Runtime.Remoting.Messaging.IMessage message)
mscorlib.dll!System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(ref System.Runtime.Remoting.Proxies.MessageData msgData, int type)

所以通常情况下，它将等待来自 Windows 套接字的响应。在这种情况下，线程可能正在等待连接变得可用以处理请求 - 如其他 DebugDiag 消息所示。

Answer 1

消息说它似乎没有在线上等待的原因是因为线程上的最后一帧是waithandle.waitone。

对于真正在线等待的线程，我们应该看到 ws2_32(在 native 堆栈中)，它是 Windows 套接字库或托管堆栈中 system.net.sockets 的一些函数。

您能够捕获多个转储吗？如果是，您是否看到一次转储中的线程状态与第二次转储中的线程状态发生了变化？