docker - 为什么 kubelet 无法从 docker 获取统计信息(错误 500)？

Question

我刚刚注意到我的 kubelet 日志中有很多错误，如下所示:

Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.010282    6851 server.go:281] HTTP InternalServerError: Internal Error: unable to find Docker container "d3c1a6936a84028bd420f75c44a13a6ad7c816b651d33877352f2708dccbd943"
Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.015338    6851 server.go:944] GET /stats/kube-system/kubedash-lm97r/cfa615f2-a405-11e5-8e30-525400ad92ba/kubedash: (10.337027ms) 500
Dec 16 15:21:00 kubenode4 kubelet[6851]: goroutine 2288 [running]:
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc20953cfc0, 0x1f4)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:188 +0x9a
Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.Error(0x7f74e8150938, 0xc20953cfc0, 0xc2088c9e00, 0x72, 0x1f4)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1274 +0x9a
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).error(0xc2080df360, 0x7f74e8150938, 0xc20953cfc0, 0x7f74ebf63dc0, 0xc2089665f0)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:282 +0x213
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).serveStats(0xc2080df360, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:1034 +0x78f
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).handleStats(0xc2080df360, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:440 +0x46
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.*Server.(k8s.io/kubernetes/pkg/kubelet.handleStats)��fm(0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:154 +0x45
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*httpHandler).ServeHTTP(0xc2081de558, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:275 +0x45
Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.(*ServeMux).ServeHTTP(0xc2081f5350, 0x7f74e8150938, 0xc20953cfc0, 0xc2086c6d00)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1541 +0x17d
Dec 16 15:21:00 kubenode4 kubelet[6851]: github.com/emicklei/go-restful.Container.ServeHTTP(0x0, 0x0, 0x0, 0xc20824b3e0, 0x2, 0x2, 0xc2081f5350, 0x0, 0x1fc09a0, 0x0, ...)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/emicklei/go-restful/container.go:264 +0x4f
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).ServeHTTP(0xc2080df340, 0x7f74e8150900, 0xc208c38000, 0xc2086c6d00)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:943 +0x162
Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.serverHandler.ServeHTTP(0xc208045f2 [[Go 1.1 package http] 10.233.64.1:57859]
Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.010309    6851 server.go:281] HTTP InternalServerError: Internal Error: unable to find Docker container "534ae8d2b4f4e2a111b7c38f9c67b9189f001f72a26fb6e76c3d71e3ec790a01"
Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.015509    6851 server.go:944] GET /stats/kube-system/influxdb-kj5j2/c92a2514-a405-11e5-8e30-525400ad92ba/influxdb: (10.195837ms) 500
Dec 16 15:21:00 kubenode4 kubelet[6851]: goroutine 2306 [running]:
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/httplog.(*respLogger).WriteHeader(0xc20953d140, 0x1f4)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/httplog/log.go:188 +0x9a
Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.Error(0x7f74e8150938, 0xc20953d140, 0xc20938f400, 0x72, 0x1f4)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1274 +0x9a
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).error(0xc2080df360, 0x7f74e8150938, 0xc20953d140, 0x7f74ebf63dc0, 0xc208e9c240)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:282 +0x213
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).serveStats(0xc2080df360, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:1034 +0x78f
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).handleStats(0xc2080df360, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:440 +0x46
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.*Server.(k8s.io/kubernetes/pkg/kubelet.handleStats)��fm(0x7f74e8150938, 0xc20953d140, 0xc2086c7040)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:154 +0x45
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*httpHandler).ServeHTTP(0xc2081de558, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:275 +0x45
Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.(*ServeMux).ServeHTTP(0xc2081f5350, 0x7f74e8150938, 0xc20953d140, 0xc2086c7040)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /usr/src/go/src/net/http/server.go:1541 +0x17d
Dec 16 15:21:00 kubenode4 kubelet[6851]: github.com/emicklei/go-restful.Container.ServeHTTP(0x0, 0x0, 0x0, 0xc20824b3e0, 0x2, 0x2, 0xc2081f5350, 0x0, 0x1fc09a0, 0x0, ...)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/Godeps/_workspace/src/github.com/emicklei/go-restful/container.go:264 +0x4f
Dec 16 15:21:00 kubenode4 kubelet[6851]: k8s.io/kubernetes/pkg/kubelet.(*Server).ServeHTTP(0xc2080df340, 0x7f74e8150900, 0xc208c38140, 0xc2086c7040)
Dec 16 15:21:00 kubenode4 kubelet[6851]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/server.go:943 +0x162
Dec 16 15:21:00 kubenode4 kubelet[6851]: net/http.serverHandler.ServeHTTP(0xc208045f2 [[Go 1.1 package http] 10.233.64.1:57861]
Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.016198    6851 server.go:281] HTTP InternalServerError: Internal Error: unable to find Docker container "9db4584cdd04a376db5c43197073e7e1e014d8c3213dbe6ace1e8abf54fe4de2"
Dec 16 15:21:00 kubenode4 kubelet[6851]: I1216 15:21:00.016272    6851 server.go:944] GET /stats/kube-system/influxdb-kj5j2/c92a2514-a405-11e5-8e30-525400ad92ba/grafana: (131.759��s) 500

我使用 heapster 来监视集群，似乎对“stats”的调用得到了答案，但由于未知原因，它被视为错误 500。

我在裸机服务器上运行一个集群。

kubernetes version : v1.1.3
docker: 1.9.1
os : debian 8.1

Answer 1

对我来说似乎是一个错误。我已提交 https://github.com/kubernetes/kubernetes/issues/18868