kubernetes - Prometheus没有收集Pod指标

Question

我将Prometheus和Grafana部署到了集群中。

当我打开仪表板时，没有获得有关pod CPU使用率的数据。

当我检查Prometheus UI时，它将Pod 0/0向上显示，但是群集中有许多Pod正在运行。

可能是什么原因？我有在所有节点上运行的节点导出器。



我正在为此获取kube-state-metrics，

I0218 14:52:42.595711       1 builder.go:112] Active collectors: configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,jobs,limitranges,namespaces,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets
I0218 14:52:42.595735       1 main.go:208] Starting metrics server: 0.0.0.0:8080

这是我的Prometheus配置文件:
https://gist.github.com/karthikeayan/41ab3dc4ed0c344bbab89ebcb1d33d16

我可以找到以下数据:

http://localhost:8080/api/v1/nodes/<my_worker_node>/proxy/metrics/cadvisor

Answer 1

正如karthikeayan在评论中提到的那样:

ok, i found something interesting in the values.yaml comments, prometheus.io/scrape: Only scrape pods that have a value of true, when i remove this relabel_config in k8s configmap, i got the data in prometheus ui.. unfortunately k8s configmap doesn't have comments, i believe helm will remove the comments before deploying it.

只是为了澄清:

kube-state-metrics vs. metrics-server

The metrics-server is a project that has been inspired by Heapster and is implemented to serve the goals of the Kubernetes Monitoring Pipeline. It is a cluster level component which periodically scrapes metrics from all Kubernetes nodes served by Kubelet through Summary API. The metrics are aggregated, stored in memory and served in Metrics API format. The metric-server stores the latest values only and is not responsible for forwarding metrics to third-party destinations.

kube-state-metrics is focused on generating completely new metrics from Kubernetes' object state (e.g. metrics based on deployments, replica sets, etc.). It holds an entire snapshot of Kubernetes state in memory and continuously generates new metrics based off of it. And just like the metric-server it too is not responsibile for exporting its metrics anywhere.

Having kube-state-metrics as a separate project also enables access to these metrics from monitoring systems such as Prometheus.