nginx - 在普罗米修斯中看不到 nginx 入口指标

Question

我正在尝试让 prometheus 监控我的 ingress nginx 部署工作。我可以在 prometheus ui 中查看默认指标，但与 nginx 无关。

• 我已经通过集群监控部署了 prometheus-operator(同时更新了 serviceMonitorSelector 和 serviceMonitorNamespaceSelector 值):https://github.com/carlosedp/cluster-monitoring
• 我已经通过官方 helm chart 部署了 ingress-nginx:https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx

kubectl get prometheus -A

...
      runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector:
      matchLabels:
        release: prometheus-operator
    serviceMonitorSelector:
      matchLabels:
        release: prometheus-operator
    version: v2.19.1
...

kubectl 获取 ns nginx

apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-02-08T02:37:35Z"
  labels:
    release: prometheus-operator
  managedFields:
  - apiVersion: v1
...

kubectl get servicemonitor nginx-ingress-nginx-controller -n monitoring -o yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: nginx
  creationTimestamp: "2021-02-14T19:09:01Z"
  generation: 5
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
    release: prometheus-operator
  managedFields:
...
spec:
  endpoints:
  - interval: 30s
    port: metrics
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      release: prometheus-operator

kubectl get svc nginx-ingress-nginx-controller-metrics -n nginx -o yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: nginx
    prometheus.io/port: "10254"
    prometheus.io/scrape: "true"
  creationTimestamp: "2021-02-11T03:58:38Z"
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
    release: prometheus-operator
...
spec:
  clusterIP: 10.43.90.125
  clusterIPs:
  - 10.43.90.125
  ports:
  - name: metrics
    port: 9913
    protocol: TCP
    targetPort: metrics
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: ClusterIP

更新 2/17: kubectl get prometheus -n monitoring -o yaml

的完整输出

➜  ~ kubectl get -n monitoring prometheus -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"monitoring.coreos.com/v1","kind":"Prometheus","metadata":{"annotations":{},"labels":{"prometheus":"k8s"},"name":"k8s","namespace":"monitoring"},"spec":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"prometheus","operator":"In","values":["k8s"]}]},"namespaces":["monitoring"],"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"alerting":{"alertmanagers":[{"name":"alertmanager-main","namespace":"monitoring","port":"web"}]},"externalUrl":"http://prometheus.192.168.1.15.nip.io","image":"prom/prometheus:v2.19.1","nodeSelector":{"kubernetes.io/os":"linux"},"podMonitorNamespaceSelector":{},"podMonitorSelector":{},"replicas":1,"resources":{"requests":{"memory":"400Mi"}},"retention":"15d","ruleSelector":{"matchLabels":{"prometheus":"k8s","role":"alert-rules"}},"scrapeInterval":"30s","scrapeTimeout":"30s","securityContext":{"fsGroup":2000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"prometheus-k8s","serviceMonitorNamespaceSelector":{},"serviceMonitorSelector":{"matchLabels":{"release":"prometheus-operator"}},"version":"v2.19.1"}}
    creationTimestamp: "2021-02-13T04:13:49Z"
    generation: 4
    labels:
      prometheus: k8s
    managedFields:
    - apiVersion: monitoring.coreos.com/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:prometheus: {}
        f:spec:
          .: {}
          f:affinity:
            .: {}
            f:podAntiAffinity:
              .: {}
              f:preferredDuringSchedulingIgnoredDuringExecution: {}
          f:alerting:
            .: {}
            f:alertmanagers: {}
          f:externalUrl: {}
          f:image: {}
          f:nodeSelector:
            .: {}
            f:kubernetes.io/os: {}
          f:podMonitorNamespaceSelector: {}
          f:podMonitorSelector: {}
          f:replicas: {}
          f:resources:
            .: {}
            f:requests:
              .: {}
              f:memory: {}
          f:retention: {}
          f:ruleSelector:
            .: {}
            f:matchLabels:
              .: {}
              f:prometheus: {}
              f:role: {}
          f:scrapeInterval: {}
          f:scrapeTimeout: {}
          f:securityContext:
            .: {}
            f:fsGroup: {}
            f:runAsNonRoot: {}
            f:runAsUser: {}
          f:serviceAccountName: {}
          f:serviceMonitorNamespaceSelector: {}
          f:serviceMonitorSelector:
            .: {}
            f:matchLabels:
              .: {}
              f:release: {}
          f:version: {}
      manager: kubectl-client-side-apply
      operation: Update
      time: "2021-02-15T15:07:05Z"
    name: k8s
    namespace: monitoring
    resourceVersion: "1321004"
    uid: a9829d00-2765-416b-b068-2c1e57623832
  spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: prometheus
                operator: In
                values:
                - k8s
            namespaces:
            - monitoring
            topologyKey: kubernetes.io/hostname
          weight: 100
    alerting:
      alertmanagers:
      - name: alertmanager-main
        namespace: monitoring
        port: web
    externalUrl: http://prometheus.192.168.1.15.nip.io
    image: prom/prometheus:v2.19.1
    nodeSelector:
      kubernetes.io/os: linux
    podMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    replicas: 1
    resources:
      requests:
        memory: 400Mi
    retention: 15d
    ruleSelector:
      matchLabels:
        prometheus: k8s
        role: alert-rules
    scrapeInterval: 30s
    scrapeTimeout: 30s
    securityContext:
      fsGroup: 2000
      runAsNonRoot: true
      runAsUser: 1000
    serviceAccountName: prometheus-k8s
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector:
      matchLabels:
        release: prometheus-operator
    version: v2.19.1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

我也尝试过让监控与其他部署一起工作，但无济于事(所以我显然做错了什么)，但这是我的普罗米修斯日志:

➜  ~ kubectl logs prometheus-k8s-0  -n monitoring -c prometheus --tail 25
level=error ts=2021-02-17T15:29:18.588Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:29:25.978Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:29:43.562Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:29:46.748Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:29:56.841Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:30:05.943Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:14.735Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:30:18.489Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:28.171Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:30:30.412Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:57.878Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:30:59.128Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:31:15.689Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:31:21.136Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:31:27.498Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:31:36.739Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:31:54.742Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:03.270Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:13.738Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:20.258Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""
level=error ts=2021-02-17T15:32:24.315Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"pihole\""
level=error ts=2021-02-17T15:32:44.099Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:361: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:32:58.299Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"services\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:33:03.925Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
level=error ts=2021-02-17T15:33:05.050Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:385: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"mqtt\""

Answer 1

经过大量研究，我遇到的问题涉及 ClusterRole 定义。 prometheus 日志表明我试图监控的许多命名空间存在权限问题，因此我最终不得不向我的 ClusterRole 添加权限以包含 pods 等资源， services 和 endpoints，并包含 list 和 watch 等动词:

 apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    name: prometheus-k8s
  rules:
  - apiGroups:
    - ""
    resources:
    - nodes/metrics
    - nodes
    - services
    - endpoints
    - pods
    verbs:
    - get
    - list
    - watch
  - nonResourceURLs:
    - /metrics
    verbs:
    - get