kubernetes - GCP-记录k8s:向Stackdriver googleapi发送请求时出错:错误400:无法编写一个或多个TimeSeries

Question

我最近在GCP实例上添加了GKE实例的日志记录。如今，以下错误每秒发生3次，因此会产生大量错误。不幸的是，由于日志中存在大量错误，所有重要错误都会丢失。以下JSON是这些错误之一:

{
  "insertId": "42",
  "jsonPayload": {
    "pid": "1",
    "source": "stackdriver.go:60",
    "message": "Error while sending request to Stackdriver googleapi: Error 400: One or more TimeSeries could not be written: Unknown metric:
kubernetes.io/internal/addons/workload_identity/go_gc_duration_seconds_count: timeSeries[31]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_gc_duration_seconds_sum: timeSeries[4]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_goroutines: timeSeries[0]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_info: timeSeries[47]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_alloc_bytes: timeSeries[55]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_alloc_bytes_total: timeSeries[40]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_buck_hash_sys_bytes: timeSeries[13]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_frees_total: timeSeries[2]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_gc_cpu_fraction: timeSeries[56]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_gc_sys_bytes: timeSeries[19]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_alloc_bytes: timeSeries[46]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_idle_bytes: timeSeries[32]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_inuse_bytes: timeSeries[42]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_objects: timeSeries[1]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_released_bytes: timeSeries[8]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_sys_bytes: timeSeries[43]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_last_gc_time_seconds: timeSeries[33]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_lookups_total: timeSeries[34]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mallocs_total: timeSeries[3]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mcache_inuse_bytes: timeSeries[18]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mcache_sys_bytes: timeSeries[11]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mspan_inuse_bytes: timeSeries[38]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mspan_sys_bytes: timeSeries[23]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_next_gc_bytes: timeSeries[10]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_other_sys_bytes: timeSeries[16]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_stack_inuse_bytes: timeSeries[17]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_stack_sys_bytes: timeSeries[12]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_sys_bytes: timeSeries[21]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_threads: timeSeries[41]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_cpu_seconds_total: timeSeries[20]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_max_fds: timeSeries[22]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_open_fds: timeSeries[9]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_resident_memory_bytes: timeSeries[39]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_start_time_seconds: timeSeries[45]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_virtual_memory_bytes: timeSeries[30]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_virtual_memory_max_bytes: timeSeries[44]; Unknown metric: kubernetes.io/internal/addons/workload_identity/promhttp_metric_handler_requests_in_flight: timeSeries[7]; Unknown metric: kubernetes.io/internal/addons/workload_identity/promhttp_metric_handler_requests_total: timeSeries[35-37]; Value type for metric kubernetes.io/internal/addons/workload_identity/metadata_server_build_info must be DOUBLE, but is INT64.: timeSeries[48], badRequest"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "cluster_name": "cluster-a",
      "location": "europe-west3",
      "pod_name": "prometheus-to-sd-jcmwn",
      "project_id": "my-nice-project-id",
      "container_name": "prometheus-to-sd-new-model",
      "namespace_name": "kube-system"
    }
  },
  "timestamp": "2020-07-30T06:26:01.784963Z",
  "severity": "ERROR",
  "labels": {
    "k8s-pod/pod-template-generation": "1",
    "k8s-pod/controller-revision-hash": "7984bf4f95",
    "k8s-pod/k8s-app": "prometheus-to-sd"
  },
  "logName": "projects/my-nice-project-id/logs/stderr",
  "sourceLocation": {
    "file": "stackdriver.go",
    "line": "60"
  },
  "receiveTimestamp": "2020-07-30T06:26:03.411798926Z"
}

此行为有什么问题，我该如何解决？

Answer 1

在启用了Workload Identity功能的GKE群集中，它看起来像是一个错误。
该错误在带有Workload Identity的1.14.10-gke.42中为我重现，但与在1.15.12-gke.2版本中部署的GKE集群一样，可以正常使用。
GitHub中有一个open issue。如果您无法升级集群版本，建议您与Google Cloud支持小组联系，并要求他们提供建议的缓解措施(尽管他们可能还会指示您升级集群版本)。