logging - 根据字段查询 GCP 记录器以获取不同的日志

Question

这是我的项目在 gcp 记录器中的示例日志

{
  "insertId": "________",
  "jsonPayload": {
    "stacktrace": "github.com_____",
    "level": "error",
    "msg": "could not update usage",
    "caller": "pkg/_______.go:118",
    "ts": ______.______,
    "requestID": "c7taeaa23akg00e8r0tg",
    "error": "write exception: write errors: [The field 'fieldName' must be an array but is of type null in document {_id: ObjectId('objectID001')}]"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "configuration_name": "configName",
      "service_name": "serviceName",
      "location": "us-central1",
      "project_id": "projectID",
      "revision_name": "revisionName"
    }
  },
  "timestamp": "2022-02-02T15:45:45.867386Z",
  "labels": {
    "instanceId": "____________"
  },
  "logName": "projects/_____/logs/run.googleapis.com%2Fstderr",
  "receiveTimestamp": "2022-02-02T15:45:45.967298989Z"
}

问题是有太多包含此确切内容的日志。 我的问题是，是否有查询或一组查询可用于检索具有相同 jsonPayload.error 的一个日志。

例如，如果有 6 个日志，其中 3 个具有相同的 jsonPayload.error，我需要实现的是获取 4 个日志，其中重复的日志将被取消，并且只有一个其中将与其他 3 个不同的日志一起出现在输出中。

Answer 1

有趣的问题。

Google 的 Logging query language是一种过滤机制。应用过滤器会减少返回的条目数，但不允许格式化结果以转换条目。

转换结果you're gonna to need a bigger boat ...我建议您考虑使用Google的Cloud SDK command-line tools aka gcloud .

使用此功能，您可以使用您使用日志查看器开发的查询来过滤日志:

gcloud logging read "${FILTER}" \
--project=${PROJECT}

而且(!)你可以transform (--format)结果:

gcloud logging read "${FILTER}" \
--format="${FORMAT}" \
--project=${PROJECT}"

NOTE gcloud's formatting does not appear to include unique|distinct functions and so we'll resort to using standard linux (sort|uniq) commands to achieve this.

作为一个示例，cloud.audit.logging 操作的通用查询:

PROJECT="..." # Your Project ID

# You would use "logName=\"projects/${PROJECT}/logs/run.googleapis.com%2Fstderr\""
FILTER="logName=\"projects/${PROJECT}/logs/cloudaudit.googleapis.com%2Factivity\""

# You would use "value(jsonPayload.error)"
FORMAT="value(operation.producer)"

gcloud logging read  "${FILTER}" \
--project=${PROJECT} \
--format="${FORMAT}" \
--limit=50 \
 > test.log

cat test.log | sort | uniq

产量:

cloudfunctions.googleapis.com
compute.googleapis.com
container.googleapis.com
k8s.io
servicemanagement.googleapis.com
serviceusage.googleapis.com

NOTE gcloud logging read "${FILTER}" submits the filter to the platform and is run "service-side". The results (which may be large) are then e.g. --format'ted client-side and this can be time/processor-consuming. In the example above, to save repeatedly retrieving the data from the server and then piping it through sort and uniq, it's more efficient to dump the logs into a file first. I've also used --limit to artificially restrict the number of results returned for testing purposes. You may want to use a time filter or something other constraint.

因为您引用了 jsonPayload，您也可以 gcloud ... --format=json(...) 来提取 JSON 格式的日志。如上所述，gcloud 包含格式化功能，但也如上所示，有时使用通用工具是最简单的。在这种情况下，jq提供了转换 JSON 的强大方法。

gcloud logging read  "${FILTER}" \
--project=${PROJECT} \
--format="${FORMAT}" \
--limit=50 \
> test.json

cat test.json | jq -r unique

产量:

[
  null,
  {
    "operation": {
      "producer": "cloudfunctions.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "compute.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "container.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "k8s.io"
    }
  },
  {
    "operation": {
      "producer": "servicemanagement.googleapis.com"
    }
  },
  {
    "operation": {
      "producer": "serviceusage.googleapis.com"
    }
  }
]