kubernetes - 我们如何在 filebeat kubernetes 中过滤命名空间？

Question

我正在设置管道以将 kubernetes pods 日志发送到弹性集群。我已经在我的集群中安装了 filebeat 作为 deamonset(流:stdout)并将输出连接到 logstash。 Beats 与 logstash 连接没有问题，现在我想要来自应用程序命名空间的日志，而不是来自集群中所有命名空间的日志。有人可以指导我如何在beat dn中过滤它，也可以在es中查看来自json的源消息吗？
这是我的配置:

data:
  kubernetes.yml: |-
    - type: docker
      containers:
        path: "/var/lib/docker/containers"
        stream: "stdout"
        ids: "*"
        multiline.pattern: '^\s'
        multiline.match: after
      fields:
         logtype: container
      multiline:
         pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
         negate: true
         match: after
      ignore_older: 1h
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
        - decode_json_fields:
            fields: ["log"]
            overwrite_keys: true
            target: ""

kibana 中的输出:

{
  "_index": "filebeat-6.8.4-2020.03.06",
  "_type": "doc",
  "_id": "vHkzsHABJ57Tsdxxxxx",
  "_version": 1,
  "_score": null,
  "_source": {
    "log": {
      "file": {
        "path": "/var/lib/docker/containers/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c/sdnksdsdlsdnfsdlfslfnsdslfnsnlnflksdnflkdsfnsdflsdfndslffndslf-json.log"
      }
    },
    "tags": [
      "beats_input_codec_plain_applied",
      "_grokparsefailure"
    ],
    "input": {
      "type": "docker"
    },
    "@version": "1",
    "prospector": {
      "type": "docker"
    },
    "beat": {
      "version": "6.8.4",
      "name": "filebeat-vtp2f",
      "hostname": "filebeat-vtp2f"
    },
    "host": {
      "name": "filebeat-vtp2f"
    },
    "offset": 5798785,
    "stream": "stdout",
    "fields": {
      "logtype": "container"
    },
    "kubernetes": {
      "node": {
        "name": "k8-test-22313607-0"
      },
      "labels": {
        "version": "v1",
        "kubernetes": {
          "io/cluster-service": "true"
        },
        "controller-revision-hash": "6b56cfcb69",
        "pod-template-generation": "1",
        "k8s-app": "fluent"
      },
      "container": {
        "name": "fluentd"
      },
      "pod": {
        "uid": "72c50b54-5ef0-11ea-83e1-26018882335d",
        "name": "fluent-4lft2"
      },
      "namespace": "fluentd"
    },
    "source": "/var/lib/docker/containers/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c/aa54562be9448183d69d8d2e1953e74560309176f044aed23484ac9e3260982c-json.log",
    "@timestamp": "2020-03-06T14:15:18.561Z"
  },
  "fields": {
    "@timestamp": [
      "2020-03-06T14:15:18.561Z"
    ]
  },
  "highlight": {
    "prospector.type": [
      "@kibana-highlighted-field@docker@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1583504118561
  ]
}

Answer 1

如何删除一些命名空间，我在这里记录:https://ezyforanykey.blogspot.com/2020/11/filebeat-exclude-kubernetes-namespace.html
示例如下:

- type: container
      paths:
        - /var/log/containers/*.log
      exclude_files:
        - /var/log/containers/java.*
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_event.when:
            or:
            - equals:
                kubernetes.namespace: "kube-system"
            - equals:
                kubernetes.namespace: "calico-system"