logging - 流利的匹配标签

Question

我试图弄清楚如何用Fluentd重命名字段(或创建具有相同值的新字段)

喜欢:

agent: Chrome ....

至:

agent: Chrome
user-agent: Chrome

但针对特定类型的日志，例如**nginx**。

我正在尝试使用record_reformer，但不适用于第二个过滤器:

<filter kubernetes.**.nginx-ingress-controller-**.log>
  @type parser
  format /^(?<host>[^ ]*) (?<domain>[^ ]*) \[(?<x_forwarded_for>[^\]]*)\] (?<server_port>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+[^\"])(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")? (?<request_length>[^ ]*) (?<request_time>[^ ]*) (?:\[(?<proxy_upstream_name>[^\]]*)\] )?(?<addr>[^ ]*) (?<response_length>[^ ]*) (?<response_time>[^ ]*) (?<status>[^ ]*)$/
  time_format %d/%b/%Y:%H:%M:%S %z
  key_name log
  types server_port:integer,code:integer,size:integer,request_length:integer,request_time:float,upstream_response_length:integer,upstream_response_time:float,upstream_status:integer
  reserve_data true
</filter>

<filter kubernetes.**>
    @type kubernetes_metadata
</filter>

<filter kubernetes.**>
 @type grep
 <regexp>
     key $.kubernetes.labels.fluentd
     pattern true
 </regexp>
</filter>

<filter kubernetes.**.deployment-name**>
  @type record_transformer
  <record>
    level ${record["Level"]}
  </record>
</filter>

<match kubernetes.**>
  @type elasticsearch
  include_tag_key true
  host "#{ENV['OUTPUT_HOST']}"
  port "#{ENV['OUTPUT_PORT']}"
  scheme "#{ENV['OUTPUT_SCHEME']}"
  reload_connections true
  logstash_format true
</match>

<match kubernetes.**>
  @type record_reformer
  remove_keys log,kubernetes
  tag mytag.generic
  <record>
    name ${record['kubernetes']['labels']['app']}
    namespace ${record['kubernetes']['namespace_name']}
  </record>
</match>

<match kubernetes.**api**>
  @type record_reformer
  remove_keys log,kubernetes
  tag mytag.api
  <record>
    user_agent ${record['req']['headers']['user-agent']}
  </record>
</match>

在这种情况下，它不会发送mytag.generic的日志

Answer 1

我有一个非常相似的用例，就像@embik所说，使用record_transformer似乎是一个更合适的选择。就我而言，我正在尝试在fluentd正在经历的json记录中对键进行小写转换，这是conf-

<filter kubernetes.**.deployment-name**>
  @type record_transformer
  <record>
    level ${record["Level"]}
  </record>
</filter>

希望能有所帮助。