csv - 如何禁用流利的csv格式程序中包含定界符的字段值周围的引号？

Question

我使用fluentd从golang应用程序收集CSV格式的日志。
这是流利的conf文件

<source>
    @type  forward
    @id    app_logs
    @label @mainstream
    port  24224
</source>



<label @mainstream>
   <match **>
      @type file
      @id   v6_logs
    <format>
      @type csv
      fields log_version,day_time,event_id,request_id,app_id,app_version,account_id,country_id,server_name,remote_ip,process_id,thread_id,item_id,message,parameters
      force_quotes false
    </format>
      path         /fluentd/log/app.log
      append       true
  </match>
</label>

我使用Fluent golang客户端从应用程序https://github.com/fluent/fluent-logger-golang写入日志

logger, _ := fluent.New(fluent.Config{FluentPort: 24224, FluentHost: "fluentd"})
defer logger.Close()
tag := "web"
var data = map[string]interface{}{
    "log_version": 6,
    "day_time":    time.Now().UTC().String(),
    "event_id":    1700,
    "request_id":  "54321",
    "account_id":  12345,
    "server_name": hostname,
    "process_id":  os.Getpid(),
    "message":     "Test Message(param1; param2)",
    "parameters":  "value1, value2",
}

error := logger.Post(tag, data)

输出结果是这样的。

6,2020-09-23 23:48:44.5731073 +0000 UTC,1700,54321,,,123467,,cabf36399a5c,,1,,,Test Message(param1; param2),"value1,value2"

我如何获取要在“value1，value2”周围删除的引号(使其作为单独的字段出现)。

Answer 1

我通过按照https://docs.fluentd.org/v/0.12/developer/plugin-development此处的说明在ruby中编写自定义CSV格式器插件并将文件放置在
路径/ etc / fluent / plugin /

   require 'fluent/plugin/formatter'
        
   module Fluent::Plugin
   class MyCSVFormatter < Formatter
       # Register MyCSVFormatter as 'my_csv'.
       Fluent::Plugin.register_formatter('my_csv', self)
       config_param :csv_fields, :array, value_type: :string

       # This method does further processing. Configuration parameters can be
       # accessed either via `conf` hash or member variables.
       def configure(conf)
         super
       end

       # This is the method that formats the data output.
      def format(tag, time, record)
         values = []

         # Look up each required field and collect them from the record
         @csv_fields.each do |field|
            if field == "parameters"
                parametervalues = []
                parametervalues = record[field].split(",").map(&:strip)
                values.push(*parametervalues)
                next
            end

            v = record[field]
            values << v.to_s
          end

         # Output by joining the fields with a comma
         values.join(',') + "\n"
      end
    end
   end

更新了fluentd conf文件以使用自定义格式

<label @mainstream>
   <match **>
      @type file
      @id   v6_logs
    <format>
      @type my_csv
      csv_fields log_version,day_time,event_id,request_id,app_id,app_version,account_id,country_id,server_name,remote_ip,process_id,thread_id,item_id,message,parameters
    </format>
      path         /fluentd/log/app.log
      append       true
  </match>
</label>

这将产生所需的输出

6,2020-09-24 06:27:52.1826684 +0000 UTC,1700,54321,,,123467,,hostname,,1,,,Test Message(param1; param2),value1,value2