amazon-web-services - 用 Go 插入 firehose

Question

我有以下 JSON 文件

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose."
}

我有以下 Go 函数

func insertIntoFireHose(sess *session.Session, hoseName string) {
    svc := firehose.New(sess, aws.NewConfig().WithRegion("us-east-1"))
    //firehoseData := getObjectFromS3Bucket(sess)
    firehoseData, _ := os.ReadFile("/temp/test.json")

    var rec firehose.Record

    var recInput firehose.PutRecordInput

    dataJson, _ := json.Marshal(firehoseData)
    rec.SetData(dataJson)
    recInput.SetDeliveryStreamName(hoseName)
    recInput.SetRecord(&rec)

    res, err1 := svc.PutRecord(&recInput)

    if err1 != nil {
        log.Fatal(err1)
    }
    fmt.Println(res)

}

我想做的是获取一个文件并将其插入到消防软管中，但我收到此错误消息:

{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":
{"type":"not_x_content_exception",
"reason":"not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}}

我不太确定我做错了什么。

更改记录以直接从文件获取数据会返回此错误:

One or more records are malformed. Please ensure that each record is single valid JSON object and that it does not contain newlines.

Answer 1

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose.",
}

我认为这不是有效的 JSON，它在第三行有一个尾随逗号。

这是有效的

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose."
}

并且 firehoseData 已经是 []byte，所以我认为您不需要再次 json.Marshal 。

这是编码(marshal)的结果

代码:

package main

import (
    "encoding/json"
    "fmt"
    "os"
)

func main() {
    firehoseData, _ := os.ReadFile("./file.json") // same value

    fmt.Printf("%+v\n", string(firehoseData))

    test, err := json.Marshal(firehoseData)

    fmt.Printf("%+v\n", string(test))
    fmt.Printf("%+v\n", err)
}

输出:

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose.",   
}
"ew0KICAgICJAdGltZXN0YW1wIjogIjIwMjEtMTEtMTlUMjE6MzI6NTUuMTk2WiIsDQogICAgIkB2ZXJzaW9uIjogIjEiLA0KICAgICJtZXNzYWdlIjogIk1hbnVhbCB0ZXN0IHRvIGZpcmVob3NlLiIsDQp9"
<nil>