我在 MySQL 中有一个表,我想将其导入 Elasticsearch
作为示例,数据如下所示
team buyer
==== ======
one Q76876
one Q66567
one T99898
two Q45456
two S77676
我想使用logstash将其导入elasticsearch并创建一个如下所示的索引
{
"id": "one",
"team": one,
"buyers": ["Q76876", "Q66567", "T99898"]
},
{
"id": "two",
"team": "two",
"buyers": ["Q45456", "S77676"]
}
我将如何编写 .conf 脚本来实现此目的?
最佳答案
Logstash 将事件放入索引中,除非您应用一些过滤器。你的情况看起来很简单。如果您将 sql 查询格式化为以您需要的格式返回数据,那么您不需要应用任何过滤器,只需连接数据库和 sql 查询即可在logstash 配置中运行,并将输出作为 Elasticsearch 索引。
例如:
MySql 查询看起来像这样:(我不擅长 mysql,下面只是给出一个想法 - 请验证它是否有效)
SELECT team as id,
team,
GROUP_CONCAT(DISTINCT buyer SEPARATOR ', ') as buyers
FROM tablename GROUP BY team
这将返回类似以下内容:
+-----+------+------------------------+
| id | team | buyers |
+-----+------+------------------------+
| one | one | Q76876, Q66567, T99898 |
| two | two | Q45456, S77676 |
+-----+------+------------------------+
logstash 配置看起来就像:
input {
jdbc {
jdbc_driver_library => "${DATABASE_DRIVER_PATH}"
jdbc_driver_class => "${DATABASE_DRIVER_PATH}"
jdbc_connection_string => "{CONNECTIONSTRING}"
jdbc_user => "${DATABASE_USERNAME}"
jdbc_password => "${DATABASE_PASSWORD}"
statement_filepath => "${LOGSTASH_SQL_FILEPATH}" #this will be the sql written above
}
}
filter {
}
output {
elasticsearch {
action => "index"
hosts => ["${ELASTICSEARCH_HOST}"]
user => "${ELASTICSEARCH_USER}"
password => "${ELASTICSEARCH_PASSWORD}"
index => "${INDEX_NAME}"
document_type => "doc"
document_id => "%{id}"
}
stdout { codec => rubydebug }
stdout { codec => dots }
}
关于mysql - Logstash - 从关系数据库组创建术语,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/50973733/