mysql - Elasticsearch JDBC River MySQL连接超时

Question

我使用以下脚本设置了我的河流:

curl -XPUT 'localhost:9200/_river/foo/_meta' -d '{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://...:3306/....",
        "user" : "...",
        "password" : "...",
        "sql" : "SELECT v.id as _id,v.name,v.entrydate, v.link, v.html,v.created AS _created,vc.name AS company, vp.name AS position FROM foo v LEFT JOIN foocompany vc ON vc.id=v.company LEFT JOIN fooposition vp ON vp.id=v.position ",
        "fetchsize" : 100,
        "bulk_size" : 100,
        "max_bulk_requests" : 2,
        "bulk_flush_interval" : "30s",
        "strategy": "simple",
        "poll": "30s",
        "autocommit": true
    }
}'

这条河流运行一段时间后，我得到一个异常，这可能是因为 MySQL 服务器本身的配置:

[2014-11-27 16:54:02,301][ERROR][org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow] com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:231)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.move(SimpleRiverFlow.java:129)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverFlow.run(SimpleRiverFlow.java:88)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 10 milliseconds ago.  The last packet sent successfully to the server was 52,296 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:928)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2053)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:406)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:385)
        at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:163)
        at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7472)
        at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:919)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.close(SimpleRiverSource.java:613)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.execute(SimpleRiverSource.java:263)
        at org.xbib.elasticsearch.river.jdbc.strategy.simple.SimpleRiverSource.fetch(SimpleRiverSource.java:227)
        ... 3 more
Caused by: java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3166)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3620)
        ... 15 more

这里的问题是在我的设置中重新配置 MySQL 不是一个选项。 唉，我必须在其他地方寻找选择。

1. ElasticSearch 是否可以重新建立连接 一段时间后然后恢复索引？
2. 还有其他的吗 在不使用 JDBC 的情况下桥接 ElasticSearch 和 MySQL 的方法 河流？

Answer 1

我对弹性河流有很多很多的头痛。不仅仅是 JDBC 的，还有自定义编写的河流、网络爬虫河流等。

一个重要的注意事项是河流很快就会被弃用。 (Preferred method of indexing bulk data into ElasticSearch?)

我遇到的问题之一是，当 Elastic 重新启动时，河流并不总是可靠地启动。有时河流根本没有源头，有时却有源头。非常令人沮丧。

Elastic 的官方建议是将流程移至 Elastic 之外并将数据注入(inject)。

我已将所有 JDBC 河流替换为在 Linux 上运行的小型 C# 应用程序，作为同一 Elastic 服务器上的 cron 作业。效果很好，而且更可靠，更容易启动/重新启动。在 Elastic 中重新创建河流对我来说总是很痛苦。