java - 从 MySql 读取大型集合

Question

有一个表event_logs，表中约有1600万条条目。数据库是MySQL。

在应用程序中，需要逐一读取所有行[特定于我的应用程序要求]

我只是计算检索速度

statement = connection.createStatement(java.sql.ResultSet.TYPE_FORWARD_ONLY,
                        java.sql.ResultSet.CONCUR_READ_ONLY);
statement.setFetchSize(Integer.MIN_VALUE);   
statement.executeQuery("select * from test.event_logs");

long startTime = 0;
long endTime = 0;
int counter = 0;

while (resultSet.next()) {  

if (counter == 0){
    startTime = System.currentTimeMillis();         
}

if(counter == 100000){
    endTime = System.currentTimeMillis();           
    long diff = endTime - startTime;
    System.out.println(diff);
    long diffMinutes = diff / (60 * 1000) % 60;
    System.out.println("diffMinutes" + " " + diffMinutes);
    counter = 0;        
}
counter++;
}

对于第一个 100k 记录，大约花费了不到一分钟的时间 接下来的 100k 条记录大约需要一分钟 接下来的 100k 条记录大约需要 2 分钟 接下来的 10 万条记录大约需要 11 分钟

之后我遇到了以下异常

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 195 milliseconds ago.  The last packet sent successfully to the server was 152,824 milliseconds ago.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3673)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3562)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4113)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:927)
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:2006)
    at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:420)
    at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:399)
    at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:7012)
    at com.google.io.InsertCampaignLogs.main(InsertCampaignLogs.java:55)
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
    at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
    at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3116)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3629)
    ... 8 more

我的问题

1.为什么相同数量的记录检索结果时间会增加

2.为什么我在运行查询的过程中遇到通信失败异常

提前致谢

Answer 1

您可能需要增加事务长度或读取垃圾中的行。读取前 10k，然后读取接下来的 10k，每个都在单独的语句中。