从大表读取时出现 java.lang.OutOfMemoryError

Question

我正在尝试从 PostgreSQL 中的一个非常大的表 (300GB) 预先生成报告。我做这样的事情:

rs = stmt.executeQuery("SELECT * FROM tbl");
System.out.println("select all finished");
while (rs.next()) {
    /* generate report and save it in report table */
    /* generated reports are not in memory, 
     * They are saved in a summary table in each iteration */
}

当我启动应用程序时，它给出了 Exception in thread "main"java.lang.OutOfMemoryError: Java heap space。我尝试使用 stmt.setFetchSize(1000) 但它没有解决问题。

解决方案是什么？我在 Debian 6.0.5 和 openJDK 6 上使用 PostgreSQL 8.4.11。

[更新]

打印的堆栈跟踪显示 OutOfMemoryError 异常已在 rs = stmt.executeQuery("SELECT * FROM tbl"); 行中生成。也 System.out.println("select all finished"); 从不显示。

1. 我在 autocommit 模式下运行。
2. stmt.getResultSetConcurrency() 返回 1007。
3. stmt.getResultSetHoldability() 返回 2。
4. rs.getType() 返回 1003。

Answer 1

问题可能是 PostgreSQL 只在少数情况下使用 fetchSize。请参阅:http://jdbc.postgresql.org/documentation/91/query.html#fetchsize-example

• The connection to the server must be using the V3 protocol. This is the default for (and is only supported by) server versions 7.4 and later.
• The Connection must not be in autocommit mode. The backend closes cursors at the end of transactions, so in autocommit mode the backend will have closed the cursor before anything can be fetched from it.
• The Statement must be created with a ResultSet type of ResultSet.TYPE_FORWARD_ONLY. This is the default, so no code will need to be rewritten to take advantage of this, but it also means that you cannot scroll backwards or otherwise jump around in the ResultSet.
• The query given must be a single statement, not multiple statements strung together with semicolons.

因此，如果您在自动提交中执行此操作，或者使用 TYPE_FORWARD_ONLY 以外的结果集类型，PostgreSQL 将获取所有行。同时查看 PostgreSQL JDBC 9.0-801 驱动程序的源代码，它看起来像使用可持有的结果集也将使其获取所有行。