jdbc - clojure sqlkorma 库 : out of memory error

Question

我正在做我认为相当简单的任务:使用 sqlkorma 库 ( http://sqlkorma.com ) 运行一个 sql 查询(超过大约 65K 行数据)，并以某种方式对每一行进行转换，然后编写到 CSV 文件。考虑到我有一台 8GB 的​​笔记本电脑，我真的不认为 65K 行那么大，但我还假设一个 sql 结果集将被延迟获取，因此整个事情永远不会同时保存在内存中。所以当我最终得到这个堆栈跟踪时，我真的很惊讶:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at clojure.lang.PersistentHashMap$BitmapIndexedNode.assoc(PersistentHashMap.java:777)
at clojure.lang.PersistentHashMap.createNode(PersistentHashMap.java:1101)
at clojure.lang.PersistentHashMap.access$600(PersistentHashMap.java:28)
at clojure.lang.PersistentHashMap$BitmapIndexedNode.assoc(PersistentHashMap.java:749)
at clojure.lang.PersistentHashMap$TransientHashMap.doAssoc(PersistentHashMap.java:269)
at clojure.lang.ATransientMap.assoc(ATransientMap.java:64)
at clojure.lang.PersistentHashMap.create(PersistentHashMap.java:56)
at clojure.lang.PersistentHashMap.create(PersistentHashMap.java:100)
at clojure.lang.PersistentArrayMap.createHT(PersistentArrayMap.java:61)
at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:201)
at clojure.lang.PersistentArrayMap.assoc(PersistentArrayMap.java:29)
at clojure.lang.RT.assoc(RT.java:702)
at clojure.core$assoc.invoke(core.clj:187)
at clojure.core$zipmap.invoke(core.clj:2715)
at clojure.java.jdbc$resultset_seq$thisfn__204.invoke(jdbc.clj:243)
at clojure.java.jdbc$resultset_seq$thisfn__204$fn__205.invoke(jdbc.clj:243)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
at clojure.lang.Cons.next(Cons.java:39)
at clojure.lang.PersistentVector.create(PersistentVector.java:51)
at clojure.lang.LazilyPersistentVector.create(LazilyPersistentVector.java:31)
at clojure.core$vec.invoke(core.clj:354)
at korma.db$exec_sql$fn__343.invoke(db.clj:203)
at clojure.java.jdbc$with_query_results_STAR_.invoke(jdbc.clj:669)
at korma.db$exec_sql.invoke(db.clj:202)
at korma.db$do_query$fn__351.invoke(db.clj:225)
at clojure.java.jdbc$with_connection_STAR_.invoke(jdbc.clj:309)
at korma.db$do_query.invoke(db.clj:224)
at korma.core$exec.invoke(core.clj:474)
at db$query_db.invoke(db.clj:23)
at main$_main.doInvoke(main.clj:32)
at clojure.lang.RestFn.applyTo(RestFn.java:137)

据我从堆栈中可以看出，它还没有超出查询代码(意味着它根本没有达到我的转换/写入 CSV 代码)。如果重要的话，我的 sql 相当简单，基本上是 SELECT * FROM my_table WHERE SOME_ID IS NOT NULL AND ROWNUM < 65000 ORDER BY some_id ASC .这是 oracle(解释上面的 rownum)，但我认为这不重要。

编辑:

代码示例:

(defmacro query-and-print [q] `(do (dry-run ~q) ~q))
(defn query-db []  
    (query-and-print 
        (select my_table 
            (where (and (not= :MY_ID "BAD DATA")
                        (not= :MY_ID nil)
                        (raw (str "rownum < " rows))))
            (order :MY_ID :asc))))

; args contains rows 65000, and configure-app sets up the jdbc
; connection string, and sets a var with rows value
(defn -main [& args]
    (when (configure-app args) 
        (let [results (query-db)
              dedup (dedup-with-merge results)]
            (println "Result size: " (count results))
            (println "Dedup size: " (count dedup))
            (to-csv "target/out.csv" (transform-data dedup)))))

Answer 1

clojure.java.sql 创建惰性序列:

(defn resultset-seq
"Creates and returns a lazy sequence of maps corresponding to
 the rows in the java.sql.ResultSet rs. Based on clojure.core/resultset-seq
 but it respects the current naming strategy. Duplicate column names are
 made unique by appending _N before applying the naming strategy (where
 N is a unique integer)."
[^ResultSet rs]
(let [rsmeta (.getMetaData rs)
      idxs (range 1 (inc (.getColumnCount rsmeta)))
      keys (->> idxs
             (map (fn [^Integer i] (.getColumnLabel rsmeta i)))
             make-cols-unique
             (map (comp keyword *as-key*)))
      row-values (fn [] (map (fn [^Integer i] (.getObject rs i)) idxs))
      rows (fn thisfn []
             (when (.next rs)
               (cons (zipmap keys (row-values)) (lazy-seq (thisfn)))))]
  (rows)))

Korma 通过将每一行放入一个向量来完全实现序列:

(defn- exec-sql [{:keys [results sql-str params]}]
(try
(case results
  :results (jdbc/with-query-results rs (apply vector sql-str params)
             (vec rs))
  :keys (jdbc/do-prepared-return-keys sql-str params)
  (jdbc/do-prepared sql-str params))
(catch Exception e
  (handle-exception e sql-str params))))