performance - sqlite.fetchall() 慢是正常的吗？

Question

我有一个从两个内部连接表中进行选择的 SQL 查询。 select 语句大约需要 50 秒。然而，fetchall() 花费了 788 秒才得到 981 个结果:

time0 = time.time()
self.cursor.execute("SELECT spectrum_id, feature_table_id "+
                    "FROM spectrum AS s "+
                    "INNER JOIN feature AS f "+
                    "ON f.msrun_msrun_id = s.msrun_msrun_id "+
                    "INNER JOIN (SELECT feature_feature_table_id, min(rt) AS rtMin, max(rt) AS rtMax, min(mz) AS mzMin, max(mz) as mzMax "+
                                 "FROM convexhull GROUP BY feature_feature_table_id) AS t "+
                    "ON t.feature_feature_table_id = f.feature_table_id "+
                    "WHERE s.msrun_msrun_id = ? "+
                    "AND s.scan_start_time >= t.rtMin "+
                    "AND s.scan_start_time <= t.rtMax "+
                    "AND base_peak_mz >= t.mzMin "+
                    "AND base_peak_mz <= t.mzMax", spectrumFeature_InputValues)
print 'query took:',time.time()-time0,'seconds'
    
time0 = time.time()
spectrumAndFeature_ids = self.cursor.fetchall()      
print time.time()-time0,'seconds since to fetchall'

fetchall() 需要这么长时间是有原因的吗？正在做:

while 1:
    info = self.cursor.fetchone()
    if info:
        <do something>
    else:
        break

与以下一样慢:

allInfo = self.cursor.fetchall()         
for info in allInfo:
    <do something>

Answer 1

默认fetchall()就像循环 fetchone() 一样慢由于arraysize Cursor的对象被设置为 1。

为了加快速度，您可以循环 fetchmany() ，但要看到性能提升，您需要为其提供大于 1 的大小参数，否则它将按批 arraysize 获取“很多”。 ，即 1。

很可能只需提高 arraysize 的值即可获得性能增益。 ，但我没有这样做的经验，所以您可能想首先通过执行以下操作来尝试:

>>> import sqlite3
>>> conn = sqlite3.connect(":memory:")
>>> cu = conn.cursor()
>>> cu.arraysize
1
>>> cu.arraysize = 10
>>> cu.arraysize
10

更多关于上述内容:http://docs.python.org/library/sqlite3.html#sqlite3.Cursor.fetchmany