我正在编写一个脚本来 SELECT 查询数据库并解析大约 33,000 条记录。不幸的是,我在 cursor.fetchone()
/cursor.fetchall()
阶段遇到了问题。
我首先尝试像这样一次通过游标迭代一条记录:
# Run through every record, extract the kanji, then query for FK and weight
printStatus("Starting weight calculations")
while True:
# Get the next row in the cursor
row = cursor.fetchone()
if row == None:
break
# TODO: Determine if there's any kanji in row[2]
weight = float((row[3] + row[4]))/2
printStatus("Weight: " + str(weight))
根据 printStatus
的输出(它打印出时间戳以及传递给它的任何字符串),脚本处理每一行大约需要 1 秒。这让我相信每次循环迭代时都会重新运行查询(使用 LIMIT 1 或其他东西),因为在类似 SQLiteStudio [i]and[/i] 返回所有 33,000 行。我计算出,按照这个速度,完成所有 33,000 条记录大约需要 7 个小时。
我没有坐在那里,而是尝试使用 cursor.fetchall() 代替:
results = cursor.fetchall()
# Run through every record, extract the kanji, then query for FK and weight
printStatus("Starting weight calculations")
for row in results:
# TODO: Determine if there's any kanji in row[2]
weight = float((row[3] + row[4]))/2
printStatus("Weight: " + str(weight))
不幸的是,Python 可执行文件在到达 cursor.fetchall()
行时锁定在 25% CPU 和 ~6MB RAM。我让脚本运行了约 10 分钟,但什么也没发生。
大约 33,000 行返回行(大约 5MB 的数据)对于 Python 来说是否太多而无法一次获取?我一次只能迭代一个吗?或者我可以做些什么来加快速度?
编辑:这是一些控制台输出
12:56:26.019: Adding new column 'weight' and related index to r_ele
12:56:26.019: Querying database
12:56:28.079: Starting weight calculations
12:56:28.079: Weight: 1.0
12:56:28.079: Weight: 0.5
12:56:28.080: Weight: 0.5
12:56:28.338: Weight: 1.0
12:56:28.339: Weight: 3.0
12:56:28.843: Weight: 1.5
12:56:28.844: Weight: 1.0
12:56:28.844: Weight: 0.5
12:56:28.844: Weight: 0.5
12:56:28.845: Weight: 0.5
12:56:29.351: Weight: 0.5
12:56:29.855: Weight: 0.5
12:56:29.856: Weight: 1.0
12:56:30.371: Weight: 0.5
12:56:30.885: Weight: 0.5
12:56:31.146: Weight: 0.5
12:56:31.650: Weight: 1.0
12:56:32.432: Weight: 0.5
12:56:32.951: Weight: 0.5
12:56:32.951: Weight: 0.5
12:56:32.952: Weight: 1.0
12:56:33.454: Weight: 0.5
12:56:33.455: Weight: 0.5
12:56:33.455: Weight: 1.0
12:56:33.716: Weight: 0.5
12:56:33.716: Weight: 1.0
这是 SQL 查询:
//...snip (it wasn't the culprit)...
SQLiteStudio 的 EXPLAIN QUERY PLAN 输出:
0 0 0 SCAN TABLE r_ele AS re USING COVERING INDEX r_ele_fk (~500000 rows)
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 1
1 0 0 SEARCH TABLE re_pri USING INDEX re_pri_fk (fk=?) (~10 rows)
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 2
2 0 0 SEARCH TABLE ke_pri USING INDEX ke_pri_fk (fk=?) (~10 rows)
2 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 3
3 0 0 SEARCH TABLE k_ele USING AUTOMATIC COVERING INDEX (value=?) (~7 rows)
3 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 4
4 0 0 SEARCH TABLE k_ele USING COVERING INDEX idx_k_ele (fk=?) (~10 rows)
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 5
5 0 0 SEARCH TABLE k_ele USING COVERING INDEX idx_k_ele (fk=?) (~10 rows)
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 6
6 0 0 SEARCH TABLE re_pri USING INDEX re_pri_fk (fk=?) (~10 rows)
0 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 7
7 0 0 SEARCH TABLE ke_pri USING INDEX ke_pri_fk (fk=?) (~10 rows)
7 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 8
8 0 0 SEARCH TABLE k_ele USING AUTOMATIC COVERING INDEX (value=?) (~7 rows)
8 0 0 EXECUTE CORRELATED SCALAR SUBQUERY 9
9 0 0 SEARCH TABLE k_ele USING COVERING INDEX idx_k_ele (fk=?) (~10 rows)
最佳答案
SQLite 即时计算结果记录。
fetchone
很慢,因为它必须为 r_ele
中的每条记录执行所有子查询。
fetchall
甚至更慢,因为它所花的时间与对所有记录执行 fetchone
所花的时间一样长。
SQLite 3.7.13 估计在 value
列上的所有查找都会非常慢,因此为此查询创建了一个临时索引。
您应该创建一个永久索引,以便 SQLite 3.6.21 可以使用它:
CREATE INDEX idx_k_ele_value ON k_ele(value);
如果这没有帮助,请更新到具有较新 SQLite 版本的 Python,或使用另一个内置了较新 SQLite 版本的数据库,例如 APSW .
关于Python 在 fetchone 上运行缓慢,在 fetchall 上挂起,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/18343445/