python - 如何关闭pandas read_sql连接

Question

以下代码会产生错误，表明数据库已锁定:

    df_chunks = pd.read_sql('cdr', engine, chunksize=100000)
    engine.dispose()
    with engine.connect() as conn:
        trans = conn.begin()
        query = """
        CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
        """
        print(query)
        conn.execute(query)
        trans.commit()
        conn.close()

但如果我反过来做:

    with engine.connect() as conn:
        trans = conn.begin()
        query = """
        CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
        """
        print(query)
        conn.execute(query)
        trans.commit()
        conn.close()
    df_chunks = pd.read_sql('cdr', engine, chunksize=100000)

它有效，我能够生成新表。所以从这里来看，问题似乎是 pd.read_sql (请参阅 docs )锁定数据库。我发现this question但 engine.dispose() 对我不起作用。有什么方法可以解决这个问题？

Answer 1

因为 chunksize 构建了一个数据帧生成器，并且您从未对 df_chunks 执行任何操作，所以该对象可能仍然持有指向源对象 engine 的指针code>，因此“锁定”数据库。要解决您的第一次尝试，请考虑在 df_chunks 上运行迭代:

with engine.connect() as conn:
    trans = conn.begin()
    query = """CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
            """
    print(query)
    conn.execute(query)
    trans.commit()

df_chunks = pd.read_sql('cdr', engine, chunksize=100000)

for df in df_chunks:
    # DO SOMETHING WITH EACH df

engine.dispose()                     # ALLOWED SINCE GENERATOR IS EXHAUSTED AFTER for LOOP

或者，要解决您的第二次尝试，请将 read_sql 调用集成到 with block 内并使用 conn 对象。

with engine.connect() as conn:
    trans = conn.begin()
    query = """CREATE TABLE test (row_id BIGINT,ego_id BIGINT,alter_id BIGINT)
            """
    print(query)
    conn.execute(query)
    trans.commit()

    # INDENT LINE AND USE conn OBJECT
    df_chunks = pd.read_sql('cdr', conn, chunksize=100000)

    for df in df_chunks:
        # DO SOMETHING WITH EACH df

engine.dispose()                     # CLOSE engine OBJECT NOT conn

此外，当使用 with 作为上下文管理器时，无需调用 close:conn.close()。