python - 加载索引具有冗余列的 parquet 文件时出现 pyarrow 问题

Question

我正在使用 pandas/dask 进行计算，并将数据存储在磁盘上的 Parquet 文件中。问题是，我有一列“时间”，还有一个称为时间的索引。我想两者都保留。当我存储数据并稍后加载时，出现以下错误:

import pyarrow as pa
import pyarrow.parquet as pq
%matplotlib inline
dfx.to_dict()

Out[115]: 
{'close': {Timestamp('2017-06-30 01:31:00'): 154.99958999999998,
  Timestamp('2017-06-30 01:32:00'): 154.99958999999998,
  Timestamp('2017-06-30 01:33:00'): 154.01109,
  Timestamp('2017-06-30 01:34:00'): 154.01109,
  Timestamp('2017-06-30 01:35:00'): 152.60051000000001},
 'time': {Timestamp('2017-06-30 01:31:00'): Timestamp('2017-06-30 01:31:00'),
  Timestamp('2017-06-30 01:32:00'): Timestamp('2017-06-30 01:32:00'),
  Timestamp('2017-06-30 01:33:00'): Timestamp('2017-06-30 01:33:00'),
  Timestamp('2017-06-30 01:34:00'): Timestamp('2017-06-30 01:34:00'),
  Timestamp('2017-06-30 01:35:00'): Timestamp('2017-06-30 01:35:00')}}

# set index column 
dfx.set_index('time', drop=False, inplace=True)

dfx.head()
Out[117]: 
                                   time      close
time                                              
2017-06-30 01:31:00 2017-06-30 01:31:00  154.99959
2017-06-30 01:32:00 2017-06-30 01:32:00  154.99959
2017-06-30 01:33:00 2017-06-30 01:33:00  154.01109
2017-06-30 01:34:00 2017-06-30 01:34:00  154.01109
2017-06-30 01:35:00 2017-06-30 01:35:00  152.60051

# store to parquet file format
tdfx = pa.Table.from_pandas(dfx)
pq.write_table(tdfx, 'data.parquet' )


# recovering
dfx = pq.read_table('data.parquet').to_pandas()
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-119-5e9d7cd2ea0d> in <module>()
      1 # recovering
----> 2 dfx = pq.read_table('data.parquet').to_pandas()

pyarrow/table.pxi in pyarrow.lib.Table.to_pandas (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:37990)()

/home/ghildebrand/anaconda3/envs/p36/lib/python3.6/site-packages/pyarrow/pandas_compat.py in table_to_blockmanager(options, table, memory_pool, nthreads)
    296         i = schema.get_field_index(name)
    297         if i != -1:
--> 298             col = table.column(i)
    299             index_name = (None if is_unnamed_index_level(name)
    300                           else name)

pyarrow/table.pxi in pyarrow.lib.Table.column (/arrow/python/build/temp.linux-x86_64-3.6/lib.cxx:38622)()

IndexError: Table column index 2 is out of range

这是 pyarrow 中的错误，还是 Parquet 不可能这样做，还是我做错了什么？？

更新:删除冗余列“时间”并仅保留索引解决方案。所以我猜问题是在 Parquet 中的某个地方创建了唯一的列标识符集。

Answer 1

我觉得有点问题。我打开了错误报告https://issues.apache.org/jira/browse/ARROW-1754 ，我们继续讨论吧。