使用 Pandas 处理一些基于时间序列的数据,这些数据包含日期、数字、类别等。
我遇到的问题是让 Pandas 从 CSV 创建的 DataFrame 中正确处理我的日期/时间列。我的数据中有 18 个日期列,它们不是连续的,原始 CSV 中的未知值的字符串值为“未知”。有些列的所有单元格中都有一个有效的日期时间,并且正确地得到了 pandas read_csv 方法猜测的它们的数据类型。然而,在特定数据样本中,有些列的所有单元格都为“未知”,并且这些列被键入为对象。
我加载 CSV 的代码如下:
self.datecols = ['Claim Date', 'Lock Date', 'Closed Date', 'Service Date', 'Latest_Submission', 'Statement Date 1', 'Statement Date 2', 'Statement Date 3', 'Patient Payment Date 1', 'Patient Payment Date 2', 'Patient Payment Date 3', 'Primary 1 Payment Date', 'Primary 2 Payment Date', 'Primary 3 Payment Date', 'Secondary 1 Payment Date', 'Secondary 2 Payment Date', 'Tertiary Payment Date']
self.csvbear = pd.read_csv(file_path, index_col="Claim ID", parse_dates=True, na_values=['Unknown'])
self.csvbear = pd.DataFrame.convert_objects(self.csvbear, convert_dates='coerce')
print self.csvbear.dtypes
print self.csvbear['Tertiary Payment Date'].values
打印 self.csvbear.dtypes 的输出
Prac object
Doctor Name object
Practice Name object
Specialty object
Speciality Code int64
Claim Date datetime64[ns]
Lock Date datetime64[ns]
Progress Note Locked object
Aging by Claim Date int64
Aging by Lock Date int64
Closed Date datetime64[ns]
Service Date datetime64[ns]
Week Number int64
Month datetime64[ns]
Current Insurance object
...
Secondary 2 Deductible float64
Secondary 2 Co Insurance float64
Secondary 2 Member Balance float64
Secondary 2 Paid float64
Secondary 2 Witheld float64
Secondary 2 Ins object
Tertiary Payment Date object
Tertiary Payment ID float64
Tertiary Allowed float64
Tertiary Deductible float64
Tertiary Co Insurance float64
Tertiary Member Balance float64
Tertiary Paid float64
Tertiary Witheld float64
Tertiary Ins float64
Length: 96, dtype: object
[nan nan nan ..., nan nan nan]
Press any key to continue . . .
如您所见,第三付款日期 col 应该是 datetime64 dtype,但它只是一个对象,它的实际内容只是 NaN(从字符串“Unknown”的 read_csv 函数中放置)。
我如何才能可靠地将所有日期列转换为 datetime64 作为 dtype 并为“未知”单元格设置 NaT?
最佳答案
如果您有一个全 nan 列,它不会被 read_csv
正确强制转换。最简单的就是这样做(如果列已经是 datetime64[ns] 将直接通过)。
In [3]: df = DataFrame(dict(A = Timestamp('20130101'), B = np.random.randn(5), C = np.nan))
In [4]: df
Out[4]:
A B C
0 2013-01-01 00:00:00 -0.859994 NaN
1 2013-01-01 00:00:00 -2.562136 NaN
2 2013-01-01 00:00:00 0.410673 NaN
3 2013-01-01 00:00:00 0.480578 NaN
4 2013-01-01 00:00:00 0.464771 NaN
[5 rows x 3 columns]
In [5]: df.dtypes
Out[5]:
A datetime64[ns]
B float64
C float64
dtype: object
In [6]: df['A'] = pd.to_datetime(df['A'])
In [7]: df['C'] = pd.to_datetime(df['C'])
In [8]: df
Out[8]:
A B C
0 2013-01-01 00:00:00 -0.859994 NaT
1 2013-01-01 00:00:00 -2.562136 NaT
2 2013-01-01 00:00:00 0.410673 NaT
3 2013-01-01 00:00:00 0.480578 NaT
4 2013-01-01 00:00:00 0.464771 NaT
[5 rows x 3 columns]
In [9]: df.dtypes
Out[9]:
A datetime64[ns]
B float64
C datetime64[ns]
dtype: object
convert_objects
不会强制将列转换为 datetime,除非它至少有 1 个非 nan 日期(这就是您的示例失败的原因)。 to_datetime
可以更具侵略性,因为它“知道”您真的想要转换它。
关于python - 将 Pandas 列转换为 datetime64,包括缺失值,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/20689288/