python - 读取 csv 时加快日期时间格式化速度

Question

我有一个包含 4+m 条记录的 csv 文件。

我使用 pd.read_csv('big_file.csv', dtype=object) 导入它

此文件有 2 列，日期格式如下: 'yyyy-mm-ddThh:mm:ss.nsTZ' 例如'2018-05-05T04:39:09.447Z'

我需要将它们转换为 'yyyy-mm-dd H:M:S' 例如'2018-09-23 06:03:12'

我使用以下代码来执行此操作:

df['created'] = pd.to_datetime(arg=df.created).dt.strftime('%Y-%m-%d %H:%M:%S')
df['lastLogin'] = pd.to_datetime(arg=df.lastLogin).dt.strftime('%Y-%m-%d %H:%M:%S')
df['lastUpdated'] = pd.to_datetime(arg=df.lastUpdated).dt.strftime('%Y-%m-%d %H:%M:%S')
df['created'] = pd.to_datetime(arg=df.created)
df['lastLogin'] = pd.to_datetime(arg=df.lastLogin)
df['lastUpdated'] = pd.to_datetime(arg=df.lastUpdated)

这个过程非常慢:

CPU times: user 1min 48s, sys: 1.19 s, total: 1min 49s
Wall time: 1min 49s

有没有办法加快速度？

Answer 1

从评论中，我了解到您不需要日期的含义，但您只想更改表示日期的字符串的美观外观。然后，您可以将数据简单地视为字符串。所以，我就这样做了。

#!/usr/bin/python3

import numpy as np
import sys

def gen_sample(numdata, outfname):
    yy=np.random.randint(1905, 2018, 2*numdata)
    mm=np.random.randint(   1,   13, 2*numdata)
    dd=np.random.randint(   1,   29, 2*numdata)
    hhh=np.random.randint(   0,   25, 2*numdata)
    mmm=np.random.randint(   0,   61, 2*numdata)
    sss=np.random.randint(   0,   61, 2*numdata)
    baboon=np.random.randint(   0, 1000, 2*numdata)
    with open(outfname, 'w') as outf:
        for jj in range(numdata):
            outf.write('%4.4i-%2.2i-%2.2iT%2.2i:%2.2i:%2.2i.%3.3iZ,%4.4i-%2.2i-%2.2iT%2.2i:%2.2i:%2.2i.%3.3iZ\n'
                       %(yy[2*jj],   mm[2*jj],   dd[2*jj],
                         hhh[2*jj],  mmm[2*jj],  sss[2*jj],  baboon[2*jj],
                         yy[2*jj+1], mm[2*jj+1], dd[2*jj+1],
                         hhh[2*jj+1], mmm[2*jj+1], sss[2*jj+1], baboon[2*jj+1]))


def convert(infname,outfname):

    data=np.loadtxt(infname, dtype=np.str, delimiter=',', ndmin=2)
    with open(outfname,'w') as outf:
        for jr in range(data.shape[0]):
            outf.write('%s %s,%s %s\n'%(
                data[jr,0][0:10],
                data[jr,0][11:19],
                data[jr,1][0:10],
                data[jr,1][11:19] ))


if __name__=='__main__':
    sample_fname= 'daa.csv'
    out_fname= 'daadaa.csv'
    if len(sys.argv)>1:
        numdata=int(sys.argv[1])
        gen_sample(numdata, sample_fname)
    else:
        convert(sample_fname, out_fname)

我的电脑上4M*2的数据大约花了15秒。请看这个

#!/bin/bash

for jj in 0 1 2
do
  echo "generating sample.."
  ./main.py 4000000
  echo "loading, converting, and writing.."
  echo "----"
  /usr/bin/time ./main.py 
  echo "----"
done

还有这个

$ ./run.sh 
generating sample..
loading, converting, and writing..
----
14.96user 0.94system 0:15.05elapsed 105%CPU (0avgtext+0avgdata 818724maxresident)k
8inputs+312504outputs (0major+315787minor)pagefaults 0swaps
----
generating sample..
loading, converting, and writing..
----
14.91user 0.93system 0:14.99elapsed 105%CPU (0avgtext+0avgdata 818848maxresident)k
16inputs+312504outputs (0major+315864minor)pagefaults 0swaps
----
generating sample..
loading, converting, and writing..
----
15.39user 0.95system 0:15.52elapsed 105%CPU (0avgtext+0avgdata 818736maxresident)k
8inputs+312504outputs (0major+315857minor)pagefaults 0swaps
----

输入文件就像

$ head daa.csv 
2016-10-05T08:07:03.214Z,1973-10-01T12:36:21.367Z
1961-08-24T02:08:57.436Z,1953-03-06T00:56:12.486Z
1986-09-07T17:15:60.322Z,1952-11-19T19:02:56.159Z
1939-08-17T05:13:19.659Z,1920-12-15T16:46:52.628Z
2004-11-09T02:29:25.905Z,1925-02-07T10:37:49.142Z
2011-12-12T10:46:38.583Z,1992-02-10T08:58:60.284Z
1968-01-23T05:05:05.151Z,1935-09-17T07:12:49.392Z
1916-04-05T18:55:35.281Z,1919-10-12T10:05:10.249Z
1970-10-04T21:45:16.751Z,1951-01-08T16:58:51.190Z
1910-01-19T22:12:04.088Z,2006-03-08T09:26:45.690Z

输出文件就像

$ head daadaa.csv 
2016-10-05 08:07:03,1973-10-01 12:36:21
1961-08-24 02:08:57,1953-03-06 00:56:12
1986-09-07 17:15:60,1952-11-19 19:02:56
1939-08-17 05:13:19,1920-12-15 16:46:52
2004-11-09 02:29:25,1925-02-07 10:37:49
2011-12-12 10:46:38,1992-02-10 08:58:60
1968-01-23 05:05:05,1935-09-17 07:12:49
1916-04-05 18:55:35,1919-10-12 10:05:10
1970-10-04 21:45:16,1951-01-08 16:58:51
1910-01-19 22:12:04,2006-03-08 09:26:45

如果不需要将转换后的数据写回到文件中， 也许运行时间会更快。 您需要使该函数对于您自己的数据的变化更加稳健，但我希望这个想法能够实现。