我正在寻找一种加速文件加载的方法,如下所示:
数据包含约100万行,制表符以“\t”(制表符)分隔,utf8编码,使用下面的代码解析完整文件大约需要9秒。然而,我希望能达到大约一秒!
def load(filename):
features = []
with codecs.open(filename, 'rb', 'utf-8') as f:
previous = ""
for n, s in enumerate(f):
splitted = tuple(s.rstrip().split("\t"))
if len(splitted) != 2:
sys.exit("wrong format!")
if previous >= splitted:
sys.exit("unordered feature")
previous = splitted
features.append(splitted)
return features
我想知道是否有任何二进制格式数据可以加快速度?或者,如果我可以从一些 NumPy
或任何其他库中受益,以获得更快的加载速度。
也许你可以给我关于另一个速度瓶颈的建议?
编辑:所以我尝试了你的一些想法,谢谢!顺便说一句,我真的需要巨大列表中的元组(字符串,字符串)...这是结果,我获得了 50% 的时间:) 现在我要照顾 NumPy 二进制数据,正如我所注意到的另一个大文件的加载速度真的非常快......
import codecs
def load0(filename):
with codecs.open(filename, 'rb', 'utf-8') as f:
return f.readlines()
def load1(filename):
with codecs.open(filename, 'rb', 'utf-8') as f:
return [tuple(x.rstrip().split("\t")) for x in f.readlines()]
def load3(filename):
features = []
with codecs.open(filename, 'rb', 'utf-8') as f:
for n, s in enumerate(f):
splitted = tuple(s.rstrip().split("\t"))
features.append(splitted)
return features
def load4(filename):
with codecs.open(filename, 'rb', 'utf-8') as f:
for s in f:
yield tuple(s.rstrip().split("\t"))
a = datetime.datetime.now()
r0 = load0(myfile)
b = datetime.datetime.now()
print "f.readlines(): %s" % (b-a)
a = datetime.datetime.now()
r1 = load1(myfile)
b = datetime.datetime.now()
print """[tuple(x.rstrip().split("\\t")) for x in f.readlines()]: %s""" % (b-a)
a = datetime.datetime.now()
r3 = load3(myfile)
b = datetime.datetime.now()
print """load3: %s""" % (b-a)
if r1 == r3: print "OK: speeded and similars!"
a = datetime.datetime.now()
r4 = [x for x in load4(myfile)]
b = datetime.datetime.now()
print """load4: %s""" % (b-a)
if r4 == r3: print "OK: speeded and similars!"
结果:
f.readlines(): 0:00:00.208000
[tuple(x.rstrip().split("\t")) for x in f.readlines()]: 0:00:02.310000
load3: 0:00:07.883000
OK: speeded and similars!
load4: 0:00:07.943000
OK: speeded and similars!
非常奇怪的是,我注意到我可以在连续两次运行中获得几乎双倍的时间(但不是每次):
>>> ================================ RESTART ================================
>>>
f.readlines(): 0:00:00.220000
[tuple(x.rstrip().split("\t")) for x in f.readlines()]: 0:00:02.479000
load3: 0:00:08.288000
OK: speeded and similars!
>>> ================================ RESTART ================================
>>>
f.readlines(): 0:00:00.279000
[tuple(x.rstrip().split("\t")) for x in f.readlines()]: 0:00:04.983000
load3: 0:00:10.404000
OK: speeded and similars!
编辑最新:好吧,我尝试修改以使用numpy.load
...这对我来说很奇怪...从“正常”文件与我的1022860 个字符串,10 KB。
执行完 numpy.save(numpy.array(load1(myfile))) 后,我的大小达到了 895 MB!然后使用 numpy.load() 重新加载它,我在连续运行中得到这种计时:
>>> ================================ RESTART ================================
loading: 0:00:11.422000 done.
>>> ================================ RESTART ================================
loading: 0:00:00.759000 done.
可能 numpy 会做一些内存操作以避免将来重新加载吗?
最佳答案
尝试这个版本,因为你提到检查并不重要,所以我已经消除了它。
def load(filename):
with codecs.open(filename, 'rb', 'utf-8') as f:
for s in f:
yield tuple(s.rstrip().split("\t"))
results = [x for x in load('somebigfile.txt')]
关于 python : how to speed up this file loading,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/12250781/