python - 需要在 python 中比较 1.5GB 左右的非常大的文件

标签 python csv numpy pandas large-data-volumes 

"DF","00000000@11111.COM","FLTINT1000130394756","26JUL2010","B2C","6799.2"
"Rail","00000.POO@GMAIL.COM","NR251764697478","24JUN2011","B2C","2025"
"DF","0000650000@YAHOO.COM","NF2513521438550","01JAN2013","B2C","6792"
"Bus","00009.GAURAV@GMAIL.COM","NU27012932319739","26JAN2013","B2C","800"
"Rail","0000.ANU@GMAIL.COM","NR251764697526","24JUN2011","B2C","595"
"Rail","0000MANNU@GMAIL.COM","NR251277005737","29OCT2011","B2C","957"
"Rail","0000PRANNOY0000@GMAIL.COM","NR251297862893","21NOV2011","B2C","212"
"DF","0000PRANNOY0000@YAHOO.CO.IN","NF251327485543","26JUN2011","B2C","17080"
"Rail","0000RAHUL@GMAIL.COM","NR2512012069809","25OCT2012","B2C","5731"
"DF","0000SS0@GMAIL.COM","NF251355775967","10MAY2011","B2C","2000"
"DF","0001HARISH@GMAIL.COM","NF251352240086","22DEC2010","B2C","4006"
"DF","0001HARISH@GMAIL.COM","NF251742087846","12DEC2010","B2C","1000"
"DF","0001HARISH@GMAIL.COM","NF252022031180","09DEC2010","B2C","3439"
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136"
"Rail","000AYUSH@GMAIL.COM","NR2151213260036","28NOV2012","B2C","41"
"Rail","000AYUSH@GMAIL.COM","NR2151313264432","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2151413266728","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96"

以上是示例数据。 数据按照邮箱地址排序，文件很大，1.5Gb左右

我想在另一个类似这样的 csv 文件中输出

"DF","00000000@11111.COM","FLTINT1000130394756","26JUL2010","B2C","6799.2",1,0 days
"Rail","00000.POO@GMAIL.COM","NR251764697478","24JUN2011","B2C","2025",1,0 days
"DF","0000650000@YAHOO.COM","NF2513521438550","01JAN2013","B2C","6792",1,0 days
"Bus","00009.GAURAV@GMAIL.COM","NU27012932319739","26JAN2013","B2C","800",1,0 days
"Rail","0000.ANU@GMAIL.COM","NR251764697526","24JUN2011","B2C","595",1,0 days
"Rail","0000MANNU@GMAIL.COM","NR251277005737","29OCT2011","B2C","957",1,0 days
"Rail","0000PRANNOY0000@GMAIL.COM","NR251297862893","21NOV2011","B2C","212",1,0 days
"DF","0000PRANNOY0000@YAHOO.CO.IN","NF251327485543","26JUN2011","B2C","17080",1,0 days
"Rail","0000RAHUL@GMAIL.COM","NR2512012069809","25OCT2012","B2C","5731",1,0 days
"DF","0000SS0@GMAIL.COM","NF251355775967","10MAY2011","B2C","2000",1,0 days
"DF","0001HARISH@GMAIL.COM","NF251352240086","09DEC2010","B2C","4006",1,0 days
"DF","0001HARISH@GMAIL.COM","NF251742087846","12DEC2010","B2C","1000",2,3 days
"DF","0001HARISH@GMAIL.COM","NF252022031180","22DEC2010","B2C","3439",3,10 days
"Rail","000AYUSH@GMAIL.COM","NR2151213260036","28NOV2012","B2C","41",1,0 days
"Rail","000AYUSH@GMAIL.COM","NR2151313264432","29NOV2012","B2C","96",2,1 days
"Rail","000AYUSH@GMAIL.COM","NR2151413266728","29NOV2012","B2C","96",3,0 days
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96",4,9 days
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96",5,0 days
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96",6,4 days
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96",7,0 days
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136",8,44 days
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136",9,0 days

即如果条目第一次出现我需要附加 1 如果它第二次出现我需要附加 2 同样我的意思是我需要计算文件中电子邮件地址的出现次数以及电子邮件是否存在两次或更多次我想要日期之间的差异并记住日期未排序所以我们也必须根据特定的电子邮件地址对它们进行排序我正在寻找使用 numpy 或 pandas 库或任何其他库的 python 解决方案处理这种类型的巨大数据而不会出现超出限制的内存异常我有双核处理器和 centos 6.3 并且有 4GB 的 ram

最佳答案

确保你有 0.11，阅读这些文档:http://pandas.pydata.org/pandas-docs/dev/io.html#hdf5-pytables ，以及这些食谱:http://pandas.pydata.org/pandas-docs/dev/cookbook.html#hdfstore (特别是“合并数百万行”

这是一个似乎有效的解决方案。这是工作流程:

  1. 按 block 从您的 csv 中读取数据并附加到 hdfstore
  2. 遍历商店，这将创建另一个执行组合器的商店

本质上，我们是从表中取出一个 block ，然后与文件其他部分的 block 组合。 combiner 函数不会减少，而是计算该 block 中所有元素之间的函数(以天为单位的差异)，同时消除重复项，并在每次循环后获取最新数据。有点像递归归约。

这应该是O(num_of_chunks**2)内存和计算时间 在你的情况下，chunksize 可以是 1m(或更多)

processing [0] [datastore.h5]
processing [1] [datastore_0.h5]
    count                date  diff                        email
4       1 2011-06-24 00:00:00     0           0000.ANU@GMAIL.COM
1       1 2011-06-24 00:00:00     0          00000.POO@GMAIL.COM
0       1 2010-07-26 00:00:00     0           00000000@11111.COM
2       1 2013-01-01 00:00:00     0         0000650000@YAHOO.COM
3       1 2013-01-26 00:00:00     0       00009.GAURAV@GMAIL.COM
5       1 2011-10-29 00:00:00     0          0000MANNU@GMAIL.COM
6       1 2011-11-21 00:00:00     0    0000PRANNOY0000@GMAIL.COM
7       1 2011-06-26 00:00:00     0  0000PRANNOY0000@YAHOO.CO.IN
8       1 2012-10-25 00:00:00     0          0000RAHUL@GMAIL.COM
9       1 2011-05-10 00:00:00     0            0000SS0@GMAIL.COM
12      1 2010-12-09 00:00:00     0         0001HARISH@GMAIL.COM
11      2 2010-12-12 00:00:00     3         0001HARISH@GMAIL.COM
10      3 2010-12-22 00:00:00    13         0001HARISH@GMAIL.COM
14      1 2012-11-28 00:00:00     0           000AYUSH@GMAIL.COM
15      2 2012-11-29 00:00:00     1           000AYUSH@GMAIL.COM
17      3 2012-12-08 00:00:00    10           000AYUSH@GMAIL.COM
18      4 2012-12-12 00:00:00    14           000AYUSH@GMAIL.COM
13      5 2013-01-25 00:00:00    58           000AYUSH@GMAIL.COM
import pandas as pd
import StringIO
import numpy as np
from time import strptime
from datetime import datetime

# your data
data = """
"DF","00000000@11111.COM","FLTINT1000130394756","26JUL2010","B2C","6799.2"
"Rail","00000.POO@GMAIL.COM","NR251764697478","24JUN2011","B2C","2025"
"DF","0000650000@YAHOO.COM","NF2513521438550","01JAN2013","B2C","6792"
"Bus","00009.GAURAV@GMAIL.COM","NU27012932319739","26JAN2013","B2C","800"
"Rail","0000.ANU@GMAIL.COM","NR251764697526","24JUN2011","B2C","595"
"Rail","0000MANNU@GMAIL.COM","NR251277005737","29OCT2011","B2C","957"
"Rail","0000PRANNOY0000@GMAIL.COM","NR251297862893","21NOV2011","B2C","212"
"DF","0000PRANNOY0000@YAHOO.CO.IN","NF251327485543","26JUN2011","B2C","17080"
"Rail","0000RAHUL@GMAIL.COM","NR2512012069809","25OCT2012","B2C","5731"
"DF","0000SS0@GMAIL.COM","NF251355775967","10MAY2011","B2C","2000"
"DF","0001HARISH@GMAIL.COM","NF251352240086","22DEC2010","B2C","4006"
"DF","0001HARISH@GMAIL.COM","NF251742087846","12DEC2010","B2C","1000"
"DF","0001HARISH@GMAIL.COM","NF252022031180","09DEC2010","B2C","3439"
"Rail","000AYUSH@GMAIL.COM","NR2151120122283","25JAN2013","B2C","136"
"Rail","000AYUSH@GMAIL.COM","NR2151213260036","28NOV2012","B2C","41"
"Rail","000AYUSH@GMAIL.COM","NR2151313264432","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2151413266728","29NOV2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2512912359037","08DEC2012","B2C","96"
"Rail","000AYUSH@GMAIL.COM","NR2517612385569","12DEC2012","B2C","96"
"""


# read in and create the store
data_store_file = 'datastore.h5'
store = pd.HDFStore(data_store_file,'w')

def dp(x, **kwargs):
    return [ datetime(*strptime(v,'%d%b%Y')[0:3]) for v in x ]

chunksize=5
reader = pd.read_csv(StringIO.StringIO(data),names=['x1','email','x2','date','x3','x4'],
                     header=0,usecols=['email','date'],parse_dates=['date'],
                     date_parser=dp, chunksize=chunksize)

for i, chunk in enumerate(reader):
    chunk['indexer'] = chunk.index + i*chunksize

    # create the global index, and keep it in the frame too
    df = chunk.set_index('indexer')

    # need to set a minimum size for the email column
    store.append('data',df,min_itemsize={'email' : 100})

store.close()

# define the combiner function
def combiner(x):

    # given a group of emails (the same), return a combination
    # with the new data

    # sort by the date
    y = x.sort('date')

    # calc the diff in days (an integer)
    y['diff'] = (y['date']-y['date'].iloc[0]).apply(lambda d: float(d.item().days))
    y['count'] = pd.Series(range(1,len(y)+1),index=y.index,dtype='float64')  
    
    return y

# reduce the store (and create a new one by chunks)
in_store_file = data_store_file
in_store1 = pd.HDFStore(in_store_file)

# iter on the store 1
for chunki, df1 in enumerate(in_store1.select('data',chunksize=2*chunksize)):
    print "processing [%s] [%s]" % (chunki,in_store_file)

    out_store_file = 'datastore_%s.h5' % chunki
    out_store = pd.HDFStore(out_store_file,'w')

    # iter on store 2
    in_store2 = pd.HDFStore(in_store_file)
    for df2 in in_store2.select('data',chunksize=chunksize):

        # concat & drop dups
        df = pd.concat([df1,df2]).drop_duplicates(['email','date'])

        # group and combine
        result = df.groupby('email').apply(combiner)
            
        # remove the mi (that we created in the groupby)
        result = result.reset_index('email',drop=True)
            
        # only store those rows which are in df2!
        result = result.reindex(index=df2.index).dropna()

        # store to the out_store
        out_store.append('data',result,min_itemsize={'email' : 100})
    in_store2.close()
    out_store.close()
    in_store_file = out_store_file

in_store1.close()

# show the reduced store
print pd.read_hdf(out_store_file,'data').sort(['email','diff'])

关于python - 需要在 python 中比较 1.5GB 左右的非常大的文件，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/16110252/

上一篇：python - python中冲突的变量和函数名称

下一篇：python - 有什么优雅的方法可以使包含一些整数的列表变成包含一些元组的列表？

相关文章：

validation - Laravel 验证规则的关键

java - 如何在不捕获错误分隔符的情况下解析 CSV？

java - 使用 Groovy 重新映射 CSV 结构

python - 如果每个进程的根都是其自身，为什么 Gather() 会失败？

python - 在 pandas dataframe map 函数中使用 eval 语句的正确方法

python - 有什么方法可以解决 python 中的 float Counter 值

python - Python运行时目录

python - 如何在 Python 中从 Numpy 矩阵创建列表

Python - 对数组进行排序以在其中进行搜索的最有效方法

python - Otsu 掩膜内阈值化

©2024 IT工具网  联系我们