python - pandas groupBy date 然后将日期和字符串过滤到新的数据框中

Question

我在这里苦苦挣扎，我希望获取以下数据，按日期分组，然后检查组内的行以确定该组是否具有与其关联的任何位置数据，如果有，则提取它.

我的数据样本:

id,dates,text,place
1,2017-01-26 01:06:47,text,"Place(country_code='US', full_name='Manhattan, NY', place_type='city', name='Manhattan', contained_within=[], _api=<tweepy.api.API object at 0x10336f320>, attributes={}, country='United States', bounding_box=BoundingBox(type='Polygon', coordinates=[[[-74, 40], [-73, 40], [-73, 40], [-74, 40]]], _api=<tweepy.api.API object at 0x10336f320>))"
2,2017-01-26 01:05:51,text,"Place(country_code='US', full_name='Manhattan, NY', place_type='city', name='Manhattan', contained_within=[], _api=<tweepy.api.API object at 0x10336f320>, attributes={}, country='United States', bounding_box=BoundingBox(type='Polygon', coordinates=[[[-74, 40], [-73, 40], [-73, 40], [-74, 40]]], _api=<tweepy.api.API object at 0x10336f320>))"
4,2017-01-23 01:38:29,text,
5,2017-01-23 01:36:53,text,

我首先加载 csv 并对日期进行分组

import pandas as pd
import matplotlib.pyplot as plt
import datetime

fig = plt.figure(figsize=(5,5))
df1 = pd.read_csv('data.csv')
df = df1[['dates','place']]
df['dates']=pd.to_datetime(df['dates'],format='%Y-%m-%d')
df.index=df['dates']

grp = pd.groupby(df,by=[df.index.year,df.index.month,df.index.day])
for date,group in grp:
    print(date)
    print(group)

这将产生一个如下所示的数据框:

(2017, 1, 26)
                                  dates  \
dates
2017-01-26 01:06:47 2017-01-26 01:06:47
2017-01-26 01:05:51 2017-01-26 01:05:51

                                                                 place
dates
2017-01-26 01:06:47  Place(country_code='US', full_name='Manhattan,...
2017-01-26 01:05:51                                                NaN

这是我遇到过滤/条件问题的地方，我的目标是拥有一个可以保存到如下所示的 csv 的数据框:

date, item_count, has_location, location
2017-01-26, 2, yes, Manhattan
2017-01-23, 2, no, na

最好的方法是什么？谢谢

Answer 1

我认为你可以使用:

extract首先是 name 列 place，然后是 groupby by dt.date (如果dates列的dtype是datetime，to_datetime可以被删除)并按size聚合一些列例如id 和first 列place。最后insert由 numpy.where 创建的新列:

print (df)
   id                dates  text  \
0   1  2017-01-26 01:06:47  text   
1   2  2017-01-26 01:05:51  text   
2   4  2017-01-23 01:38:29  text   
3   5  2017-01-23 01:36:53  text   

                                               place  
0  Place(country_code='US', full_name='Manhattan,...  
1  Place(country_code='US', full_name='Manhattan,...  
2                                                NaN  
3                                                NaN  

df.place = df.place.str.extract(", name='(.*)', contained_within", expand=True)
print (df)
   id                dates  text      place
0   1  2017-01-26 01:06:47  text  Manhattan
1   2  2017-01-26 01:05:51  text  Manhattan
2   4  2017-01-23 01:38:29  text        NaN
3   5  2017-01-23 01:36:53  text        NaN

df1 = df.groupby(pd.to_datetime(df.dates).dt.date).agg({'id':'size', 'place':'first'})
df1.columns = ['item_count','location']
df1.insert(1, 'has_location', np.where(df1.location.isnull(), 'no', 'yes'))
print (df1)
            item_count has_location   location
dates                                         
2017-01-23           2           no        NaN
2017-01-26           2          yes  Manhattan