python - 以特定频率扩展数据框 - Python

Question

我有一个包含以下列的数据框(只是摘录):

     START                    END               FREQ     VARIABLE    
'2017-03-26 16:55:00'  '2017-10-28 16:55:00'   1234567      x
'2017-03-26 20:35:00'  '2017-10-28 20:35:00'   1234567      y
'2017-03-26 14:55:00'  '2017-10-28 14:55:00'   ..3.567      y
'2017-03-26 11:15:00'  '2017-10-28 11:15:00'   1234567      y
'2017-03-26 09:30:00'  '2017-06-11 09:30:00'   ......7      x

我的目标是创建一个新的数据框，根据“FREQ”列生成从“START”日期开始到“END”日期结束的每日行，从而扩展此数据框。在此“FREQ”列中，1 = 星期一，7 = 星期日。 “点”表示不应在一周中的特定日期创建该行。因此，..3.5.7 仅对应于周三、周五和周日的 3 个新行。对于每个创建的行，“VARIABLE”列应始终具有相同的值。

我的主要问题是新的数据框将有数百万行，因此，我一直在寻找一个真正有效的解决方案。

用Python代码编写的数据框:

import pandas as pd
import numpy as np

df = pd.DataFrame(np.array([ 
'2017-03-26 16:55:00','2017-10-28 16:55:00', '1234567', 'x',
'2017-03-26 20:35:00','2017-10-28 20:35:00','1234567','y',
'2017-03-26 14:55:00','2017-10-28 14:55:00','..3.567','y',
'2017-03-26 11:15:00','2017-10-28 11:15:00','1234567','y',
'2017-03-26 09:30:00','2017-06-11 09:30:00','......7','x']).reshape((5, 4)))
df.columns = ['START','END','FREQ','VARIABLE']

Answer 1

修改后的答案:

这使用 pandas iloc 和 numpy Repeat 从原始数据帧索引创建一个新的数据帧，但在根据日期范围和有效工作日确定重复索引之后。

import pandas as pd
import numpy as np

df_arr = np.array([ 
    '2017-03-26 16:55:00', '2017-10-28 16:55:00', '1234567', 'x',
    '2017-03-26 20:35:00', '2017-10-28 20:35:00', '1234567', 'y',
    '2017-03-26 14:55:00', '2017-10-28 14:55:00', '..3.567', 'y',
    '2017-03-26 11:15:00', '2017-10-28 11:15:00', '1234567', 'y',
    '2017-03-26 09:30:00', '2017-06-11 09:30:00', '......7',' x'])

df = pd.DataFrame(df_arr.reshape(5, 4),
                  columns=['START', 'END', 'FREQ', 'VARIABLE'])

def get_weekdays_dates_repeats(start, end, valid_weekday_nums):
    date_range = pd.date_range(start, end, freq="D", normalize=True)
    all_day_nums = date_range.dayofweek.values + 1
    filtered_idx = np.where(np.isin(all_day_nums, valid_weekday_nums))
    day_nums = all_day_nums[filtered_idx]
    dates = date_range[filtered_idx]
    return day_nums, dates.values.astype('<M8[D]'), day_nums.size

starts = df.START.values
ends = df.END.values
freqs = df.FREQ.str.replace('.', '').values

repeats = np.zeros(len(df))
weekdays_arr_list = []
dates_arr_list = []
for i in range(len(df)):
    valid_day_nums = [int(s) for s in list(freqs[i])]
    days, dates, repeat = \
        get_weekdays_dates_repeats(starts[i], ends[i], valid_day_nums)
    weekdays_arr_list.append(days)
    dates_arr_list.append(dates)
    repeats[i] = repeat

weekday_col = np.concatenate(weekdays_arr_list)
dates_col = np.concatenate(dates_arr_list)
repeats = repeats.astype(int)

df2 = df.iloc[np.repeat(df.index.values, repeats)].reset_index(drop=True)

df2['day_num'] = weekday_col
df2['date'] = dates_col

df2.head()

                  START        END          FREQ    VARIABLE    day_num date
0   2017-03-26 16:55:00 2017-10-28 16:55:00 1234567 x   7   2017-03-26
1   2017-03-26 16:55:00 2017-10-28 16:55:00 1234567 x   1   2017-03-27
2   2017-03-26 16:55:00 2017-10-28 16:55:00 1234567 x   2   2017-03-28
3   2017-03-26 16:55:00 2017-10-28 16:55:00 1234567 x   3   2017-03-29
4   2017-03-26 16:55:00 2017-10-28 16:55:00 1234567 x   4   2017-03-30

df2.tail()

                  START                END  FREQ    VARIABLE    day_num date
782 2017-03-26 09:30:00 2017-06-11 09:30:00 ......7 x   7   2017-05-14
783 2017-03-26 09:30:00 2017-06-11 09:30:00 ......7 x   7   2017-05-21
784 2017-03-26 09:30:00 2017-06-11 09:30:00 ......7 x   7   2017-05-28
785 2017-03-26 09:30:00 2017-06-11 09:30:00 ......7 x   7   2017-06-04
786 2017-03-26 09:30:00 2017-06-11 09:30:00 ......7 x   7   2017-06-11