Python Pandas - 查找并分组异常值

Question

我有一个包含多列的 pd 数据框，如下所示(为了便于阅读而进行了简化) - 每行由一个 id (uuid)、索引和一个或多个功能组成:

uuid         index           Atrium       Ventricle 
di-abc           0            20.73           26.21
di-abc           1            18.92           25.14
di-efg           7            19.02            0.30 
di-efg           9             1.23            0.51
di-efg           6            21.24           26.02
di-hjk           3            22.10           25.16
di-hjk           6            19.16           25.57

我愿意:

1. 查找每个特征的异常值(即“Atrium”和“Ventricle”列)
2. 按以下格式导出异常值:

outliers = {
    'Atrium' : [
         {'uuid' : 'di-efg', 'index' : 9, 'value' : 1.23},
     ],
     'Ventricle' : [
         {'uuid' : 'di-efg', 'index' : 7, 'value' : 0.30},
         {'uuid' : 'di-efg', 'index' : 9, 'value' : 0.53},
    ]
}

注意事项(提交此内容可获得奖励积分):

1. 特征(以及列)的数量是动态的
2. 单行可以包含零个、一个、两个或多个异常值

我在双 for 循环之外的两个步骤都遇到困难。 有没有一种有效的方法来计算此数据帧中的异常值？

这是一种有效但效率不高的方法来捕获我想要完成的任务:

# initialize variables:
outliers = {}
features = ['Atrium', 'Ventricle']

# iterate over each feature:
for feature in features:

    # set feature on outlier to empty list:
    outliers[feature] = []
            
    # create a dataframe of outliers for that specific feature:
    outlier_df = df[df[feature] > (df[feature].mean() + df[feature].std())] # can mess with this if needed
    outlier_df = outlier_df[['dicom', 'frame', 'index', feature]]
            
    # iterate through the data frame and find the uuid, index, and feature:
    for index, row in outlier_df.iterrows():

        # append each outlier to the outlier dictionary:
        outliers[feature].append({
             'uuid' : row['uuid'],
             'index' : row['index'],
             'value' : row[feature],
         })

Answer 1

下面是解决该问题的一种方法，即定义一个函数，该函数将输入参数作为列名，并以所需的格式返回当前列中的所有异常值:

def detect_outliers(col):
    # Define your outlier detection condition here
    mask = (df[col] - df[col].mean()).abs() > df[col].std()
    return df.loc[mask, ['uuid', 'index', col]]\
             .rename(columns={col: 'value'}).to_dict('records')

outliers = {col: detect_outliers(col) for col in features}

替代方法更多地涉及 pandas 操作，例如堆叠、分组和聚合:

# Select only feature columns
feature_df = df.set_index(['uuid', 'index'])[features]

# Define your outlier detection condition
mask = (feature_df - feature_df.mean()).abs() > feature_df.std()

# Prepare outlier dataframe
outlier_df = feature_df[mask].stack().reset_index(level=[0, 1], name='value')
outlier_df['records'] = outlier_df.to_dict('r')

# Get the outliers in the desired format
outliers = outlier_df.groupby(level=0).agg(list)['records'].to_dict()

---

>>> outliers

{
    'Atrium': [
        {'uuid': 'di-efg', 'index': 9, 'value': 1.23}
    ],
    'Ventricle': [
        {'uuid': 'di-efg', 'index': 7, 'value': 0.3},
        {'uuid': 'di-efg', 'index': 9, 'value': 0.51}
    ]
}