python:数据清理 - 检测欺诈性电子邮件地址的模式

Question

我正在清理一个包含我正在删除的欺诈性电子邮件地址的数据集。

我建立了多个规则来捕获重复和欺诈域。但是有一个场景，我想不出如何在 python 中编写规则来标记它们。

所以我有这样的规则:

#delete punction
df['email'].apply(lambda x:''.join([i for i in x if i not in string.punctuation]))    

#flag yopmail
pattern = "yopmail"
match = df['email'].str.contains(pattern)
df['yopmail'] = np.where(match, 'Y', '0')

#flag duplicates
df['duplicate']=df.email.duplicated(keep=False)

这是我无法找出捕获它的规则的数据。基本上我正在寻找一种方法来标记以相同方式开始但最后有连续数字的地址。

abc7020@gmail.com
abc7020.1@gmail.com
abc7020.10@gmail.com
abc7020.11@gmail.com
abc7020.12@gmail.com
abc7020.13@gmail.com
abc7020.14@gmail.com
abc7020.15@gmail.com
attn1@gmail.com
attn12@gmail.com
attn123@gmail.com
attn1234@gmail.com
attn12345@gmail.com
attn123456@gmail.com
attn1234567@gmail.com
attn12345678@gmail.com

Answer 1

我的解决方案既不高效也不美观。但是检查一下，看看它是否适合你@jeangelj。它绝对适用于您提供的示例。祝你好运!

import os
from random import shuffle
from difflib import SequenceMatcher

emails = [... ...] # for example the 16 email addresses you gave in your question
shuffle(emails) # everyday i'm shuffling
emails = sorted(emails) # sort that shit!
names = [email.split('@')[0] for email in emails]

T = 0.7 # <- set your string similarity threshold here!!

split_indices=[]
for i in range(1,len(emails)):
    if SequenceMatcher(None, emails[i], emails[i-1]).ratio() < T:
        split_indices.append(i) # we want to remember where dissimilar email address occurs

grouped=[]
for i in split_indices:
    grouped.append(emails[:i])
grouped.append(emails[i:])
# now we have similar email addresses grouped, we want to find the common prefix for each group
prefix_strings=[]
for group in grouped:
    prefix_strings.append(os.path.commonprefix(group))

# finally
ham=[]
spam=[]
true_ids = [names.index(p) for p in prefix_strings]
for i in range(len(emails)):
    if i in true_ids:
        ham.append(emails[i])
    else:
        spam.append(emails[i])

In [30]: ham
Out[30]: ['abc7020@gmail.com', 'attn1@gmail.com']

In [31]: spam
Out[31]: 
['abc7020.10@gmail.com',
 'abc7020.11@gmail.com',
 'abc7020.12@gmail.com',
 'abc7020.13@gmail.com',
 'abc7020.14@gmail.com',
 'abc7020.15@gmail.com',
 'abc7020.1@gmail.com',
 'attn12345678@gmail.com',
 'attn1234567@gmail.com',
 'attn123456@gmail.com',
 'attn12345@gmail.com',
 'attn1234@gmail.com',
 'attn123@gmail.com',
 'attn12@gmail.com']  

# THE TRUTH YALL!