python - 使用 pandas 查找数据框中重复句子的数量

Question

我试图找出我的数据框有多少重复的句子，这是重复超过一个的任何完全匹配的句子，我使用Dataframe.Duplicated，但它忽略了第一个 oucurency句子，我想要它而不是打印重复的句子，只需打印重复的句子之一及其出现的次数

我正在尝试的代码是

wdata = pd.read_csv(fileinput, nrows=0).columns[0]
skip = int(wdata.count(' ') == 0)
wdata = pd.read_csv(fileinput, names=['sentences'], skiprows=skip)
data=wdata[wdata.duplicated()]
print(data)



#dataframe example
#hi how are you
#hello sam how are you doing
#hello sam how are you doing
#helll Alex how are you doing
#hello sam how are you doing
#let us go eat
#where is the dog
#let us go eat 

我希望我的输出是这样的

#hello sam how are you doing   3
#let us go eat                 2

使用重复的函数我得到这个输出

#hello sam how are you doing
#hello sam how are you doing
#let us go eat

这是我通过第二个答案得到的输出

wdata = pd.read_csv(fileinput, nrows=0).columns[0]
skip = int(wdata.count(' ') == 0)
wdata = pd.read_csv(fileinput, names=['sentences'], skiprows=skip)

data=wdata.groupby(['sentences']).size().reset_index(name='counts')


#                      sentences  counts
#0  hello Alex how are you doing       1
#1   hello sam how are you doing       3
#2                hi how are you       1
#3                 let us go eat       1
#4                let us go eat        1
#5              where is the dog       1

我希望我的输出是这样的

#hello sam how are you doing   3
#let us go eat                 2

Answer 1

因为有空格，解决方案是通过 Series.str.strip 删除它们与 GroupBy.size :

data=wdata.groupby(wdata['sentences'].str.strip()).size().reset_index(name='counts')

然后按boolean indexing过滤:

data = data[data['counts'].gt(1)]

另一个想法是使用 Series.value_counts对于系列，过滤并最后转换为 2 列 DataFrame:

s = wdata['sentences'].str.strip().value_counts()
data = s[s.gt(1)].rename_axis('sentences').reset_index(name='counts')