我有一系列文本,它们是自定义 WebText 类的实例。每个文本都是一个对象,具有与其关联的评级(-10 到 +10)和字数(nltk.FreqDist):
>>trainingTexts = [WebText('train1.txt'), WebText('train2.txt'), WebText('train3.txt'), WebText('train4.txt')]
>>trainingTexts[1].rating
10
>>trainingTexts[1].freq_dist
<FreqDist: 'the': 60, ',': 49, 'to': 38, 'is': 34,...>
现在如何获得两个列表(或字典),其中包含仅在正面评分文本中使用的每个单词 (trainingText[]. rating>0),以及另一个列表,其中包含仅在负面文本中使用的每个单词 (trainingText[] .评级<0)。让每个列表包含所有正面或负面文本的总字数,以便您得到如下内容:
>>only_positive_words
[('sky', 10), ('good', 9), ('great', 2)...]
>>only_negative_words
[('earth', 10), ('ski', 9), ('food', 2)...]
我考虑使用集合,因为集合包含唯一的实例,但我看不出如何使用 nltk.FreqDist 来完成此操作,而且最重要的是,集合不会按词频排序。有什么想法吗?
最佳答案
好吧,假设您出于测试目的而开始这样做:
class Rated(object):
def __init__(self, rating, freq_dist):
self.rating = rating
self.freq_dist = freq_dist
a = Rated(5, nltk.FreqDist('the boy sees the dog'.split()))
b = Rated(8, nltk.FreqDist('the cat sees the mouse'.split()))
c = Rated(-3, nltk.FreqDist('some boy likes nothing'.split()))
trainingTexts = [a,b,c]
那么你的代码将如下所示:
from collections import defaultdict
from operator import itemgetter
# dictionaries for keeping track of the counts
pos_dict = defaultdict(int)
neg_dict = defaultdict(int)
for r in trainingTexts:
rating = r.rating
freq = r.freq_dist
# choose the appropriate counts dict
if rating > 0:
partition = pos_dict
elif rating < 0:
partition = neg_dict
else:
continue
# add the information to the correct counts dict
for word,count in freq.iteritems():
partition[word] += count
# Turn the counts dictionaries into lists of descending-frequency words
def only_list(counts, filtered):
return sorted(filter(lambda (w,c): w not in filtered, counts.items()), \
key=itemgetter(1), \
reverse=True)
only_positive_words = only_list(pos_dict, neg_dict)
only_negative_words = only_list(neg_dict, pos_dict)
结果是:
>>> only_positive_words
[('the', 4), ('sees', 2), ('dog', 1), ('cat', 1), ('mouse', 1)]
>>> only_negative_words
[('nothing', 1), ('some', 1), ('likes', 1)]
关于python - 将 nltk.FreqDist 单词分成两个列表?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/10687920/