我正在尝试使用 NLTK 和 pandas 创建术语文档矩阵。 我写了以下函数:
def fnDTM_Corpus(xCorpus):
import pandas as pd
'''to create a Term Document Matrix from a NLTK Corpus'''
fd_list = []
for x in range(0, len(xCorpus.fileids())):
fd_list.append(nltk.FreqDist(xCorpus.words(xCorpus.fileids()[x])))
DTM = pd.DataFrame(fd_list, index = xCorpus.fileids())
DTM.fillna(0,inplace = True)
return DTM.T
运行它
import nltk
from nltk.corpus import PlaintextCorpusReader
corpus_root = 'C:/Data/'
newcorpus = PlaintextCorpusReader(corpus_root, '.*')
x = fnDTM_Corpus(newcorpus)
它适用于语料库中的几个小文件,但当我尝试使用包含 4,000 个文件(每个文件约 2 kb)的语料库运行它时,它会给我一个MemoryError。
我错过了什么吗?
我正在使用 32 位 python。 (我在 Windows 7、64 位操作系统、Core Quad CPU、8 GB RAM 上)。对于这种大小的语料库,我真的需要使用 64 位吗?
最佳答案
我知道 OP 想在 NLTK 中创建一个 tdm,但是 textmining
包(pip install textmining
)让它变得非常简单:
import textmining
# Create some very short sample documents
doc1 = 'John and Bob are brothers.'
doc2 = 'John went to the store. The store was closed.'
doc3 = 'Bob went to the store too.'
# Initialize class to create term-document matrix
tdm = textmining.TermDocumentMatrix()
# Add the documents
tdm.add_doc(doc1)
tdm.add_doc(doc2)
tdm.add_doc(doc3)
# Write matrix file -- cutoff=1 means words in 1+ documents are retained
tdm.write_csv('matrix.csv', cutoff=1)
# Instead of writing the matrix, access its rows directly
for row in tdm.rows(cutoff=1):
print row
输出:
['and', 'the', 'brothers', 'to', 'are', 'closed', 'bob', 'john', 'was', 'went', 'store', 'too']
[1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0]
[0, 2, 0, 1, 0, 1, 0, 1, 1, 1, 2, 0]
[0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1]
或者,可以使用 pandas 和 sklearn [source] :
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
docs = ['why hello there', 'omg hello pony', 'she went there? omg']
vec = CountVectorizer()
X = vec.fit_transform(docs)
df = pd.DataFrame(X.toarray(), columns=vec.get_feature_names())
print(df)
输出:
hello omg pony she there went why
0 1 0 0 0 1 0 1
1 1 1 1 0 0 0 0
2 0 1 0 1 1 1 0
关于python - 使用 NLTK 的高效术语文档矩阵,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/15899861/