Python:在句子分段器、分词器和词性标注器中遇到问题

Question

我正在尝试将文本文件读入Python，然后进行句子分段器、分词器和词性标注器。

这是我的代码:

file=open('C:/temp/1.txt','r')
sentences = nltk.sent_tokenize(file)
sentences = [nltk.word_tokenize(sent) for sent in sentences]
sentences = [nltk.pos_tag(sent) for sent in sentences]

当我尝试第二个命令时，它显示错误:

Traceback (most recent call last):
File "<pyshell#26>", line 1, in <module>
sentences = nltk.sent_tokenize(file)
File "D:\Python\lib\site-packages\nltk\tokenize\__init__.py", line 76, in sent_tokenize
return tokenizer.tokenize(text)
File "D:\Python\lib\site-packages\nltk\tokenize\punkt.py", line 1217, in tokenize
return list(self.sentences_from_text(text, realign_boundaries))
File "D:\Python\lib\site-packages\nltk\tokenize\punkt.py", line 1262, in sentences_from_text
sents = [text[sl] for sl in self._slices_from_text(text)]
File "D:\Python\lib\site-packages\nltk\tokenize\punkt.py", line 1269, in _slices_from_text
for match in self._lang_vars.period_context_re().finditer(text):
TypeError: expected string or buffer

再次尝试: 当我尝试只用一个句子，例如“一只黄狗对猫吠叫”时 前三个命令有效，但最后一行，我收到此错误:(我想知道我是否没有完全下载软件包？)

Traceback (most recent call last):
File "<pyshell#16>", line 1, in <module>
sentences = [nltk.pos_tag(sent) for sent in sentences]
File "D:\Python\lib\site-packages\nltk\tag\__init__.py", line 99, in pos_tag
tagger = load(_POS_TAGGER)
File "D:\Python\lib\site-packages\nltk\data.py", line 605, in load
resource_val = pickle.load(_open(resource_url))
ImportError: No module named numpy.core.multiarray

Answer 1

嗯...您确定错误出现在第二行中吗？

您似乎使用了标准 ASCII ' 和 , 字符之外的单引号和逗号字符:

file=open(‘C:/temp/1.txt’，‘r’) # your version (WRONG)
file=open('C:/temp/1.txt', 'r') # right

Python 甚至不应该能够编译这个。事实上，当我尝试它时，它由于语法错误而失败。

更新:您发布了具有正确语法的更正版本。来自回溯的错误消息非常简单:您正在调用的函数似乎需要一大块文本作为其参数，而不是文件对象。虽然具体对NLTK一无所知，花五秒钟Google一下confirms this .

尝试这样的事情:

file = open('C:/temp/1.txt','r')
text = file.read() # read the contents of the text file into a variable
result1 = nltk.sent_tokenize(text)
result2 = [nltk.word_tokenize(sent) for sent in result1]
result3 = [nltk.pos_tag(sent) for sent in result2]

更新:我将sentences重命名为result1/2/3，因为由于重复而对代码实际执行的操作感到困惑覆盖同一个变量。这不会影响语义，只是澄清第二行实际上对最终结果3有影响。