我正在使用 nltk 的 PunkSentenceTokenizer
将文本标记为一组句子。但是,分词器似乎不会将新段落或新行视为新句子。
>>> from nltk.tokenize.punkt import PunktSentenceTokenizer
>>> tokenizer = PunktSentenceTokenizer()
>>> tokenizer.tokenize('Sentence 1 \n Sentence 2. Sentence 3.')
['Sentence 1 \n Sentence 2.', 'Sentence 3.']
>>> tokenizer.span_tokenize('Sentence 1 \n Sentence 2. Sentence 3.')
[(0, 24), (25, 36)]
我希望它也将换行视为句子的边界。无论如何要这样做(我也需要保存偏移量)?
最佳答案
好吧,我遇到了同样的问题,我所做的是将文本拆分为 '\n'。像这样的:
# in my case, when it had '\n', I called it a new paragraph,
# like a collection of sentences
paragraphs = [p for p in text.split('\n') if p]
# and here, sent_tokenize each one of the paragraphs
for paragraph in paragraphs:
sentences = tokenizer.tokenize(paragraph)
这是我在生产中的简化版本,但总体思路是相同的。而且,对于葡萄牙语的评论和文档字符串感到抱歉,这是为了巴西观众的“教育目的”
def paragraphs(self):
if self._paragraphs is not None:
for p in self._paragraphs:
yield p
else:
raw_paras = self.raw_text.split(self.paragraph_delimiter)
gen = (Paragraph(self, p) for p in raw_paras if p)
self._paragraphs = []
for p in gen:
self._paragraphs.append(p)
yield p
关于python - nltk 句子标记器,将新行视为句子边界,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/29041603/