python - 正则表达式模式在剥离结尾标点符号的同时标记句子？

Question

我需要制作一个正则表达式模式，可以标记一个句子，并且还可以单独标记标点符号，首字母缩略词、缩写词和连字符除外。

"This is a test sentence. I won't write this sentence. J. Smith lives in the U.S.A. and it is nice there."

应返回为

["This" "is" "a" "test" "sentence" "." "I" "won't" "write" "this" "sentence" "." "J." "Smith" "lives" 
 "in" "the" "U.S.A." "and" "it" "is" "nice" "there" ."]

我当前的代码是:

tokens = re.findall(r'((\.\s)|(\S+))', sentence)

但这不能正常工作。它匹配单词末尾的句号作为单词的一部分。

Answer 1

您可以使用 [^\s.]{2,}|(?:\w|\.)+ 来标记这个特定的示例，但正如 Ryan 提到的，没有自然语言工具包，这是徒劳的做法。

此正则表达式可以匹配两种情况:

1. [^\s.]{2,} 匹配 2 个或多个连续的非空格、非文字句点字符
2. (?:\w|\.)+ 匹配一个或多个连续的单词字符或文字句点

用法:

import re

s = """This is a test sentence. 
I won't write this sentence. J. Smith lives in the U.S.A.  and it is nice there.
"""

for token in re.findall(r"[^\s.]{2,}|(?:\w|\.)+", s):
    print(repr(token))

输出:

'This'
'is'
'a'
'test'
'sentence'
'.'
'I'
"won't"
'write'
'this'
'sentence'
'.'
'J.'
'Smith'
'lives'
'in'
'the'
'U.S.A.'
'and'
'it'
'is'
'nice'
'there'
'.'