我有一个包含几千行文本数据的 df。我正在使用 spaCy 在该 df 的单个列上执行一些 NLP,并尝试使用以下方法从我的文本数据中删除专有名词、停用词和标点符号:
tokens = []
lemma = []
pos = []
for doc in nlp.pipe(df['TIP_all_txt'].astype('unicode').values, batch_size=9845,
n_threads=3):
if doc.is_parsed:
tokens.append([n.text for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
lemma.append([n.lemma_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
pos.append([n.pos_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
else:
tokens.append(None)
lemma.append(None)
pos.append(None)
df['s_tokens_all_txt'] = tokens
df['s_lemmas_all_txt'] = lemma
df['s_pos_all_txt'] = pos
df.head()
但是我得到了这个错误,我不确定为什么:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-34-73578fd46847> in <module>()
6 n_threads=3):
7 if doc.is_parsed:
----> 8 tokens.append([n.text for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
9 lemma.append([n.lemma_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
10 pos.append([n.pos_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
<ipython-input-34-73578fd46847> in <listcomp>(.0)
6 n_threads=3):
7 if doc.is_parsed:
----> 8 tokens.append([n.text for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
9 lemma.append([n.lemma_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
10 pos.append([n.pos_ for n in doc if not n.is_punct and not n.is_stop and not n.is_space and not n.is_propn])
AttributeError: 'spacy.tokens.token.Token' object has no attribute 'is_propn'
如果我取出 not n.is_propn,代码将按预期运行。我用谷歌搜索并阅读了 spaCy 文档,但到目前为止还没有找到答案。
最佳答案
我没有在 Token
object 上看到可用的 is_propn
属性.
我认为您应该将词性类型检查为PROPN
( reference ):
from spacy.parts_of_speech import PROPN
def is_proper_noun(token):
if token.doc.is_tagged is False: # check if the document was POS-tagged
raise ValueError('token is not POS-tagged')
return token.pos == PROPN
关于python - 从文本中剥离专有名词,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/48103536/