python - 标记化单词列表

Question

我在 pandas df 中有一个列，它已使用以下方法进行标记化:

df['token_col'] = df.col.apply(word_tokenize)

现在我尝试使用以下方法标记这些标记化单词:

df['pos_col'] = nltk.tag.pos_tag(df['token_col'])
df['wordnet_tagged_pos_col'] = [(w,get_wordnet_pos(t)) for (w, t) in (df['pos_col'])]

但是我遇到了一个我不太明白的错误:

AttributeError                            Traceback (most recent call last)
<ipython-input-28-99d28433d090> in <module>()
      1 #tag tokenized lists
----> 2 df['pos_col'] = nltk.tag.pos_tag(df['token_col'])
      3 df['wordnet_tagged_pos_col'] = [(w,get_wordnet_pos(t)) for (w, t) in (df['pos_col'])]

C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tag\__init__.py in pos_tag(tokens, tagset, lang)
    125     """
    126     tagger = _get_tagger(lang)
--> 127     return _pos_tag(tokens, tagset, tagger)
    128 
    129 

C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tag\__init__.py in _pos_tag(tokens, tagset, tagger)
     93 
     94 def _pos_tag(tokens, tagset, tagger):
---> 95     tagged_tokens = tagger.tag(tokens)
     96     if tagset:
     97         tagged_tokens = [(token, map_tag('en-ptb', tagset, tag)) for (token, tag) in tagged_tokens]

C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tag\perceptron.py in tag(self, tokens)
    150         output = []
    151 
--> 152         context = self.START + [self.normalize(w) for w in tokens] + self.END
    153         for i, word in enumerate(tokens):
    154             tag = self.tagdict.get(word)

C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tag\perceptron.py in <listcomp>(.0)
    150         output = []
    151 
--> 152         context = self.START + [self.normalize(w) for w in tokens] + self.END
    153         for i, word in enumerate(tokens):
    154             tag = self.tagdict.get(word)

C:\Users\egagne\AppData\Local\Continuum\Anaconda3\lib\site-packages\nltk\tag\perceptron.py in normalize(self, word)
    236         if '-' in word and word[0] != '-':
    237             return '!HYPHEN'
--> 238         elif word.isdigit() and len(word) == 4:
    239             return '!YEAR'
    240         elif word[0].isdigit():

AttributeError: 'list' object has no attribute 'isdigit'

如果有影响，我的下一步将使用以下方法对这些标记的标记进行词形还原:

df['lmtzd_col'] = [(lmtzr.lemmatize(w, pos=t if t else 'n').lower(),t) for (w,t) in wordnet_tagged_pos_col]
print(len(set(wordnet_tagged_pos_col)),(len(set(df['lmtzd_col']))))

我的 df 超过 70 列宽，所以这里是一个小快照:

ID_number   Meeting1    Meeting2    Meeting3    Meeting4    Meeting5    col    
123456789   9/15/2015   1/8/2016    4/27/2016   NaN         NaN         [Assessment, of, Improvement, will, be, on-goi...   
987654321   9/22/2016   NaN         2/25/2017   NaN         NaN         [A, member, of, the, administrative, team, wil..   
456789123   10/1/2015   11/30/2015  NaN         NaN         NaN         [During, our, second, and, third, meetings, we...

Answer 1

您可以使用 apply 来获取词性标签，即

df['pos_col'] = df['token_col'].apply(nltk.tag.pos_tag)

df['pos_col']

0    [(Assessment, NNP), ( of, NNP), ( Improvement,...
1    [(A, DT), ( member, NNP), ( of, NNP), ( the, N...
2    [(During, IN), ( our, JJ), ( second, NN), ( an...
Name: pos_col, dtype: object

similarly its better you use apply function with lambda to apply the function on every row than passing the series to the function like

df['wordnet_tagged_pos_col'] = df['pos_col'].apply(lambda x : [(w,get_wordnet_pos(t)) for (w, t) in x],1)

因为您需要对列的每个单元格应用 get_wordnet_pos 。

df['wordnet_tagged_pos_col']

0    [(Assessment, (N, n)), ( of, (N, n)), ( Improv...
1    [(A, (D, n)), ( member, (N, n)), ( of, (N, n))...
2    [(During, (I, n)), ( our, (J, a)), ( second, (...
Name: wordnet_tagged_pos_col, dtype: object

希望有帮助。