python - 在 Pandas 数据框中使用 RegexpTokenizer

Question

我正在尝试在数据框的列中应用 RegexpTokenizer。

数据框:

    all_cols
0   who is your hero and why
1   what do you do to relax
2   can't stop to eat
4   how many hours of sleep do you get a night
5   describe the last time you were relax

脚本:

import re
import nltk
import pandas as pd
from nltk import RegexpTokenizer

#tokenization of data and suppression of None (NA)
df['all_cols'].dropna(inplace=True)

tokenizer = RegexpTokenizer("[\w']+")
df['all_cols'] = df['all_cols'].apply(tokenizer)

错误:

TypeError: 'RegexpTokenizer' object is not callable

但是我不明白。当我使用其他 nltk 标记化模式 word_tokenize 时，它​​工作得很好......

Answer 1

请注意，调用 RegexpTokenizer 时，您只是使用一组参数创建类的实例(调用其 __init__ 方法)。 为了使用指定的模式实际标记数据框列，您必须调用其 RegexpTokenizer.tokenize方法:

tokenizer = RegexpTokenizer("[\w']+")
df['all_cols'] = df['all_cols'].map(tokenizer.tokenize)

       all_cols
0  [who, is, your, hero, and, why]
1   [what, do, you, do, to, relax]
...