Python:列表对象没有属性 'lower' - 但语料库已经是小写

Question

我的语料库是一系列包含 Twitter 数据的文档，并已根据我所知进行了清理和预处理(甚至包括表情符号) - 示例如下:

    0         [national, interest, think, worth, holding, ta...
    1         [must, accurate, diane, abbott, done, calculat...

然后我实例化 TFIDF:

    # Instantiate vectoriser
    vect = TfidfVectorizer()

    # Fit
    vect = TfidfVectorizer(min_df=10, ngram_range = (1,3)).fit(text)

当我尝试适应这个时，我得到:

   AttributeError: 'list' object has no attribute 'lower' 

但我已经将所有内容转换为小写。这与它是一个系列有关吗？

Answer 1

Convert a collection of raw documents to a matrix of TF-IDF features.

从这个意义上讲，您正在复制此处的数据框中传递一系列列表:

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

l1 = 'national, interest, think, worth, holding,'.split(',')
l2 = 'must, accurate, diane, abbott, done'.split(',')

df = pd.DataFrame([[l1],[l2]])

text = df[0]

它将您的文本参数返回为:

0    [national,  interest,  think,  worth,  holding, ]
1            [must,  accurate,  diane,  abbott,  done]
Name: 0, dtype: object

这显然行不通，正如所指出的，TfidfVectorizer 接受字符串或文档。在您的情况和示例中，尽管与您的 example 有点违反直觉。 .

corpus = text.apply(lambda x: ','.join(x)).to_list() # converts your series into a list of strings

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)

print(vectorizer.get_feature_names())

['abbott', 'accurate', 'diane', 'done', 'holding', 'interest', 'must', 'national', 'think', 'worth']