python - 带有 TFIDF 的 Spark MLLIB K-means 中的索引超出范围用于文本杂乱

Question

我正在尝试使用 spark MLlib 运行 k-means，但出现索引超出范围错误。

我拆分了非常小的示例输入文件，输出如下:-

['hello', 'world', 'this', 'is', 'earth']
['what', 'are', 'you', 'trying', 'to', 'do']
['trying', 'to', 'learn', 'something']
['I', 'am', 'new', 'at', 'this', 'thing']
['what', 'about', 'you']

现在我正在使用 spark 提供的 TFIDF 代码来进行稀疏表示。输出是:-

(1048576,[50570,432125,629096,921177,928731],  [1.09861228867,1.09861228867,0.69314718056,1.09861228867,1.09861228867])
(1048576,[110522,521365,697409,725041,749730,962395],[0.69314718056,1.09861228867,1.09861228867,0.69314718056,0.69314718056,0.69314718056])
(1048576,[4471,725041,850325,962395],[1.09861228867,0.69314718056,1.09861228867,0.69314718056])
(1048576,[36748,36757,84721,167368,629096,704697],[1.09861228867,1.09861228867,1.09861228867,1.09861228867,0.69314718056,1.09861228867])
(1048576,[110522,220898,749730],[0.69314718056,1.09861228867,0.69314718056])

现在我正在运行 MLlib 在 spark 中给出的 k 均值算法:-

clusters = KMeans.train(tfidf_vectors, 2, maxIterations=10)  

def error(point):
    center = clusters.centers[clusters.predict(point)]
    return sqrt(sum([x**2 for x in (point - center)]))

WSSSE = tfidf_vectors.map(lambda point: error(point)).reduce(lambda x, y: x + y)
print("Within Set Sum of Squared Error = " + str(WSSSE))

clusters.save(sc, "myModelPath")
sameModel = KMeansModel.load(sc, "myModelPath")

但我在 WSSSE 步骤中遇到索引超出范围错误。 我做错了什么？

Answer 1

我今天已经遇到过类似的问题，它看起来像 is a bug . TFIDF 像这样创建 SparseVectors:

>>> from pyspark.mllib.linalg import Vectors
>>> sv = Vectors.sparse(5, {1: 3})

并且使用大于最后一个非零值的索引的索引访问值会导致异常:

>>> sv[0]
0.0
>>> sv[1]
3.0
>>> sv[2]
Traceback (most recent call last):
...
IndexError: index out of bounds

虽然效率不高，但解决方法是快速将 SparseVector 转换为 NumPy 数组:

def error(point):                                                         
    center = clusters.centers[clusters.predict(point)]
    return sqrt(sum([x**2 for x in (point.toArray() - center)]))