我有一个形状为 (149,1001) 的 TF-IDF 矩阵。想要的是计算最后一列的余弦相似度,所有列
这是我所做的
from numpy import dot
from numpy.linalg import norm
for i in range(mat.shape[1]-1):
cos_sim = dot(mat[:,i], mat[:,-1])/(norm(mat[:,i])*norm(mat[:,-1]))
cos_sim
但是这个循环让它变慢了。那么,有没有什么有效的方法呢?我只想用 numpy
最佳答案
杠杆2D
矢量化 matrix-multiplication
这是 NumPy 对二维数据使用矩阵乘法的一个 -
p1 = mat[:,-1].dot(mat[:,:-1])
p2 = norm(mat[:,:-1],axis=0)*norm(mat[:,-1])
out1 = p1/p2
说明: p1
是 dot(mat[:,i], mat[:,-1])
循环的矢量化等价物. p2
是 (norm(mat[:,i])*norm(mat[:,-1]))
.用于验证的 sample 运行 -
In [57]: np.random.seed(0)
...: mat = np.random.rand(149,1001)
In [58]: out = np.empty(mat.shape[1]-1)
...: for i in range(mat.shape[1]-1):
...: out[i] = dot(mat[:,i], mat[:,-1])/(norm(mat[:,i])*norm(mat[:,-1]))
In [59]: p1 = mat[:,-1].dot(mat[:,:-1])
...: p2 = norm(mat[:,:-1],axis=0)*norm(mat[:,-1])
...: out1 = p1/p2
In [60]: np.allclose(out, out1)
Out[60]: True
时间——In [61]: %%timeit
...: out = np.empty(mat.shape[1]-1)
...: for i in range(mat.shape[1]-1):
...: out[i] = dot(mat[:,i], mat[:,-1])/(norm(mat[:,i])*norm(mat[:,-1]))
18.5 ms ± 977 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [62]: %%timeit
...: p1 = mat[:,-1].dot(mat[:,:-1])
...: p2 = norm(mat[:,:-1],axis=0)*norm(mat[:,-1])
...: out1 = p1/p2
939 µs ± 29.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
# @yatu's soln
In [89]: a = mat
In [90]: %timeit cosine_similarity(a[None,:,-1] , a.T[:-1])
2.47 ms ± 461 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
进一步优化 norm
与 einsum
或者,我们可以计算
p2
与 np.einsum
.所以,
norm(mat[:,:-1],axis=0)
可以替换为:np.sqrt(np.einsum('ij,ij->j',mat[:,:-1],mat[:,:-1]))
因此,给我们一个修改后的 p2
:p2 = np.sqrt(np.einsum('ij,ij->j',mat[:,:-1],mat[:,:-1]))*norm(mat[:,-1])
与之前相同设置的时间 -In [82]: %%timeit
...: p1 = mat[:,-1].dot(mat[:,:-1])
...: p2 = np.sqrt(np.einsum('ij,ij->j',mat[:,:-1],mat[:,:-1]))*norm(mat[:,-1])
...: out1 = p1/p2
607 µs ± 132 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
30x+
加速超过循环!
关于python - 如何找到一个向量与矩阵的余弦相似度,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/63690068/