python - 在 Python 中加速一对多相关性计算

Question

我想计算 Python 中向量与数组每一行之间的 Pearson 相关系数(假定为 numpy 和/或 scipy)。由于实际数据数组的大小和内存限制，将无法使用标准相关矩阵计算函数。这是我天真的实现:

import numpy as np
import scipy.stats as sps

np.random.seed(0)

def correlateOneWithMany(one, many):
    """Return Pearson's correlation coef of 'one' with each row of 'many'."""
    pr_arr = np.zeros((many.shape[0], 2), dtype=np.float64)
    pr_arr[:] = np.nan
    for row_num in np.arange(many.shape[0]):
        pr_arr[row_num, :] = sps.pearsonr(one, many[row_num, :])
    return pr_arr

obs, varz = 10 ** 3, 500
X = np.random.uniform(size=(obs, varz))

pr = correlateOneWithMany(X[0, :], X)

%timeit correlateOneWithMany(X[0, :], X)
# 10 loops, best of 3: 38.9 ms per loop

如有任何关于加快这一进程的想法，我们将不胜感激!

Answer 1

模块scipy.spatial.distance实现“相关距离”，它只是一个减去相关系数。您可以使用函数 cdist计算一对多距离，用1减去结果得到相关系数。

这是您的脚本的修改版本，其中包括使用 cdist 计算相关系数:

import numpy as np
import scipy.stats as sps
from scipy.spatial.distance import cdist

np.random.seed(0)

def correlateOneWithMany(one, many):
    """Return Pearson's correlation coef of 'one' with each row of 'many'."""
    pr_arr = np.zeros((many.shape[0], 2), dtype=np.float64)
    pr_arr[:] = np.nan
    for row_num in np.arange(many.shape[0]):
        pr_arr[row_num, :] = sps.pearsonr(one, many[row_num, :])
    return pr_arr

obs, varz = 10 ** 3, 500
X = np.random.uniform(size=(obs, varz))

pr = correlateOneWithMany(X[0, :], X)

c = 1 - cdist(X[0:1, :], X, metric='correlation')[0]

print(np.allclose(c, pr[:, 0]))

时间:

In [133]: %timeit correlateOneWithMany(X[0, :], X)
10 loops, best of 3: 37.7 ms per loop

In [134]: %timeit 1 - cdist(X[0:1, :], X, metric='correlation')[0]
1000 loops, best of 3: 1.11 ms per loop