python-3.x - 用 Python 模拟 10,000 次硬币翻转非常慢

Question

我正在编写一个模拟，该模拟创建 10,000 个周期，共 25 组，每组由 48 次抛硬币组成。这段代码中的某些内容使其运行速度非常慢。它已经运行了至少 20 分钟并且仍在工作。 R 中的类似模拟运行时间不到 10 秒。

这是我正在使用的Python代码:

import pandas as pd
from random import choices

threshold=17
all_periods = pd.DataFrame()

for i in range(10000):
    simulated_period = pd.DataFrame()
    for j in range(25):
        #Data frame with 48 weeks as rows. Each run through loop adds one more year as column until there are 25
        simulated_period = pd.concat([simulated_period, pd.DataFrame(choices([1, -1], k=48))],\
                                      ignore_index=True, axis=1)
        positives = simulated_period[simulated_period==1].count(axis=1)
        negatives = simulated_period[simulated_period==-1].count(axis=1)
        #Combine positives and negatives that are more than the threshold into single dataframe
        sig = pd.DataFrame([[sum(positives>=threshold), sum(negatives>=threshold)]], columns=['positive', 'negative'])
        sig['total'] = sig['positive'] + sig['negative']
    #Add summary of individual simulation to the others
    all_periods = pd.concat([all_periods, sig])

如果有帮助，这里是快速运行的 R 脚本:

flip <- function(threshold=17){
  #threshold is min number of persistent results we want to see. For example, 17/25 positive or 17/25 negative

  outcomes <- c(1, -1)
  trial <- do.call(cbind, lapply(1:25, function (i) sample(outcomes, 48, replace=T)))
  trial <- as.data.frame(t(trial)) #48 weeks in columns, 25 years in rows.

  summary <- sapply(trial, function(x) c(pos=length(x[x==1]), neg=length(x[x==-1])))
  summary <- as.data.frame(t(summary)) #use data frame so $pos/$neg can be used instead of [1,]/[2,]

  sig.pos <- length(summary$pos[summary$pos>=threshold])
  sig.neg <- length(summary$neg[summary$neg>=threshold])

  significant <- c(pos=sig.pos, neg=sig.neg, total=sig.pos+sig.neg) 

  return(significant)
}

  results <- do.call(rbind, lapply(1:10000, function(i) flip(threshold)))
  results <- as.data.frame(results)

谁能告诉我我在 python 中运行的是什么导致进程变慢？谢谢。

Answer 1

为什么不生成整个大集

idx = pd.MultiIndex.from_product((range(10000), range(25)),
                                 names=('period', 'set'))
df = pd.DataFrame(data=np.random.choice([1,-1], (10000*25, 48)), index=idx)

在我的计算机上花费了大约 120 毫秒。然后是其他操作:

positives = df.eq(1).sum(level=0).gt(17).sum(axis=1).to_frame(name='positives')
negatives = df.eq(-1).sum(level=0).gt(17).sum(axis=1).to_frame(name='negatives')

all_periods = pd.concat( (positives, negatives), axis=1 )

all_periods['total'] = all_periods.sum(1)

额外花费大约 600 毫秒。