python - 压缩 Conda 环境破坏 Audioread 的后端 (Python/Pyspark)

Question

我之前使用 conda 构建 pyspark 环境来打包所有依赖项并在运行时将它们发送到所有节点。以下是我创建环境的方式:

`conda/bin/conda create -p conda_env --copy -y python=2  \
numpy scipy ffmpeg gcc libsndfile gstreamer pygobject audioread librosa`

`zip -r conda_env.zip conda_env`

然后采购 conda_env 并运行 pyspark shell 我可以成功执行:

`import librosa
y, sr = librosa.load("test.m4a")`

请注意，如果没有环境源，此脚本会导致错误，因为我的本地未安装 ffmpeg/gstreamer。

向集群提交脚本会导致 librosa.load 错误，该错误可追溯到 audioread，表明后端(gstreamer 或 ffmpeg)不再能在压缩存档环境。堆栈跟踪如下:

提交:

`PYSPARK_PYTHON=./NODE/conda_env/bin/python spark-submit --verbose \
        --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./NODE/conda_env/bin/python \
        --conf spark.yarn.appMasterEnv.PYTHON_EGG_CACHE=/tmp \
        --conf spark.executorEnv.PYTHON_EGG_CACHE=/tmp \
        --conf spark.yarn.executor.memoryOverhead=1024 \
        --conf spark.hadoop.validateOutputSpecs=false \
        --conf spark.driver.cores=5 \
        --conf spark.driver.maxResultSize=0 \
        --master yarn --deploy-mode cluster --queue production \
        --num-executors 20 --executor-cores 5 --executor-memory 40G \
        --driver-memory 20G --archives conda_env.zip#NODE \
        --jars /data/environments/sqljdbc41.jar \
        script.py`

跟踪:

`Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 172, in main
    process()
  File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/worker.py", line 167, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/pyspark.zip/pyspark/serializers.py", line 263, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "script.py", line 245, in <lambda>
  File "script.py", line 119, in download_audio
  File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/librosa/core/audio.py", line 107, in load
    with audioread.audio_open(os.path.realpath(path)) as input_file:
  File "/mnt/yarn/usercache/user/appcache/application_1506634200253_39889/container_1506634200253_39889_01_000003/NODE/conda_env/lib/python2.7/site-packages/audioread/__init__.py", line 114, in audio_open
    raise NoBackendError()
NoBackendError`

我的问题是:如何打包此存档，以便 librosa(真正的 audioread)能够找到后端并加载 .m4a 文件？

Answer 1

这是一个路径问题，执行者无法找到 FFMPEG 尽管它是在 conda 环境中打包的。这个 hack 修复了它。

path = os.getenv("PATH")
if "./NODE/conda_env/bin" not in path:
    path += os.pathsep + "./NODE/conda_env/bin"
    os.environ["PATH"] = path
y, _ = librosa.load(audiofn, self.conf.sr)