python - 在 Google App Engine 中部署应用程序时出现内存错误

Question

我正在尝试在 Google App Engine 中部署 Flask API，但遇到了 MemoryError。

运行 gcloud app deploy 时，我通过终端收到以下消息:

Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build f857cba3-1bf5-4384-acec-f1ee3ba97c8b status: FAILURE.
Build error details: {"error":{"errorType":"BuildError","canonicalCode":"INVALID_ARGUMENT","errorId":"26CADD66","errorMessage":"14 May 2020 13:48:07 INFO     Arguments: ['--parser_script=/usr/local/bin/ftl.par', '--src=', '--entrypoint=', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Unparsed arguments: ['--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Executing ['/usr/local/bin/ftl.par', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\nINFO     FTL version python-v0.17.0\nINFO     Beginning FTL build for python\nINFO     FTL arg passed: virtualenv_dir /env\nINFO     FTL arg passed: ttl 168\nINFO     FTL arg passed: python_cmd /opt/python3.7/bin/python3.7\nINFO     FTL arg passed: cache True\nINFO     FTL arg passed: virtualenv_cmd virtualenv\nINFO     FTL arg passed: entrypoint None\nINFO     FTL arg passed: exposed_ports None\nINFO     FTL arg passed: pip_cmd /env/bin/python3.7 -m pip\nINFO     FTL arg passed: tar_base_image_path None\nINFO     FTL arg passed: export_cache_stats False\nINFO     FTL arg passed: builder_output_path \"\"\nINFO     FTL arg passed: destination_path /srv\nINFO     FTL arg passed: sh_c_prefix False\nINFO     FTL arg passed: base us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00\nINFO     FTL arg passed: cache_key_version v0.17.0\nINFO     FTL arg passed: cache_salt \nINFO     FTL arg passed: cache_repository us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d\nINFO     FTL arg passed: venv_cmd /opt/python3.7/bin/python3.7 -m venv /env\nINFO     FTL arg passed: name us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0\nINFO     FTL arg passed: global_cache False\nINFO     FTL arg passed: upload True\nINFO     FTL arg passed: fail_on_error True\nINFO     FTL arg passed: output_path None\nINFO     FTL arg passed: directory /workspace\nINFO     FTL arg passed: additional_directory /.googleconfig\nINFO     FTL arg passed: verbosity DEBUG\nINFO     starting: full build\nINFO     starting: builder initialization\nINFO     Loading Docker credentials for repository 'us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00'\nINFO     Loading Docker credentials for repository 'us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0'\nINFO     builder initialization took 0 seconds\nINFO     starting: build process for FTL image\nINFO     starting: checking_cached_interpreter_layer\nINFO     starting: check python version\nINFO     `python version` full cmd:\n/opt/python3.7/bin/python3.7 --version\nINFO     `python version` stderr:\n\nINFO     check python version took 0 seconds\nDEBUG   .

在 GCP 上检查构建细节时，我得到了这个:

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
  File "/usr/local/bin/ftl.par/__main__.py", line 54, in main
  File "/usr/local/bin/ftl.par/__main__/ftl/python/builder.py", line 123, in Build
  File "/usr/local/bin/ftl.par/__main__/ftl/common/layer_builder.py", line 42, in BuildLayer
  File "/usr/local/bin/ftl.par/__main__/ftl/common/ftl_util.py", line 116, in zip_dir_to_layer_sha
MemoryError

我的requirements.txt如下:

scikit-learn==0.20.3
pandas==1.0.3
xgboost==1.0.2
fuzzywuzzy==0.18.0
python-Levenshtein==0.12.0
gensim==3.8.1
nltk==3.3
Flask==1.1.2
flask-cors==3.0.7
requests==2.23.0

我的app.yaml如下:

runtime: python37
instance_class: F4_1G
resources:
  cpu: 3
  memory_gb: 16
  disk_size_gb: 30

我尝试了很多与 cpus 和内存相关的配置，但都没有用。

我认为问题是我正在尝试加载 3.5gb 的 word2vec 模型:

model = gensim.models.KeyedVectors.load_word2vec_format('./GoogleNews-vectors-negative300.bin', binary=True)

Answer 1

在 App Engine 中，app.yaml 可以使用 instance_class 和 resources 之间的标签之一，因为它们都定义了将要使用的机器类型分配给应用引擎实例。

在 App Engine Standar 中，使用的标签是“instance_class”，如前所述here所有具有 instance class 的实例附带 1 GB 的磁盘，然后在尝试上传大于此 1 GB 存储空间的文件时出现错误。

如果您想指定有关将要使用的机器类型的更多信息，例如内存、磁盘大小等，那么您需要在 [App Engine Flex] 中进行部署，您可以在其中指定资源标签。

要在 flex 上部署，app.yaml 需要指定环境是 flex 以及在这种情况下实例的自定义配置。

runtime: python
env: flex

runtime_config:
  python_version: 3

resources:
  cpu: 4
  memory_gb: 16
  disk_size_gb: 30

我将 CPU 从 3 修改为 4，因为此值应为 1 或偶数。

考虑到 App Engine Flex 比标准的更昂贵并且需要更多的时间来部署。

有时这会发送一个 Error Response: [4] Timed out waiting the app infrastructure to become healthy. 错误，预计它可以被抛出，如果你遇到这个问题再试一次一段时间后，如果再次发生，请尝试在不同区域部署该应用。