tensorflow - 连续机器学习管道因 tensorflow 需求安装而中断

Question

我正在我自己的 GitLab 服务器上进行持续机器学习 ( https://cml.dev/ )。我的目标是以 python 脚本为例测试基本的连续机器学习管道。

我的 .gitlab-ci.yml 文件是一个基本文件:

stages:
  - cml_run

cml:
  stage: cml_run
  image: dvcorg/cml-py3:latest
  script:
    - pip3 install -r requirements.txt
    - python train.py

    - cat metrics.txt >> report.md
    - cml-publish confusion_matrix.png --md >> report.md
    - cml-send-comment report.md

requirements.txt中对于pandas、sklearn和Keras来说，已经安装成功。但我收到一条因 TensorFlow 要求安装而损坏的管道

  $ pip3 install -r requirements.txt
 Collecting pandas
   Downloading pandas-1.0.5-cp36-cp36m-manylinux1_x86_64.whl (10.1 MB)
 Collecting sklearn
   Downloading sklearn-0.0.tar.gz (1.1 kB)
 Collecting keras
   Downloading Keras-2.4.3-py2.py3-none-any.whl (36 kB)
 Collecting tensorflow
   Downloading tensorflow-2.2.0-cp36-cp36m-manylinux2010_x86_64.whl (516.2 MB)
 ERROR: Exception:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 188, in _main
     status = self.run(options, args)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/req_command.py", line 185, in wrapper
     return func(self, options, args)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 333, in run
     reqs, check_supported_wheels=not options.target_dir
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 179, in resolve
     discovered_reqs.extend(self._resolve_one(requirement_set, req))
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 362, in _resolve_one
     abstract_dist = self._get_abstract_dist_for(req_to_install)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/resolution/legacy/resolver.py", line 314, in _get_abstract_dist_for
     abstract_dist = self.preparer.prepare_linked_requirement(req)
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 469, in prepare_linked_requirement
     hashes=hashes,
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 259, in unpack_url
     hashes=hashes,
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 130, in get_http_url
     link, downloader, temp_dir.path, hashes
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/operations/prepare.py", line 281, in _download_http_url
     for chunk in download.chunks:
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/progress_bars.py", line 166, in iter
     for x in it:
   File "/usr/local/lib/python3.6/dist-packages/pip/_internal/network/utils.py", line 39, in response_chunks
     decode_content=False,
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/urllib3/response.py", line 564, in stream
     data = self.read(amt=amt, decode_content=decode_content)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/urllib3/response.py", line 507, in read
     data = self._fp.read(amt) if not fp_closed else b""
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/filewrapper.py", line 65, in read
     self._close()
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/filewrapper.py", line 52, in _close
     self.__callback(self.__buf.getvalue())
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/controller.py", line 309, in cache_response
     cache_url, self.serializer.dumps(request, response, body=body)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/cachecontrol/serialize.py", line 72, in dumps
     return b",".join([b"cc=4", msgpack.dumps(data, use_bin_type=True)])
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/__init__.py", line 35, in packb
     return Packer(**kwargs).pack(o)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 936, in pack
     self._pack(obj)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 920, in _pack
     len(obj), dict_iteritems(obj), nest_limit - 1
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 1021, in _pack_map_pairs
     self._pack(v, nest_limit - 1)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 920, in _pack
     len(obj), dict_iteritems(obj), nest_limit - 1
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 1021, in _pack_map_pairs
     self._pack(v, nest_limit - 1)
   File "/usr/local/lib/python3.6/dist-packages/pip/_vendor/msgpack/fallback.py", line 865, in _pack
     return self._buffer.write(obj)
 MemoryError

关于如何解决 GitLab 上的 CML 管道问题有什么想法吗？

Answer 1

如果不了解 .gitlab-ci.yml 文件的更多信息，就很难诊断。但根据 MemoryError 消息，除了其他项目依赖项之外，运行程序似乎没有足够的内存来安装 Tensorflow。

您可以尝试installing TF with the --no-cache-dir flag