c++ - 使用 distutils 加快构建过程

Question

我正在为 Python 编写一个 C++ 扩展，并且我正在使用 distutils 来编译该项目。随着项目的发展，重建它需要的时间越来越长。有没有办法加快构建过程？

我了解到 distutils 无法进行并行构建(与 make -j 一样)。有没有比 distutils 更快的好替代品？

我还注意到，每次我调用 python setup.py build 时它都会重新编译所有目标文件，即使我只更改了一个源文件。应该是这种情况还是我在这里做错了什么？

如果有帮助，以下是我尝试编译的一些文件:https://gist.github.com/2923577

谢谢!

Answer 1

1. 尝试使用环境变量CC="ccache gcc" 进行构建，这将在源未更改的情况下显着加快构建速度。 (奇怪的是，distutils 也将 CC 用于 c++ 源文件)。当然要安装 ccache 包。

2. 由于您有一个由多个编译的目标文件组装而成的单个扩展名，因此您可以对 distutils 进行猴子补丁以并行编译这些文件(它们是独立的) - 将其放入您的 setup.py 中(根据需要调整 N=2):

   # monkey-patch for parallel compilation
   def parallelCCompile(self, sources, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None):
       # those lines are copied from distutils.ccompiler.CCompiler directly
       macros, objects, extra_postargs, pp_opts, build = self._setup_compile(output_dir, macros, include_dirs, sources, depends, extra_postargs)
       cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
       # parallel code
       N=2 # number of parallel compilations
       import multiprocessing.pool
       def _single_compile(obj):
           try: src, ext = build[obj]
           except KeyError: return
           self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
       # convert to list, imap is evaluated on-demand
       list(multiprocessing.pool.ThreadPool(N).imap(_single_compile,objects))
       return objects
   import distutils.ccompiler
   distutils.ccompiler.CCompiler.compile=parallelCCompile
   

3. 为了完整起见，如果您有多个扩展，您可以使用以下解决方案:

   import os
   import multiprocessing
   try:
       from concurrent.futures import ThreadPoolExecutor as Pool
   except ImportError:
       from multiprocessing.pool import ThreadPool as LegacyPool
   
       # To ensure the with statement works. Required for some older 2.7.x releases
       class Pool(LegacyPool):
           def __enter__(self):
               return self
   
           def __exit__(self, *args):
               self.close()
               self.join()
   
   def build_extensions(self):
       """Function to monkey-patch
       distutils.command.build_ext.build_ext.build_extensions
   
       """
       self.check_extensions_list(self.extensions)
   
       try:
           num_jobs = os.cpu_count()
       except AttributeError:
           num_jobs = multiprocessing.cpu_count()
   
       with Pool(num_jobs) as pool:
           pool.map(self.build_extension, self.extensions)
   
   def compile(
       self, sources, output_dir=None, macros=None, include_dirs=None,
       debug=0, extra_preargs=None, extra_postargs=None, depends=None,
   ):
       """Function to monkey-patch distutils.ccompiler.CCompiler"""
       macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
           output_dir, macros, include_dirs, sources, depends, extra_postargs
       )
       cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
   
       for obj in objects:
           try:
               src, ext = build[obj]
           except KeyError:
               continue
           self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
   
       # Return *all* object filenames, not just the ones we just built.
       return objects
   
   
   from distutils.ccompiler import CCompiler
   from distutils.command.build_ext import build_ext
   build_ext.build_extensions = build_extensions
   CCompiler.compile = compile