python - 将 Cythonization 添加到 setup.py 时，Travis 日志颜色消失

Question

来自https://travis-ci.org/nltk/nltk/builds/529221349 ，我们看到几个构建和所有其他构建在运行 tox 时都有带有通过/失败/跳过颜色的日志，例如

在 py-travis 环境中使用 tox.ini 配置:

[tox]
envlist =
    py{27,35,36,37}
    pypy
    py{27,35,36}-nodeps
    py{27,35,36}-jenkins
    py-cythonized
    py-travis

[testenv]
; simplify numpy installation
setenv =
    LAPACK=
    ATLAS=None
    PYTHONWARNINGS=ignore

; Copy all environment variables to the tox test environment
passenv = *

deps =
    numpy
    nose >= 1.2.1
    coverage
    text-unidecode
    twython
    pyparsing
    python-crfsuite
    rednose

changedir = nltk/test
commands =
    ; scipy and scikit-learn requires numpy even to run setup.py so
    ; they can't be installed in one command
    pip install scipy scikit-learn

    ; python runtests.py --with-coverage --cover-inclusive --cover-package=nltk --cover-html --cover-html-dir={envdir}/docs []
    python runtests.py []

commands =
    python runtests.py []


[testenv:py-travis]
extras = all
setenv =
    NLTK_DATA = {homedir}/nltk_data/
commands = {toxinidir}/tools/travis/coverage-pylint.sh

但是当来自 setup.py 的 cythonization 启动时，tox 和 travis 配置似乎相同，

[testenv:py-cythonized]
deps =
    Cython >= 0.28.5
setenv =
    CYTHONIZE_NLTK = true
    NLTK_DATA = {homedir}/nltk_data/
extras = all
commands = {toxinidir}/tools/travis/coverage-pylint.sh

运行构建时，颜色消失:

安装 setup.py 与 py-travis 和 py-cynthonized 版本完全相同:

# Work around mbcs bug in distutils.
# http://bugs.python.org/issue10945
import codecs
try:
    codecs.lookup('mbcs')
except LookupError:
    ascii = codecs.lookup('ascii')
    func = lambda name, enc=ascii: {True: enc}.get(name == 'mbcs')
    codecs.register(func)

import os

# Use the VERSION file to get NLTK version
version_file = os.path.join(os.path.dirname(__file__), 'nltk', 'VERSION')
with open(version_file) as fh:
    nltk_version = fh.read().strip()

# setuptools
from setuptools import setup, find_packages

# Specify groups of optional dependencies
extras_require = {
    'machine_learning': ['gensim', 'numpy', 'python-crfsuite', 'scikit-learn', 'scipy'],
    'plot': ['matplotlib'],
    'tgrep': ['pyparsing'],
    'twitter': ['twython'],
    'corenlp': ['requests'],
}

# Add a group made up of all optional dependencies
extras_require['all'] = set(
    package for group in extras_require.values() for package in group
)

MODULES_TO_COMPILE = [
    'nltk.grammar',
    'nltk.parse.chart',
    'nltk.tokenize.*',
    'nltk.probability',
    'nltk.util',
    'nltk.stem.*',
    'nltk.lm.*',
    'nltk.translate.*',
    'nltk.tbl.*',
    'nltk.sentiment.*',
    'nltk.cluster.*',
    'nltk.classify.*',
    'nltk.metrics.*',
    'nltk.chunk.*',
    'nltk.sem.*',

]


def compile_modules(modules):
    """
    Compile the named modules using Cython, using the clearer Python 3 semantics.
    """
    import Cython
    from Cython.Build import cythonize
    files = [name.replace('.', os.path.sep) + '.py' for name in modules]
    print("Compiling %d modules using Cython %s" % (len(modules), Cython.__version__))
    return cythonize(files, language_level=3)


if os.getenv('CYTHONIZE_NLTK') == 'true':
    ext_modules = compile_modules(MODULES_TO_COMPILE)
else:
    ext_modules = None

setup(
    name="nltk",
    description="Natural Language Toolkit",
    version=nltk_version,
    url="http://nltk.org/",
    long_description="""\
The Natural Language Toolkit (NLTK) is a Python package for
natural language processing.  NLTK requires Python 2.7, 3.5, 3.6, or 3.7.""",
    license="Apache License, Version 2.0",
    keywords=[
        'NLP',
        'CL',
        'natural language processing',
        'computational linguistics',
        'parsing',
        'tagging',
        'tokenizing',
        'syntax',
        'linguistics',
        'language',
        'natural language',
        'text analytics',
    ],
    maintainer="Steven Bird",
    maintainer_email="stevenbird1@gmail.com",
    author="Steven Bird",
    author_email="stevenbird1@gmail.com",
    classifiers=[
        'Development Status :: 5 - Production/Stable',
        'Intended Audience :: Developers',
        'Intended Audience :: Education',
        'Intended Audience :: Information Technology',
        'Intended Audience :: Science/Research',
        'License :: OSI Approved :: Apache Software License',
        'Operating System :: OS Independent',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
        'Programming Language :: Python :: 3.7',
        'Topic :: Scientific/Engineering',
        'Topic :: Scientific/Engineering :: Artificial Intelligence',
        'Topic :: Scientific/Engineering :: Human Machine Interfaces',
        'Topic :: Scientific/Engineering :: Information Analysis',
        'Topic :: Text Processing',
        'Topic :: Text Processing :: Filters',
        'Topic :: Text Processing :: General',
        'Topic :: Text Processing :: Indexing',
        'Topic :: Text Processing :: Linguistic',
    ],
    package_data={'nltk': ['test/*.doctest', 'VERSION']},
    install_requires=[
        'six',
        'singledispatch; python_version < "3.4"'
    ],
    extras_require=extras_require,
    packages=find_packages(),
    ext_modules=ext_modules,
    zip_safe=False,  # since normal files will be present too?
)

为什么 Cythonized 版本的颜色会消失？

如何为 Cythonized 构建启用颜色？

<小时/>

对于一些背景，代码来自 nltk 库，cynthonization 测试/构建的完整分支位于 https://github.com/alvations/nltk/tree/cythonize

Answer 1

Why does the color disappear for the Cythonized builds?

因为 py-cythonized 环境未安装 nose (这就是为什么首先使用 stdlib 的 unittest 运行测试的原因)和rednose( Nose 输出着色)。

发生这种情况是因为环境已覆盖deps。 py-travis 没有声明任何自己的 deps，因此它从全局 testenv 配置继承了 deps 设置。 py-cythonized 需要 Cython，因此它重新定义了 deps 列表，丢失了测试执行所需的所有包。

How to enable the color for the Cythonized builds?

将依赖项从全局 testenv 复制到 py-cythonized。建议的补丁:

diff --git a/tox.ini b/tox.ini
index a267d9a5a..41740e19b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -133,6 +133,14 @@ commands =
 [testenv:py-cythonized]
 deps =
     Cython >= 0.28.5
+    numpy
+    nose >= 1.2.1
+    coverage
+    text-unidecode
+    twython
+    pyparsing
+    python-crfsuite
+    rednose
 setenv =
     CYTHONIZE_NLTK = true
     NLTK_DATA = {homedir}/nltk_data/

应用补丁后，测试输出的颜色很好:example run on travis

更新:使用条件设置

为了避免重复依赖，您可以使用条件依赖:

[testenv]
deps =
    rednose  # this one is global dependency
    py-cythonized: cython  # this one is specific for testenv:py-cythonized

建议的补丁:

diff --git a/tox.ini b/tox.ini
index a267d9a5a..fa0839b96 100644
--- a/tox.ini
+++ b/tox.ini
@@ -26,6 +26,7 @@ deps =
     pyparsing
     python-crfsuite
     rednose
+    py-cythonized: Cython >= 0.28.5

 changedir = nltk/test
 commands =
@@ -131,8 +132,6 @@ commands =

 # Test Cython compiled installation.
 [testenv:py-cythonized]
-deps =
-    Cython >= 0.28.5
 setenv =
     CYTHONIZE_NLTK = true
     NLTK_DATA = {homedir}/nltk_data/

来源:Factors and factor-conditional settings