python - 是否可以在 pytest 中并行执行相同的测试？

Question

我知道您可以参数化测试以使用不同的参数集重复测试。我也知道测试文件中的不同测试可以使用 -n 并行运行，但我想并行执行同一组测试。在 pytest 中可以吗？

例如:

import pytest

@pytest.fixture()
def user_number(worker_id):
    return "user number : %s" %worker_id

def test_add(user_number):
    print("Adding 1+1 and returning the result and user number: {}".format(user_number))
    return 1+1

def test_subtract():
    print("subtracting 2-1 and returning the result and user number: {}".format(user_number))
    return 2-1

如果我运行以下命令: py.test -n 3 -s -v parallel_users.py

在结果中，test_add() 和 test_subtract() 并行运行，如下所示:

[gw1] PASSED parallel_users.py::test_subtract 
[gw0] PASSED parallel_users.py::test_add

如何让 test_add() 和 test_subtract() 运行两次，如下所示:

[gw1] PASSED parallel_users.py::test_add, test_subtract 
[gw0] PASSED parallel_users.py::test_add, test_subtract 

Answer 1

更新:

我想最接近 OP 正在寻找的是使用 each distscope。使用它将执行测试选择 n 次，每个进程一次:

$ pytest -n3 -v --dist=each
...
gw0 [2] / gw1 [2] / gw2 [2]
scheduling tests via EachScheduling

test_main.py::test_add 
[gw1] [ 50%] PASSED test_main.py::test_add 
[gw0] [ 50%] PASSED test_main.py::test_add 
test_main.py::test_subtract 
[gw2] [ 50%] PASSED test_main.py::test_add 
test_main.py::test_subtract 
[gw2] [100%] PASSED test_main.py::test_subtract 
[gw0] [100%] PASSED test_main.py::test_subtract 
[gw1] [100%] PASSED test_main.py::test_subtract 

test_add 和 test_subtract 在每个 worker gw0、gw1 和 gw2< 中执行一次，每个测试最多执行 3 次。

旧答案

要重复执行测试，请在您的 conftest.py 中添加一个钩子(Hook):

def pytest_collection_modifyitems(items):
    numrepeats = 2
    items.extend(items * (numrepeats - 1))

这将复制为执行 numrepeats 次而收集的每个测试。运行示例:

$ pytest test_spam.py -v -n3
============================= test session starts ==============================
platform darwin -- Python 3.6.4, pytest-3.4.2, py-1.5.3, pluggy-0.6.0 -- /Users/hoefling/.virtualenvs/stackoverflow/bin/python3.6
cachedir: .pytest_cache
rootdir: /private/tmp, inifile:
plugins: xdist-1.22.2, forked-0.2, dependency-0.3.2, cov-2.5.1
[gw0] darwin Python 3.6.4 cwd: /private/tmp
[gw1] darwin Python 3.6.4 cwd: /private/tmp
[gw2] darwin Python 3.6.4 cwd: /private/tmp
[gw0] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw1] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
[gw2] Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28)  -- [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]
gw0 [4] / gw1 [4] / gw2 [4]
scheduling tests via LoadScheduling

test_spam.py::test_add
test_spam.py::test_subtract
test_spam.py::test_add
[gw0] [ 25%] PASSED test_spam.py::test_add
[gw1] [ 50%] PASSED test_spam.py::test_subtract
[gw2] [ 50%] PASSED test_spam.py::test_add
test_spam.py::test_subtract
[gw0] [ 50%] PASSED test_spam.py::test_subtract

=========================== 4 passed in 0.63 seconds ===========================

如果你想让它可配置，添加一个自定义的 cli 参数:

import pytest

def pytest_addoption(parser):
    parser.addoption('--numrepeats', action='store', type=int, default=1)

def pytest_collection_modifyitems(items):
    numrepeats = pytest.config.getoption('--numrepeats')
    items.extend(items * (numrepeats - 1))

现在您可以使用 --numrepeats 调用您的测试，例如 pytest --numrepeats 5。

---

至于每个进程的批处理测试(问题的第二部分)，pytest-xdist 尚不支持，请参阅 this issue以及与之相关的所有内容。最近，添加了一些基本支持，例如在单独的进程中的单个模块或类中执行测试:

--dist=distmode       set mode for distributing tests to exec environments.
                      each: send each test to all available environments.
                      load: load balance by sending any pending test to any
                      available environment. loadscope: load balance by
                      sending pending groups of tests in the same scope to
                      any available environment. loadfile: load balance by
                      sending test grouped by file to any available
                      environment. (default) no: run tests inprocess, don't
                      distribute.

但是，如果您想根据某些自定义条件对测试进行负载平衡，除了编写您自己的调度程序实现之外别无他法。