python - 如何在 Azure CI 中实现具有代码覆盖率的并行 pytesting

Question

我能够在 Azure CI 中实现并行 pytesting。看这个repo以供引用。 但是代码覆盖率仍然没有按预期工作。 它是单独工作的，但并未结合所有测试覆盖范围。

Here是我正在使用的 Azure 配置文件:

# Python test sample
# Sample that demonstrates how to leverage the parallel jobs capability of Azure Pipelines to run python tests in parallel.
# Parallelizing tests helps in reducing the time spent in testing and can speed up the pipelines significantly.
variables:
  disable.coverage.autogenerate: 'true'

jobs:

- job: 'ParallelTesting'
  pool:
    vmImage: 'windows-latest'
  strategy:
    parallel: 3

  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.7'
      addToPath: true
      architecture: 'x64'

  - script: |
      python -m pip install --upgrade pip setuptools wheel
    displayName: 'Install tools'

  - script: 'pip install -r $(System.DefaultWorkingDirectory)/requirements.txt' 
    displayName: 'Install dependencies'

  - powershell: ./DistributeTests.ps1 
    displayName: 'PowerShell Script to distribute tests'

  - script: |
      pip install pytest-azurepipelines pytest-cov
    displayName: 'Install Pytest dependencies'

  - script: |
      echo $(pytestfiles)
      pytest $(pytestfiles) --junitxml=junit/$(pytestfiles)-results.xml --cov=. --cov-report=xml --cov-report=html
    displayName: 'Pytest'
    continueOnError: true

  - task: PublishTestResults@2
    displayName: 'Publish Test Results **/*-results.xml'
    inputs:
      testResultsFiles: '**/*-results.xml'
      testRunTitle: $(python.version)
    condition: succeededOrFailed()

  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: Cobertura
      summaryFileLocation: '$(System.DefaultWorkingDirectory)/**/coverage.xml'
      reportDirectory: '$(System.DefaultWorkingDirectory)/**/htmlcov'
    displayName: 'Publish code coverage results'

还有 powershell script分发测试:

<#  
.SYNOPSIS  
    Distribute the tests in VSTS pipeline across multiple agents 
.DESCRIPTION  
    This script slices tests files across multiple agents for faster execution.
    We search for specific type of file structure (in this example test*), and slice them according to agent number
    If we encounter multiple files [file1..file10] and if we have 2 agents, agent1 executes tests odd number of files while agent2 executes even number of files
    For detalied slicing info: https://learn.microsoft.com/en-us/vsts/pipelines/test/parallel-testing-any-test-runner
    We use JUnit style test results to publish the test reports.
#>

$tests = Get-ChildItem .\ -Filter "test*" # search for test files with specific pattern.
$totalAgents = [int]$Env:SYSTEM_TOTALJOBSINPHASE # standard VSTS variables available using parallel execution; total number of parallel jobs running
$agentNumber = [int]$Env:SYSTEM_JOBPOSITIONINPHASE  # current job position
$testCount = $tests.Count

# below conditions are used if parallel pipeline is not used. i.e. pipeline is running with single agent (no parallel configuration)
if ($totalAgents -eq 0) {
    $totalAgents = 1
}
if (!$agentNumber -or $agentNumber -eq 0) {
    $agentNumber = 1
}

Write-Host "Total agents: $totalAgents"
Write-Host "Agent number: $agentNumber"
Write-Host "Total tests: $testCount"

$testsToRun= @()

# slice test files to make sure each agent gets unique test file to execute
For ($i=$agentNumber; $i -le $testCount;) {
    $file = $tests[$i-1]
    $testsToRun = $testsToRun + $file
    Write-Host "Added $file"
    $i = $i + $totalAgents 
 }

# join all test files seperated by space. pytest runs multiple test files in following format pytest test1.py test2.py test3.py
$testFiles = $testsToRun -Join " "
Write-Host "Test files $testFiles"
# write these files into variable so that we can run them using pytest in subsequent task. 
Write-Host "##vso[task.setvariable variable=pytestfiles;]$testFiles" 

如果你看一下the pipeline ，可以看到 pytests 正常通过。它还相应地创建代码覆盖率报告。我认为问题在于将代码覆盖率报告整合到一个报告中。

现在如果要查找 summary of last run , 可以注意到每次运行只有一个附件。这很可能是最后执行的作业附件。 在本例中为 test_chrome.py-results.xml。

Answer 1

如果我没有遗漏任何内容，您需要在管道中的某处调用coverage combine(目前您不需要)，然后上传组合的覆盖范围。

❯ coverage --help
Coverage.py, version 6.4 with C extension
Measure, collect, and report on code coverage in Python programs.

usage: coverage <command> [options] [args]

Commands:
    annotate    Annotate source files with execution information.
    combine     Combine a number of data files.
...

关于在不同工作人员中分发 pytest 的 powershell 脚本，您可以使用 pytest_collection_modifyitems 直接指示 pytest 为您执行此操作在 conftest.py 中，或者您也可以安装 pytest-azure-devops