java - JUnit5:如何重复失败的测试？

Question

许多公司遵循的做法之一是重复不稳定测试，直到通过 x 次(连续或总共)。如果它执行了 n 次并且未能通过至少 x 次，则它被标记为失败。

TestNG 通过以下注解支持:

@Test(invocationCount = 5, successPercentage = 40)

如何使用 JUnit5 实现类似的功能？

JUnit5 中有类似的注解，称为@RepeatedTest(5) 但它不是有条件地执行的。

Answer 1

好的，我花了一点时间整理了一个小例子，说明如何使用 TestTemplateInvocationContextProvider 来做到这一点。 , ExecutionCondition , 和 TestExecutionExceptionHandler扩展点。

我能够处理失败测试的方法是将它们标记为“已中止”，而不是让它们完全失败(这样整个测试执行不会将其视为失败)并且仅在我们不能失败时才使测试失败获得最少的成功运行次数。如果最小数量的测试已经成功，那么我们也将剩余的测试标记为“已禁用”。在 ExtensionContext.Store 中跟踪测试失败以便可以在每个地方查看状态。

这是一个非常粗略的示例，肯定有一些问题，但有望作为如何组合不同注释的示例。我最终用 Kotlin 编写了它:

@Retry - 基于 TestNG 示例的类似注解:

import org.junit.jupiter.api.TestTemplate
import org.junit.jupiter.api.extension.ExtendWith

@TestTemplate
@Target(AnnotationTarget.FUNCTION)
@ExtendWith(RetryTestExtension::class)
annotation class Retry(val invocationCount: Int, val minSuccess: Int)

TestTemplateInvocationContext由模板化测试使用:

import org.junit.jupiter.api.extension.Extension
import org.junit.jupiter.api.extension.TestTemplateInvocationContext

class RetryTemplateContext(
  private val invocation: Int,
  private val maxInvocations: Int,
  private val minSuccess: Int
) : TestTemplateInvocationContext {
  override fun getDisplayName(invocationIndex: Int): String {
    return "Invocation number $invocationIndex (requires $minSuccess success)"
  }

  override fun getAdditionalExtensions(): MutableList<Extension> {
    return mutableListOf(
      RetryingTestExecutionExtension(invocation, maxInvocations, minSuccess)
    )
  }
}

@Retry 注释的

TestTemplateInvocationContextProvider 扩展:

import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ExtensionContextException
import org.junit.jupiter.api.extension.TestTemplateInvocationContext
import org.junit.jupiter.api.extension.TestTemplateInvocationContextProvider
import org.junit.platform.commons.support.AnnotationSupport
import java.util.stream.IntStream
import java.util.stream.Stream

class RetryTestExtension : TestTemplateInvocationContextProvider {
  override fun supportsTestTemplate(context: ExtensionContext): Boolean {
    return context.testMethod.map { it.isAnnotationPresent(Retry::class.java) }.orElse(false)
  }

  override fun provideTestTemplateInvocationContexts(context: ExtensionContext): Stream<TestTemplateInvocationContext> {
    val annotation = AnnotationSupport.findAnnotation(
        context.testMethod.orElseThrow { ExtensionContextException("Must be annotated on method") },
        Retry::class.java
    ).orElseThrow { ExtensionContextException("${Retry::class.java} not found on method") }

    checkValidRetry(annotation)

    return IntStream.rangeClosed(1, annotation.invocationCount)
        .mapToObj { RetryTemplateContext(it, annotation.invocationCount, annotation.minSuccess) }
  }

  private fun checkValidRetry(annotation: Retry) {
    if (annotation.invocationCount < 1) {
      throw ExtensionContextException("${annotation.invocationCount} must be greater than or equal to 1")
    }
    if (annotation.minSuccess < 1 || annotation.minSuccess > annotation.invocationCount) {
      throw ExtensionContextException("Invalid ${annotation.minSuccess}")
    }
  }
}

表示重试的简单数据类(在此示例中使用 ParameterResolver 注入(inject)到测试用例中)。

data class RetryInfo(val invocation: Int, val maxInvocations: Int)

Exception 用于表示失败的重试:

import java.lang.Exception

internal class RetryingTestFailure(invocation: Int, cause: Throwable) : Exception("Failed test execution at invocation #$invocation", cause)

实现 ExecutionCondition、ParameterResolver 和 TestExecutionExceptionHandler 的主要扩展。

import org.junit.jupiter.api.extension.ConditionEvaluationResult
import org.junit.jupiter.api.extension.ExecutionCondition
import org.junit.jupiter.api.extension.ExtensionContext
import org.junit.jupiter.api.extension.ParameterContext
import org.junit.jupiter.api.extension.ParameterResolver
import org.junit.jupiter.api.extension.TestExecutionExceptionHandler
import org.opentest4j.TestAbortedException

internal class RetryingTestExecutionExtension(
  private val invocation: Int,
  private val maxInvocations: Int,
  private val minSuccess: Int
) : ExecutionCondition, ParameterResolver, TestExecutionExceptionHandler {
  override fun evaluateExecutionCondition(
    context: ExtensionContext
  ): ConditionEvaluationResult {
    val failureCount = getFailures(context).size
    // Shift -1 because this happens before test
    val successCount = (invocation - 1) - failureCount
    when {
      (maxInvocations - failureCount) < minSuccess -> // Case when we cannot hit our minimum success
        return ConditionEvaluationResult.disabled("Cannot hit minimum success rate of $minSuccess/$maxInvocations - $failureCount failures already")
      successCount < minSuccess -> // Case when we haven't hit success threshold yet
        return ConditionEvaluationResult.enabled("Have not ran $minSuccess/$maxInvocations successful executions")
      else -> return ConditionEvaluationResult.disabled("$minSuccess/$maxInvocations successful runs have already ran. Skipping run $invocation")
    }
  }

  override fun supportsParameter(
    parameterContext: ParameterContext,
    extensionContext: ExtensionContext
  ): Boolean = parameterContext.parameter.type == RetryInfo::class.java

  override fun resolveParameter(
    parameterContext: ParameterContext,
    extensionContext: ExtensionContext
  ): Any = RetryInfo(invocation, maxInvocations)

  override fun handleTestExecutionException(
    context: ExtensionContext,
    throwable: Throwable
  ) {

    val testFailure = RetryingTestFailure(invocation, throwable)
    val failures: MutableList<RetryingTestFailure> = getFailures(context)
    failures.add(testFailure)
    val failureCount = failures.size
    val successCount = invocation - failureCount
    if ((maxInvocations - failureCount) < minSuccess) {
      throw testFailure
    } else if (successCount < minSuccess) {
      // Case when we have still have retries left
      throw TestAbortedException("Aborting test #$invocation/$maxInvocations- still have retries left",
        testFailure)
    }
  }

  private fun getFailures(context: ExtensionContext): MutableList<RetryingTestFailure> {
    val namespace = ExtensionContext.Namespace.create(
      RetryingTestExecutionExtension::class.java)
    val store = context.parent.get().getStore(namespace)
    @Suppress("UNCHECKED_CAST")
    return store.getOrComputeIfAbsent(context.requiredTestMethod.name, { mutableListOf<RetryingTestFailure>() }, MutableList::class.java) as MutableList<RetryingTestFailure>
  }
}

然后，测试消费者:

import org.junit.jupiter.api.DisplayName

internal class MyRetryableTest {
  @DisplayName("Fail all retries")
  @Retry(invocationCount = 5, minSuccess = 3)
  internal fun failAllRetries(retryInfo: RetryInfo) {
    println(retryInfo)
    throw Exception("Failed at $retryInfo")
  }

  @DisplayName("Only fail once")
  @Retry(invocationCount = 5, minSuccess = 4)
  internal fun succeedOnRetry(retryInfo: RetryInfo) {
    if (retryInfo.invocation == 1) {
      throw Exception("Failed at ${retryInfo.invocation}")
    }
  }

  @DisplayName("Only requires single success and is first execution")
  @Retry(invocationCount = 5, minSuccess = 1)
  internal fun firstSuccess(retryInfo: RetryInfo) {
    println("Running: $retryInfo")
  }

  @DisplayName("Only requires single success and is last execution")
  @Retry(invocationCount = 5, minSuccess = 1)
  internal fun lastSuccess(retryInfo: RetryInfo) {
    if (retryInfo.invocation < 5) {
      throw Exception("Failed at ${retryInfo.invocation}")
    }
  }

  @DisplayName("All required all succeed")
  @Retry(invocationCount = 5, minSuccess = 5)
  internal fun allRequiredAllSucceed(retryInfo: RetryInfo) {
    println("Running: $retryInfo")
  }

  @DisplayName("Fail early and disable")
  @Retry(invocationCount = 5, minSuccess = 4)
  internal fun failEarly(retryInfo: RetryInfo) {
    throw Exception("Failed at ${retryInfo.invocation}")
  }
}

IntelliJ 中的测试输出如下所示:

我不知道从 TestExecutionExceptionHandler.handleTestExecutionException 中抛出 TestAbortedException 是否应该中止测试，但我在这里使用它。