python - Python BDD 工具的 Bazel 测试规则 "behave"

Question

问题:

你好，亲爱的 bazel 英雄们!

对于我们的系统测试，我正在尝试获取行为驱动开发工具 behave与 bazel 一起运行

• bazel test system/acceptance_criteria:nice

需要完成什么:

• 调用命令行 behave <feature_file> (通过过滤运行文件并使用 <feature_directory> 而不是 <feature_file> 解决)
• 添加文件夹steps到运行文件(通过明确地在文件前加上路径前缀来解决)
• 将相关的 Python 步骤文件添加到运行文件(已解决)
• 将被测系统添加到 PYTHONPATH(打开)

对于如何解决这个问题，您有什么建议或有任何其他建议？

• 使用 Skylark 制定新规则(我已经尝试过了，但在 PYTHONPATH 和运行文件中苦苦挣扎。我得到了一个 ModuleNotFoundError: No module named 'my_system')
• 将 Skylark 用于具有 genrule 的宏
• 在 py_test 中调用脚本(一个可能但不太灵活的解决方法，因为我没有弄清楚如何将参数从 bazel 传递到脚本)
• 一些其他最佳实践？

干杯， 克里斯

如果可能，请提供一个最小示例来重现问题:

我尝试的几乎是复制 py_test 的行为在 Skylark 中让它运行命令 behave <feature_directory>或者 python -m behave <feature_directory> :

project/

• WORKSPACE (空)
• system/
  • BUILD
  • my_system.py
  • acceptance_criteria/
    • BUILD
    • nice.feature
    • steps/
      • reusable_steps.py
• tools/
  • BUILD (空)
  • behave_rule.bzl
  • bazel.rc

system/BUILD :

py_binary(
    name = "main",
    main = "my_system.py",
    srcs = ["my_system.py"],
    # deps = [], excluded for this example
    imports = ["."], # needed for adding PYTHONPATH in acceptance_criteria
    visibility = ["//system/acceptance_criteria:__pkg__"]
)

system/my_system.py :

class MySystem():
    def yeah(self):
        return "yeah it works"

system/acceptance_criteria/BUILD :

load("//tools:behave_rule.bzl", "py_bdd_test")

py_bdd_test(
    name = "nice",
    feats = [
        "nice.feature",
    ],
    steps = [
        "steps/reusable_steps.py",
    ],
    deps = [
        # TODO figure out how to add the implicit imports to the PYTHONPATH
        "//system:main",
    ],
    size = "small",
)

system/acceptance_criteria/nice.feature :

Feature: Multiprocess software

Scenario: 4 processes
  Given the device is powered on

steps/reusable_steps.py :

from behave import *
from my_system import MySystem

@Given("the device is powered on")
def step_impl(context):
    raise NotImplementedError(MySystem().yeah())

tools/behave_rule.bzl :

# =============================================================================
# Description: Adds a test rule for the BDD tool behave to the bazel rule set.
# Knowledge:
# * https://bazel.build/versions/master/docs/skylark/cookbook.html
# * https://bazel.build/versions/master/docs/skylark/rules.html
# * https://bazel.build/versions/master/docs/skylark/lib/ctx.html
# * http://pythonhosted.org/behave/gherkin.html
# =============================================================================

"""Private implementation of the rule py_bdd_test.
"""
def _rule_implementation(ctx):

  # Store the path of the first feature file
  features_dir = ctx.files.feats[0].dirname

  # We want a test target so make it create an executable output.
  # https://bazel.build/versions/master/docs/skylark/rules.html#test-rules
  ctx.file_action(
      # Access the executable output file using ctx.outputs.executable.
      output=ctx.outputs.executable,
      content="behave %s" % features_dir,
      executable=True
  )
  # The executable output is added automatically to this target.

  # Add the feature and step files for behave to the runfiles.
  # https://bazel.build/versions/master/docs/skylark/rules.html#runfiles
  return [DefaultInfo(
      # Create runfiles from the files specified in the data attribute.
      # The shell executable - the output of this rule - can use them at runtime.
      # It is also possible to define data_runfiles and default_runfiles.
      # However if runfiles is specified it's not possible to define the above
      # ones since runfiles sets them both.
      runfiles = ctx.runfiles(
          files = ctx.files.feats + ctx.files.steps + ctx.files.deps)
  )]

"""An example documentation.

Args:
  name:
    A unique name for this rule.
  feats:
    Feature files used to run this target.
  steps:
    Files containing the mapping of feature steps to actual system API calls.
    Note: Since this rule implicitely uses the BDD tool "behave" they must
be in the "steps" folder (https://pythonhosted.org/behave/gherkin.html).
  deps:
    System to test.
"""
py_bdd_test = rule(
    implementation=_rule_implementation,
    attrs={
      # Do not declare "name": It is added automatically.
      "feats": attr.label_list(allow_files=True),
      "steps": attr.label_list(allow_files=True),
      "deps":
        attr.label_list(
            mandatory=True,
            non_empty=True,)
    },
    test=True,
)

tools/bazel.rc :

test --test_output=errors

环境信息

• 操作系统:带 docker 的 Windows 10(参见下面的 Dockerfile)
• Bazel 版本(bazel info release 的输出):0.5.1rc1 版

Dockerfile :

FROM ubuntu:xenial

# Install essentials
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update \
  && apt-get install -y gnupg git \
  && apt-get clean

# -----------------------------------------------------------------------------

# Install the awesome build automation tool bazel
# https://bazel.build/
# version > 0.4.5 since it has a bug in the rule extension skylark
RUN echo "deb http://storage.googleapis.com/bazel-apt testing jdk1.8" \
  > /etc/apt/sources.list.d/bazel.list
RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-key 3D5919B448457EE0
RUN apt-get update \
  && apt-get install -y openjdk-8-jdk bazel \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* && mkdir /var/lib/apt/lists/partial
ENV DEBIAN_FRONTEND ""

# Setup environment for bazel
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

# Run bazel a first time for it to self-extract
RUN /usr/bin/bazel version

# -----------------------------------------------------------------------------

# Install Python (using the non-default version 3.6)
RUN echo "deb http://ppa.launchpad.net/jonathonf/python-3.6/ubuntu xenial main" \
  > /etc/apt/sources.list.d/jonathonf-ubuntu-python-3_6-xenial.list \
  && apt-key adv --keyserver pool.sks-keyservers.net --recv-key 8CF63AD3F06FC659 \
  && apt-get update \
  && apt-get install -y python3.6

# Create symlink from python3.6 to python
RUN ln -s /usr/bin/python3.6 /usr/local/bin/python

# Install the Python package manager pip (for the non-default version 3.6)
# https://en.wikipedia.org/wiki/Pip_(package_manager)
RUN apt-get update \
  && apt-get install -y wget \
  && wget https://bootstrap.pypa.io/get-pip.py \
  && python3.6 get-pip.py \
  && rm get-pip.py \
  && pip install --upgrade pip

# -----------------------------------------------------------------------------

# Install the Gherkin-based BDD tool "behave" for Python
RUN pip install behave

# -----------------------------------------------------------------------------

# Our build environment is based on bazel. Now run tests with it.
ENTRYPOINT "/usr/bin/bazel"

您是否通过网络搜索找到了任何相关内容？

不幸的是没有。如 behave_rule.bzl 的评论中所述我调查了一下

• https://bazel.build/versions/master/docs/skylark/cookbook.html
• https://bazel.build/versions/master/docs/skylark/rules.html
• https://bazel.build/versions/master/docs/skylark/lib/ctx.html

我也找到了

• https://github.com/bazelbuild/bazel/issues/702

还有什么有用的信息、日志或输出吗？

命令行输出:

root@eec1fa791491:/project# bazel test system/acceptance_criteria:nice
INFO: Found 1 test target...
FAIL: //system/acceptance_criteria:nice (see /root/.cache/bazel/_bazel_root/2ca1f4ebdc59348ffdc31d97a51a98d5/execroot/project/bazel-out/local-fastbuild/testlogs/system/acceptance_criteria/nice/test.log).
INFO: From Testing //system/acceptance_criteria:nice:
==================== Test output for //system/acceptance_criteria:nice:
Exception ModuleNotFoundError: No module named 'my_system'
Traceback (most recent call last):
  File "/usr/local/bin/behave", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/behave/__main__.py", line 109, in main
    failed = runner.run()
  File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 672, in run
return self.run_with_paths()
File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 678, in run_with_paths
    self.load_step_definitions()
File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 658, in load_step_definitions
    exec_file(os.path.join(path, name), step_module_globals)
File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 304, in exec_file
    exec(code, globals, locals)
File "system/acceptance_criteria/steps/reusable_steps.py", line 5, in <module>
    from my_system import MySystem
ModuleNotFoundError: No module named 'my_system'
================================================================================
Target //system/acceptance_criteria:nice up-to-date:
bazel-bin/system/acceptance_criteria/nice
INFO: Elapsed time: 0.538s, Critical Path: 0.22s
//system/acceptance_criteria:nice                                        FAILED in 0.2s
/root/.cache/bazel/_bazel_root/2ca1f4ebdc59348ffdc31d97a51a98d5/execroot/project/bazel-out/local-fastbuild/testlogs/system/acceptance_criteria/nice/test.log

Executed 1 out of 1 test: 1 fails locally.

起源于:

• https://github.com/bazelbuild/bazel/issues/3194#issuecomment-309236573

Answer 1

几年过去了，坐下来解决了这个问题

bdd.bzl

"""Custom Bazel Starlark rule for running behave tests.

Idea taken from https://github.com/bazelbuild/examples/
Also read https://docs.bazel.build/versions/main/skylark/concepts.html
"""

def _bdd_test_impl(ctx):
    # Assemble the command to run.
    bdd_tool = 'behave'
    default_args = '--no-capture --no-capture-stderr --no-logcapture'
    # Optional arguments are passed either
    #  explicitly as --test_arg=--foo="bar bar" (no space in between)
    #  or implicitly via ctx.attr.args
    optional_args = '"$@"'

    feature_file = ctx.file.main.basename
    features_path = ctx.label.package

    command = ' '.join([bdd_tool, default_args, features_path, '--include', feature_file])
    command = ' '.join([
        bdd_tool, features_path, '--include', feature_file, default_args, optional_args
    ])

    # Wrap an executable around the command.
    ctx.actions.write(
        output = ctx.outputs.executable,
        content = command,
    )

    # Ensure the files needed by the command are available at runtime
    srcs = ctx.files.main + ctx.files.steps + ctx.files.data
    deps = []
    for dep in ctx.attr.deps:
        # Collecting the files from the dependency tree is a little more hassle
        # since not well-documented and prone to change between bazel versions.
        deps += dep.default_runfiles.files.to_list()

    return [DefaultInfo(
        runfiles = ctx.runfiles(
            files = srcs + deps,
        ),
    )]

bdd_test = rule(
    implementation = _bdd_test_impl,
    test = True,
    attrs = {
        "main": attr.label(
            allow_single_file = [".feature"],
            mandatory = True,
        ),
        "steps": attr.label_list(
            allow_files = True,
        ),
        "deps": attr.label_list(),
        "data": attr.label_list(
            allow_files = True,
        ),
    },
)

构建

load("bdd.bzl", "bdd_test")

[bdd_test(
    name = filename[:filename.rfind(".feature")],
    main = filename,
    steps = ["environment.py"] + glob(["steps/*.py"]),
    deps = [
        # ':some_fixture',
    ],
) for filename in glob(["*.feature"])]