rules_venv

This repository implements Bazel rules for Python and is designed to be a drop-in replacement for the existing rules_python where uses of @rules_python//python:defs.bzl can be replaced with @rules_venv//python:defs.bzl.

Improvements over rules_python

While rules_python has fantastic toolchain infrastructure which this repo relies on, rules_python ultimately suffers from a few issues which this repo aims to solve:

1. Use of PYTHONPATH to construct the python environment leads to operating system limitations.

   Some details on MAX_ARG_STRLEN and ARG_MAX can be found here: https://unix.stackexchange.com/a/120842

2. Slow startup on windows systems that do not support symlinks.

   rules_python creates zipapps on systems that do not support runfiles. For large projects, this can lead to large (~500MB+) zip files being constantly compressed and uncompressed to run simple actions which is a lot more expensive than systems which support runfiles.

Setup

bazel_dep(name = "rules_venv", version = "{version}")

Venv

Core Bazel rules for defining Python targets.

Rules

• py_venv_binary
• py_venv_library
• py_venv_test
• py_venv_toolchain
• py_venv_zipapp

Functions

• py_venv_common.create_dep_info
• py_venv_common.create_py_info
• py_venv_common.create_runfiles_collection
• py_venv_common.create_venv_attrs
• py_venv_common.create_venv_config_info
• py_venv_common.create_venv_entrypoint
• py_venv_common.get_toolchain

py_venv_binary

load("@rules_venv//python/venv:defs.bzl", "py_venv_binary")

py_venv_binary(name, deps, srcs, data, env, imports, main)

A py_venv_binary is an executable Python program consisting of a collection of .py source files (possibly belonging to other py_library rules), a *.runfiles directory tree containing all the code and data needed by the program at run-time, and a stub script that starts up the program with the correct initial environment and data.

load("@rules_venv//python/venv:defs.bzl", "py_venv_binary")

py_venv_binary(
    name = "foo",
    srcs = ["foo.py"],
    data = [":transform"],  # a cc_binary which we invoke at run time
    deps = [
        ":bar",  # a py_library
    ],
)

ATTRIBUTES

py_venv_library

load("@rules_venv//python/venv:defs.bzl", "py_venv_library")

py_venv_library(name, deps, srcs, data, imports)

A library of Python code that can be depended upon.

ATTRIBUTES

py_venv_test

load("@rules_venv//python/venv:defs.bzl", "py_venv_test")

py_venv_test(name, deps, srcs, data, env, env_inherit, imports, main)

A py_venv_test rule compiles a test. A test is a binary wrapper around some test code.

ATTRIBUTES

py_venv_toolchain

load("@rules_venv//python/venv:defs.bzl", "py_venv_toolchain")

py_venv_toolchain(name, zipapp_shebang)

Declare a toolchain for rules_venv rules.

ATTRIBUTES

py_venv_zipapp

load("@rules_venv//python/venv:defs.bzl", "py_venv_zipapp")

py_venv_zipapp(name, args, binary, env, inherit_args, inherit_env, shebang)

A py_venv_zipapp is an executable Python zipapp which contains all of the dependencies and runfiles for a given executable.

load("@rules_venv//python/venv:defs.bzl", "py_venv_binary", "py_venv_zipapp")

py_venv_binary(
    name = "foo",
    srcs = ["foo.py"],
)

py_venv_zipapp(
    name = "foo_pyz",
    binary = ":foo",
)

ATTRIBUTES

py_venv_common.create_dep_info

load("@rules_venv//python/venv:defs.bzl", "py_venv_common")

py_venv_common.create_dep_info(*, ctx, deps)

Construct dependency info required for building PyInfo

PARAMETERS

RETURNS

struct: Dependency info.

py_venv_common.create_py_info

load("@rules_venv//python/venv:defs.bzl", "py_venv_common")

py_venv_common.create_py_info(*, ctx, imports, srcs, dep_info)

Construct a PyInfo provider

PARAMETERS

RETURNS

PyInfo: A PyInfo provider.

py_venv_common.create_runfiles_collection

load("@rules_venv//python/venv:defs.bzl", "py_venv_common")

py_venv_common.create_runfiles_collection(*, ctx, venv_toolchain, py_toolchain, runfiles,
                                          exclude_files, name, use_zip)

Generate a runfiles directory

This functionality exists due to the lack of native support for generating runfiles in an action. For details see: https://github.com/bazelbuild/bazel/issues/15486

PARAMETERS

RETURNS

Tuple[File, Runfiles]: The generated runfiles collection and associated runfiles.

py_venv_common.create_venv_attrs

load("@rules_venv//python/venv:defs.bzl", "py_venv_common")

py_venv_common.create_venv_attrs()

Create attributes for consumers of py_venv_common.

RETURNS

dict: Attributes unique to rules_venv rules.

py_venv_common.create_venv_config_info

load("@rules_venv//python/venv:defs.bzl", "py_venv_common")

py_venv_common.create_venv_config_info(*, label, name, imports)

Construct info used to create venvs.

PARAMETERS

RETURNS

struct: the data.

py_venv_common.create_venv_entrypoint

load("@rules_venv//python/venv:defs.bzl", "py_venv_common")

py_venv_common.create_venv_entrypoint(*, ctx, venv_toolchain, py_info, main, runfiles, py_toolchain,
                                      name, use_runfiles_in_entrypoint, force_runfiles)

Create an executable which constructs a python venv and subprocesses a given entrypoint.

PARAMETERS

RETURNS

Tuple[File, Runfiles]: The generated entrypoint and associated runfiles.

py_venv_common.get_toolchain

load("@rules_venv//python/venv:defs.bzl", "py_venv_common")

py_venv_common.get_toolchain(ctx, *, cfg)

Access the py_venv_toolchain from the current configuration.

Note that for cfg = "exec" toolchains, the rule must have the attributes from py_venv_common.create_venv_attrs.

PARAMETERS

RETURNS

py_venv_toolchain: The accessed toolchain.

Extensions

Additional rules which are built from core rules_venv to provide Bazel rules for popular python tools.

Black

Bazel rules for the Python formatter black.

Setup

First ensure rules_venv is setup by referring to rules_venv setup.

Next, the black rules work mostly off of toolchains which are used to provide the necessary python targets (aka black) for the process wrappers. Users will want to make sure they have a way to get the necessary python dependencies. Tools such as req-compile can provide these.

With the appropriate dependencies available, a py_black_toolchain will need to be configured:

load("@rules_venv//python/black:defs.bzl", "py_black_toolchain")

py_black_toolchain(
    name = "toolchain_impl",
    black = "@pip_deps//black",
    visibility = ["//visibility:public"]
)

toolchain(
    name = "toolchain",
    toolchain = ":toolchain_impl",
    toolchain_type = "@rules_venv//python/black:toolchain_type",
    visibility = ["//visibility:public"]
)

This toolchain then needs to be registered in the MODULE.bazel file.

register_toolchains("//tools/python/black:toolchain")

Usage

Python code can be formatted using the following command:

bazel run @rules_venv//python/black

In addition to this formatter, a check can be added to bazel build invocations using the py_black_aspect aspect. Simply add the following to a .bazelrc file to enable this check.

build --aspects=@rules_venv//python/black:py_black_aspect.bzl%py_black_aspect
build --output_groups=+py_black_checks

Rules

• py_black_test
• py_black_toolchain

Aspects

• py_black_aspect

py_black_test

load("@rules_venv//python/black:defs.bzl", "py_black_test")

py_black_test(name, config, target)

A rule for running black on a Python target.

ATTRIBUTES

py_black_toolchain

load("@rules_venv//python/black:defs.bzl", "py_black_toolchain")

py_black_toolchain(name, black)

A toolchain for the black formatter rules.

ATTRIBUTES

py_black_aspect

load("@rules_venv//python/black:defs.bzl", "py_black_aspect")

py_black_aspect()

An aspect for running black on targets with Python sources.

ASPECT ATTRIBUTES

ATTRIBUTES

Global Venv

Bazel rules for generating usable virtualenv from Bazel targets.

Usage

A venv can be created by invoking the following command

bazel run @rules_venv//python/global_venv

From here a venv will likely be available at .venv within your workspace that can be activated for improved IDE support.

Aspects

• py_global_venv_aspect

py_global_venv_aspect

load("@rules_venv//python/global_venv:defs.bzl", "py_global_venv_aspect")

py_global_venv_aspect()

An aspect for generating metadata required to include Python targets in a global venv.

ASPECT ATTRIBUTES

ATTRIBUTES

isort

Bazel rules for the Python formatter isort.

Setup

First ensure rules_venv is setup by referring to rules_venv setup.

Next, the rules_venv rules work mostly off of toolchains which are used to provide the necessary python targets (aka isort) for the process wrappers. Users will want to make sure they have a way to get the necessary python dependencies. Tools such as req-compile can provide these.

With the appropriate dependencies available, a py_isort_toolchain will need to be configured:

load("@rules_venv//python/isort:defs.bzl", "py_isort_toolchain")

py_isort_toolchain(
    name = "toolchain_impl",
    isort = "@pip_deps//isort",
    visibility = ["//visibility:public"]
)

toolchain(
    name = "toolchain",
    toolchain = ":toolchain_impl",
    toolchain_type = "@rules_venv//python/isort:toolchain_type",
    visibility = ["//visibility:public"]
)

This toolchain then needs to be registered in the MODULE.bazel file.

register_toolchains("//tools/python/isort:toolchain")

Usage

Python code can be formatted using the following command:

bazel run @rules_venv//python/isort

In addition to this formatter, a check can be added to bazel build invocations using the py_isort_aspect aspect. Simply add the following to a .bazelrc file to enable this check.

build --aspects=@rules_venv//python/isort:py_isort_aspect.bzl%py_isort_aspect
build --output_groups=+py_isort_checks

Sections and Ordering

Of all isort sections, FIRSTPARTY and THIRDPARTY follow unique rules to ensure ordering makes sense on a per-target basis.

First party

• Direct source dependencies passed to the srcs attribute.
• Repository relative packages.

Third party

• Other py_* targets (the deps attribute). Note that deps which use repo absolute import paths will be considered first party.

Tips

Isort is sensitive to the legacy_create_init attribute on python rules. For more correct and consistent behavior, this value should always be 0 or if the default of -1 is set, the --incompatible_default_to_explicit_init_py flag should be added to the workspace's .bazelrc file to ensure the behavior is disabled.

Rules

• py_isort_test
• py_isort_toolchain

Aspects

• py_isort_aspect

py_isort_test

load("@rules_venv//python/isort:defs.bzl", "py_isort_test")

py_isort_test(name, config, target)

A rule for running isort on a Python target.

ATTRIBUTES

py_isort_toolchain

load("@rules_venv//python/isort:defs.bzl", "py_isort_toolchain")

py_isort_toolchain(name, isort)

A toolchain for the isort formatter rules.

ATTRIBUTES

py_isort_aspect

load("@rules_venv//python/isort:defs.bzl", "py_isort_aspect")

py_isort_aspect()

An aspect for running isort on targets with Python sources.

ASPECT ATTRIBUTES

ATTRIBUTES

Module Extensions

Rules for building Module Extensions

Functions

• py_cc_extension

py_cc_extension

load("@rules_venv//python/module_extension:defs.bzl", "py_cc_extension")

py_cc_extension(*, name, srcs, conlyopts, copts, cxxopts, data, defines, deps, dynamic_deps,
                imports, includes, linkopts, local_defines, malloc, compilation_mode, abi, stripped,
                **kwargs)

Define a Python C extension module.

This target is consumed just as a py_library would be.

PARAMETERS

rules_mypy

Bazel rules for the Python linter mypy.

Setup

First ensure rules_venv is setup by referring to rules_venv setup.

Next, the mypy rules work mostly off of toolchains which are used to provide the necessary python targets (aka mypy) for the process wrappers. Users will want to make sure they have a way to get the necessary python dependencies. Tools such as req-compile can provide these.

With the appropriate dependencies available, a py_mypy_toolchain will need to be configured:

load("@rules_venv//python:defs.bzl", "py_library")
load("@rules_venv//python/mypy:defs.bzl", "py_mypy_toolchain")

py_library(
    name = "mypy_deps",
    deps = [
        "@pip_deps//mypy",

        # Types libraries can also be added here.
    ],
)

py_mypy_toolchain(
    name = "toolchain_impl",
    mypy = ":mypy_deps",
    visibility = ["//visibility:public"]
)

toolchain(
    name = "toolchain",
    toolchain = ":toolchain_impl",
    toolchain_type = "@rules_venv//python/mypy:toolchain_type",
    visibility = ["//visibility:public"]
)

This toolchain then needs to be registered in the MODULE.bazel file.

register_toolchains("//tools/python/mypy:toolchain")

From here, py_mypy_test and the py_mypy_aspect should now be usable. Both of these rules use a global flag to determine which mypy configuration file to use with in actions. The following snippet can be added to the .bazelrc file to chose the desired configuration file

build --@rules_mypy//python/mypy:config=//:.mypyrc.toml

Note that these files will need to be available via exports_files

Rules

• py_mypy_test
• py_mypy_toolchain

Aspects

• py_mypy_aspect

py_mypy_test

load("@rules_venv//python/mypy:defs.bzl", "py_mypy_test")

py_mypy_test(name, config, target)

A rule for running mypy on a Python target.

ATTRIBUTES

py_mypy_toolchain

load("@rules_venv//python/mypy:defs.bzl", "py_mypy_toolchain")

py_mypy_toolchain(name, mypy)

A toolchain for the mypy formatter rules.

ATTRIBUTES

py_mypy_aspect

load("@rules_venv//python/mypy:defs.bzl", "py_mypy_aspect")

py_mypy_aspect()

An aspect for running mypy on targets with Python sources.

ASPECT ATTRIBUTES

ATTRIBUTES

Pylint

Bazel rules for the Python linter Pylint.

Setup

First ensure rules_venv is setup by referring to rules_venv setup.

Next, the pylint rules work mostly off of toolchains which are used to provide the necessary python targets (aka pylint) for the process wrappers. Users will want to make sure they have a way to get the necessary python dependencies. Tools such as req-compile can provide these.

With the appropriate dependencies available, a py_pylint_toolchain will need to be configured:

load("@rules_venv//python/pylint:defs.bzl", "py_pylint_toolchain")

py_pylint_toolchain(
    name = "toolchain_impl",
    pylint = "@pip_deps//pylint",
    visibility = ["//visibility:public"]
)

toolchain(
    name = "toolchain",
    toolchain = ":toolchain_impl",
    toolchain_type = "@rules_venv//python/pylint:toolchain_type",
    visibility = ["//visibility:public"]
)

This toolchain then needs to be registered in the MODULE.bazel file.

register_toolchains("//tools/python/pylint:toolchain")

From here, py_pylint_test and the py_pylint_aspect should now be usable. Both of these rules use a global flag to determine which pylint configuration file to use with in actions. The following snippet can be added to the .bazelrc file to chose the desired configuration file

build --@rules_pylint//python/pylint:config=//:.pylintrc.toml

Note that these files will need to be available via exports_files

Rules

• py_pylint_test
• py_pylint_toolchain

Aspects

• py_pylint_aspect

py_pylint_test

load("@rules_venv//python/pylint:defs.bzl", "py_pylint_test")

py_pylint_test(name, config, target)

A rule for running pylint on a Python target.

ATTRIBUTES

py_pylint_toolchain

load("@rules_venv//python/pylint:defs.bzl", "py_pylint_toolchain")

py_pylint_toolchain(name, pylint)

A toolchain for the pylint formatter rules.

ATTRIBUTES

py_pylint_aspect

load("@rules_venv//python/pylint:defs.bzl", "py_pylint_aspect")

py_pylint_aspect()

An aspect for running pylint on targets with Python sources.

This aspect can be configured by adding the following snippet to a workspace's .bazelrc file:

build --aspects=@rules_venv//python/pylint:py_pylint_aspect.bzl%py_pylint_aspect
build --output_groups=+py_pylint_checks

ASPECT ATTRIBUTES

ATTRIBUTES

Pytest

Bazel rules for the python test framework Pytest.

Rules

• current_py_pytest_toolchain
• py_pytest_test
• py_pytest_toolchain

Functions

• py_pytest_test_suite

current_py_pytest_toolchain

load("@rules_venv//python/pytest:defs.bzl", "current_py_pytest_toolchain")

current_py_pytest_toolchain(name)

A rule for exposing the current registered py_pytest_toolchain.

ATTRIBUTES

py_pytest_test

load("@rules_venv//python/pytest:defs.bzl", "py_pytest_test")

py_pytest_test(name, deps, srcs, data, config, coverage_rc, env, env_inherit, numprocesses)

A rule which runs python tests using pytest as the py_test test runner.

This rule also supports a build setting for globally applying extra flags to test invocations. Users can add something similar to the following to their .bazelrc files:

build --@rules_venv//python/pytest:extra_args=--color=yes,-vv

The example above will add --colors=yes and -vv arguments to the end of the pytest invocation.

Tips:

• It's common for tests to have some utility code that does not live in a test source file. To account for this. A py_library can be created that contains only these sources which are then passed to py_pytest_test via deps.

load("@rules_venv//python:defs.bzl", "py_library")
load("@rules_venv//python/pytest:defs.bzl", "py_pytest_test")

py_library(
    name = "test_utils",
    srcs = [
        "tests/__init__.py",
        "tests/conftest.py",
    ],
    deps = ["@rules_venv//python/pytest:current_py_pytest_toolchain"],
    testonly = True,
)

py_pytest_test(
    name = "test",
    srcs = ["tests/example_test.py"],
    deps = [":test_utils"],
)

ATTRIBUTES

py_pytest_toolchain

load("@rules_venv//python/pytest:defs.bzl", "py_pytest_toolchain")

py_pytest_toolchain(name, pytest)

A toolchain for the pytest rules.

ATTRIBUTES

py_pytest_test_suite

load("@rules_venv//python/pytest:defs.bzl", "py_pytest_test_suite")

py_pytest_test_suite(name, tests, args, data, **kwargs)

Generates a test_suite which groups various test targets for a set of python sources.

Given an idiomatic python project structure:

BUILD.bazel
my_lib/
    __init__.py
    mod_a.py
    mod_b.py
    mod_c.py
requirements.in
requirements.txt
tests/
    __init__.py
    fixtures.py
    test_mod_a.py
    test_mod_b.py
    test_mod_c.py

This rule can be used to easily define test targets:

load("@rules_venv//python:defs.bzl", "py_library")
load("@rules_venv//python/pytest:defs.bzl", "py_pytest_test_suite")

py_library(
    name = "my_lib",
    srcs = glob(["my_lib/**/*.py"])
    imports = ["."],
)

py_pytest_test_suite(
    name = "my_lib_test_suite",
    # Source files containing test helpers should go here.
    # Note that the test sources are excluded. This avoids
    # a test to be updated without invalidating all other
    # targets.
    srcs = glob(
        include = ["tests/**/*.py"],
        exclude = ["tests/**/*_test.py"],
    ),
    # Any data files the tests may need would be passed here
    data = glob(["tests/**/*.json"]),
    # This field is used for dedicated test files.
    tests = glob(["tests/**/*_test.py"]),
    deps = [
        ":my_lib",
    ],
)

For each file passed to tests, a py_pytest_test target will be created. From the example above, the user should expect to see the following test targets:

//:my_lib_test_suite
//:tests/test_mod_a
//:tests/test_mod_b
//:tests/test_mod_c

Additional Notes:

• No file passed to tests should be passed found in the srcs or data attributes or tests will not be able to be individually cached.

PARAMETERS

Wheel

A framework for defining wheels from a tree of Bazel targets.

Why use this over what rules_python provides?

The rules here primarily wrap what rules_python provides but offers some improvements:

1. Requirements are collected from transitive dependencies and added as Require-Dist to the wheel.
2. Sources and data are automatically collected.

Setup

Building wheels requires no setup, however, to activate all features of the wheel rules, a toolchain must be registered.

First some external python requirements will be required. There is guidance on how to configure this in projects like rules_req_compile. Once there is a means to fetch external dependencies within your repository, a toolchain can be defined in a BUILD.bazel file.

Example //:BUILD.bazel:

load("@rules_venv//python/wheel:defs.bzl", "py_wheel_toolchain")

py_wheel_toolchain(
    name = "py_wheel_toolchain_impl",
    # Definable using bzlmod modules like: https://github.com/periareon/req-compile
    twine = "@pip_deps//twine",
    visibility = ["//visibility:public"],
)

toolchain(
    name = "py_wheel_toolchain",
    toolchain = ":py_wheel_toolchain_impl",
    toolchain_type = "@rules_venv//python/wheel:toolchain_type",
    visibility = ["//visibility:public"],
)

Once the toolchain is defined, it should be registered in the MODULE.bazel file

Example //:MODULE.bazel:

register_toolchains(
    "//:py_wheel_toolchain",
)

This will ensure all features of the wheel rules are available and usable.

Rules

• py_wheel_publisher
• py_wheel_toolchain

Functions

• package_tag
• py_wheel_library

py_wheel_publisher

load("@rules_venv//python/wheel:defs.bzl", "py_wheel_publisher")

py_wheel_publisher(name, repository_url, wheel)

A rule for publishing wheels to pypi registries.

The rule uses twine to python registries. Users should refer to the documentation there for any configuration flags (such as auth) needed to deploy to the desired location.

ATTRIBUTES

py_wheel_toolchain

load("@rules_venv//python/wheel:defs.bzl", "py_wheel_toolchain")

py_wheel_toolchain(name, twine)

A toolchain for powering the py_wheel rules.

load("@rules_venv//python/wheel:defs.bzl", "py_wheel_toolchain")

py_wheel_toolchain(
    name = "py_wheel_toolchain_impl",
    # Definable using bzlmod modules like: https://github.com/periareon/req-compile
    twine = "@pip_deps//twine",
    visibility = ["//visibility:public"],
)

toolchain(
    name = "py_wheel_toolchain",
    toolchain = ":py_wheel_toolchain_impl",
    toolchain_type = "@rules_venv//python/wheel:toolchain_type",
    visibility = ["//visibility:public"],
)

ATTRIBUTES

package_tag

load("@rules_venv//python/wheel:defs.bzl", "package_tag")

package_tag(name)

Generate a tag used to associate the target with a python package.

This tag should only be applied to other py_* targets.

Example:

load("@rules_venv//python:defs.bzl", "py_library")
load("@rules_venv//python/wheel:defs.bzl", "package_tag")

py_library(
    name = "my_target",
    tags = [package_tag("my_python_package")],
    # ...
    # ...
    # ...
)

PARAMETERS

RETURNS

str: The tag.

py_wheel_library

load("@rules_venv//python/wheel:defs.bzl", "py_wheel_library")

py_wheel_library(name, srcs, deps, data, abi, author, author_email, classifiers, constraints_file,
                 data_files, description_content_type, description_file, extra_distinfo_files,
                 homepage, license, platform, project_urls, python_requires, python_tag,
                 strip_path_prefixes, summary, version, distribution, repository_url, **kwargs)

Define a py_library with an associated wheel.

This rule will traverse dependencies (deps) to collect all data and dependencies that belong to the current package. Any dependency tagged with the package_tag whose name matches this targets name will be considered a submodule and included in the package.

Example:

load("@rules_venv//python:defs.bzl", "py_library")
load("@rules_venv//python/wheel:defs.bzl", "package_tag", "py_wheel_library")

py_library(
    name = "submodule",
    srcs = ["submodule.py"],
    tags = [
        # Note this name matches the name of the `py_wheel_library` target
        # and as a result will be included as a submodule within the wheel.
        package_tag("my_py_package")
    ],
)

py_wheel_library(
    name = "my_py_package",
    deps = [":submodule"],
)

Targets created by this library:

PARAMETERS