Understanding pytest-xdist

Table of contents

A. Preface

Over at $work we have been using alot of Pytest and its associated libraries to help us with testing alot of of our application logic.

I have not had to fully deep dive into pytest and its associated plugins for the testing needed at $work. But recently, we also started exploring the usage of Pytest-xdist at $work, which allows tests to be run in parallel. With that, came a slew of flaky tests that required debugging.

In this post - I aim to consolidated my learnings about Pytest and Pytest-xdist throughout my time debugging their workings. Specifically, I wish to note down the parts that caused me confusion, things that I learnt and hopefully these notes can serve to aid your understanding with this library.

B. An Introduction to Pytest-xdist

The pytest-xdist plugin extends pytest with new test execution modes, with the most common usage being distributing tests across multiple CPUs to speed up test execution:

As the docs suggest, the library helps to distribute and run your suite of tests concurrently.

The library offers a few cool modes, but for the sake of brevity, I will focus more on the core usage modes meant for distribution which are related to:

load - which sends pending tests to any worker available in no guaranteed order.
loadscope - The default behaviour is that tests are grouped by module and test functions are grouped by class.

What this does is that all of the tests in a group run in the same process/worker. This is usually helpful when we have expensive module-level or class level fixtures that can be shared across all of these tests.
loadgroup - Tests are grouped by the xdist_group mark. Groups are distributed to available workers as whole units. This guarantees that all tests with same xdist_group name run in the same worker.

C. A Unit of (Py)Test - Node

In pytest, a unit of test, be it a single function or a collection of tests, is known as a node, and their references or node ids their scopes are usually delimited by :: identifiers.

Most notably, it identifies the particular subset of tests that is to be run.

E.g we can do something like:

pytest test_module.py::TestClass::test_method

to execute a particular test_method nested under TestClass in the test_module.py file.

Essentially the tests identified or proverbially collected in Pytest are also identified using such a specification format.

From pytest:

Node IDs are of the form module.py::class::method or module.py::function. Node IDs control which tests are collected, so module.py::class will select all test methods on the class. Nodes are also created for each parameter of a parametrized fixture or test, so selecting a parametrized test must include the parameter value, e.g. module.py::function[param].

See Source

In pytest, the collector collects all the node ids and iteratively builds a tree out of all of these tests to perform testing later on.

D. Process of distributing tests in Pytest-xdist

In pytest-xdist, the process generally goes like this:

We specify the number of workers(nodes) we want to perform the parallel testing.
pytest-xdist spins up a new distributed test session or DSession. This DSession spins up a Node Manager which then spins up the required number of workers or nodes. The Node Manager is also synonymous to the Controller.
The Controller spins up the required number of workers by connecting to different (sub)processes running, each running a Python interpreter via execnet using gateways.
- This can be done locally or to remote server.
- Each of these processes are also known as a worker or node.

Pytest Workers

Each worker performs a full test collection of all the tests available and sends these tests or otherwise their node ids back to the Controller.
The Controller constructs a list of node_ids based off the node_idsreceived and check if all workers collected the same tests. </br>
- This is an important step as the implementation relies on a centralized in memory list of node_ids (i.e test ids). Based off this list, the Controller will then again assign the tests, based on scope or grouping to the different workers using the tests’ indexed positions in this list.
- Because the work assignment relies on this centralized list, if any of the workers collects any tests differently be it in order or number, the test will not be able to execute and result in an error like Failed due to gw0.
Each worker is essentially a pytest runner, which reimplements pytest_runtestloop and pytest_runtest_protocol, but adjusted in xdist to manage the controller-workernode relationship (see docs). What is interesting here is learning about pytest-xdist extending pytest’s existing hooks with its own custom implementation. We exploreo more of this in the next section.

But first here is a diagram that illustrates the process of from test collection to test execution and reporting.

Pytest Workers

A more complete process of how it works can also be found here

E.1. Pytest-xdist: Scoping, grouping and running of tests in the same worker.

As mentioned in section C, we can distribute the tests by their scopes either through the original implementation loadscope as well a derived implementationloadgroup.
The tests distributed across the various workers are also guaranteed to be run only once.
In loadscope, how it works is that for each test(or node id as discussed in section C) flows through the _split_scope function.

For convenience this is the documentation:

Determine the scope (grouping) of a nodeid.
    There are usually 3 cases for a nodeid::
    example/loadsuite/test/test_beta.py::test_beta0
    example/loadsuite/test/test_delta.py::Delta1::test_delta0
    example/loadsuite/epsilon/__init__.py::epsilon.epsilon

    #. Function in a test module.
    #. Method of a class in a test module.
    #. Doctest in a function in a package.

    This function will group tests with the scope determined by splitting
    the first ``::`` from the right. That is, classes will be grouped in a
    single work unit, and functions from a test module will be grouped by
    their module. In the above example, scopes will be::

    example/loadsuite/test/test_beta.py
    example/loadsuite/test/test_delta.py::Delta1
    example/loadsuite/epsilon/__init__.py

To rephrase, what it does is that _split_scope determines the scope via the first :: separator of the node ids. I.e the scope is created from a file/module -> class to function level.

By doing this, xdist achieves grouping of tests together by their scopes. A scope of tests will be guaranteed to run in a single worker/process.
What this also means is that we can overwrite this function to determine custom way of ensuring certain tests will always run in the same worker.

For example, we can take a look at the the loadgroup implementation. Specifically it redefines the _split_scope implementation to identify based on the @ operator within the node ids.

Such a node id might look like this tests/test.py::TestGroup1::test_get_now_only@group1 given the example below.

# investigation/tests/test.py

import pytest

@pytest.mark.xdist_group(name="group1")
class TestGroup1:
    def test_val_1(self):
        val = 1
        assert val == 1


@pytest.mark.xdist_group(name="group2")
class TestGroup2:
    def test_val_2(self):
        val = 2
        assert val == 2

Test Results

(venv3117xdist) ➜  investigation git:(master) ✗ pytest tests/test.py -n 2 --dist loadgroup -vvv
=========================================================================================================================================== test session starts ============================================================================================================================================
platform darwin -- Python 3.11.0rc2, pytest-8.1.0.dev176+g7690a0ddf.d20240210, pluggy-1.4.0 -- /Users/jitcorn/.pyenv/versions/3.11.0rc2/bin/python3.11
cachedir: .pytest_cache
Using --randomly-seed=1707796509
rootdir: /Users/jitcorn/pytest-xdist
configfile: tox.ini
plugins: timeout-2.2.0, randomly-3.1.0, asyncio-0.20.3, tornado-0.8.1, cov-4.0.0, xdist-3.5.0, forked-1.3.0, anyio-4.2.0, mock-3.5.1
asyncio: mode=Mode.STRICT
2 workers [2 items]
scheduling tests via LoadGroupScheduling

tests/test.py::TestGroup2::test_val_2@group2
tests/test.py::TestGroup1::test_val_1@group1
[gw1] [ 50%] PASSED tests/test.py::TestGroup2::test_val_2@group2
[gw0] [100%] PASSED tests/test.py::TestGroup1::test_val_1@group1

From this contrived test example, we demonstrate the usage of loadgroup and can make a few observations.

We first run the tests in the verbose mode -vvv and we can observe that there are 2 workers gw0 and gw1 working on the tests. This is because we specified 2 workers in -n 2 (we can also use -n auto to determine based on the number of CPUs your server has). gw0 and gw1 are also knokwn as worker_id in xdist.
We can see that the the tests are also grouped and distributed evevnly to each of the workers.
The group names group1 and group2 are appended as @<group_num> at the end of each of the node ids.
- Also note while the existing example on the documentation shows the grouping by individual test functions, you can also group tests by their classes, instead of individual test functions. This means you don’t have to put @pytest.mark.xdist_group over every single function if they are already grouped by classes, which is very convenient!

E.2 Customising the scoping capabilities of pytest-xdist

As seen from the previous section, the scoping mechanism can be customised by specifying a custom algortihm in the _split_scope function.

Suppose you wish to devise your own custom split_scope methodology specific to your repo, you can modify conftest.py in this manner to modify the scoping functionality.

import os
import logging
from xdist.scheduler.loadscope import LoadScopeScheduling

SINGLE_PROCESS_TESTFILES_TO_SCOPE_MAPPING = {
    "investigation/tests/test.py": "investigation-tests-test-scope",
}

class TestsScheduler(LoadScopeScheduling):
    def fetch_highest_test_scope(self, node_id):
        """
        Currently used for scoping out tests that cannot be run concurrently, i.e in a single worker only
        """
        first_delimiter = node_id.find("::")
        return node_id[:first_delimiter]

    def _split_scope(self, node_id):
        """
        Override pytest-xdist's _split_scope method to have scoped/grouped tests run by the same worker.
        Each node_id entails a single scope/worker.
        See: https://github.com/pytest-dev/pytest-xdist/blob/ef344e9b55a0365c1aabb738ce3db97324ed553e/src/xdist/scheduler/loadscope.py#L268-L290
        """
        highest_test_scope = self.fetch_highest_test_scope(node_id)
        if highest_test_scope in SINGLE_PROCESS_TESTFILES_TO_SCOPE_MAPPING:
            return SINGLE_PROCESS_TESTFILES_TO_SCOPE_MAPPING.get(highest_test_scope)

        return node_id

def pytest_xdist_make_scheduler(config, log):
    return TestsScheduler(config,log)

This example shows that in a list of all of the tests collected, we ensure that all the tests inside test.py are guaranteed to run within the same worker.
This could be useful in cases where you might experience race conditions amongst different tests that are read or writing to the same table or index in a database for example.
The way how the xdist groups tests by scope is by the heavy usage of ordered dicts, specially here. The exact debugging shall be left to the reader as an exercise :p.

Important! Be it grouping by @ or defining your own _split_scope function, at the point of writing of pytest version 3.5.0, while a grouped/scoped tests are guaranteed to run in the same worker, there is no mechanism that prevents 2 groups of scoped tests from running in the same worker as it comes in via a round-robin fashion*.

I.e if we have 2 workers with 3 grouped tests:

[gw0] - test_group_1
[gw1] - test_group_2
[gw0] - test_group_3

test_group_1 and test_group_3 are going to run in the same gw0 worker. This may pose a problem if there is no properly tear down performed after tear_group_1. As you can imagine, if you wish to separate test_group_1 and test_group_3 from separate workers (maybe they involve writing to the same DB), then pytest-xdist does not manage this for you at the moment. A better way to circumvent this problem is to ensure that the tests are properly setup and teardown in both test_group_1 and test_group_3.

F. Pytest: What are hooks?

As this author puts it, Pytest hooks are gateway points that allow users to inject logic at specific stages of the test execution process to modify or extend the behaviour of tests based on test events.
In the case of our pytest-xdist workers, it mostly reimplements or makes use of these 2 hooks:
- pytest_runtestloop is the default hook implementation that performs the pytest_runtest_protocol. What it means that it was built to be extendable via pytest hooks.
- pytest_runtest_protocol is basically the series of steps done for each test in a runtestloop.
- Each stage usually performs 4 actions, call, create-report, log-report and handle-exception, respectively:
  - call - pytest_runtest_setup - essentiallly collecting values such as the fixtures required for the test item
  - create-report - pytest_runtest_makereport - creates a TestReport object meant to report the outcomes of a test
  - log-report - pytest_runtest_logreport - processes the created TestReport
  - handle-exception - pytest_exception_interact- interactive handling when an exception is thrown
- Hence in a runtest_protocol, these 4 actions are done in each phase:
  1. Setup phase - This involves processes such as:
  2. Call phase - which performs the actual test function call
  3. Teardown phase - tearing down all of the tests
This is indeed a similar idea to the 3 As of testing, together with how we normally have setups and teardowns in our day to day test writing!

G. Pytest-xdist: Why are the workers collecting the tests instead of the controller?

For more details please refer to the How it works section, which also explains the entire process in greater details.

H. Debugging the pytest-xdist and pytest libraries

A quick way of testing out any library installed on a repo you working on is by:
1. Installing the library locally
2. Installing the library in your target repo by using the editable mode.

Step 1: Specifically in the context of pytest-xdist we need to first clone it and then install all of the dependencies locally and install its dependencies. I am currently using python 3.11:

git clone git@github.com:pytest-dev/pytest-xdist.git
tox -e py311

This should set up all the necessary dependencies in your local pytest-xdist setup.

Step 2: In our repo that is using pytest-xdist, we can symlink it to this local copy of the package by install it in --editable mode. This developers to implement and test changes iteratively before releasing a package.

pip install -e <local_directory_of_package>

With this set up, the pytest-xdist library will be symlinked from your repo to the library locally. You can then place debuggers like import pdb; pdb.set_trace() inside to debug the flow of the programme!

I. Takeaways

Here are my takeaways after spending alot of time trying to understand pytest-xdist internals:

There was some confusion when deconstructing the notion of a node: in pytest it refers to a unit of test while in pytest-xdist, it refers to a particular worker.
A in memory ordered list of node_ids is the underlying structure used to reference and schedule work units to each worker node. The creation of this list is based on scoping and heavy usage of OrderedDict() in the actual implementation
The @pytest.mark.xdist_group(name="<name>") decoractor can also be applied on classes for more convenient grouping
While scoping/grouping of tests ensures they run in the same worker, it does not prevent 2 groups of tests running in the same worker. Proper test set ups and tear downs are recommended if race conditions are a concern.
Pytest-xdist makes use of existing pytest-hooks to help extend pytest functionalities to make test running in a distributed setting possible.
A similar arrange, act, assert, complete with setup and teardown structure is also present in the inner workings of pytest!