Full pytest documentation¶

Installation and Getting Started¶

Pythons: Python 3.5, 3.6, 3.7, PyPy3

Platforms: Linux and Windows

PyPI package name: pytest

Documentation as PDF: download latest

pytest is a framework that makes building simple and scalable tests easy. Tests are expressive and readable—no boilerplate code required. Get started in minutes with a small unit test or complex functional test for your application or library.

Install `pytest`¶

Run the following command in your command line:

pip install -U pytest

Check that you installed the correct version:

$ pytest --version
This is pytest version 5.x.y, imported from $PYTHON_PREFIX/lib/python3.6/site-packages/pytest/__init__.py

Create your first test¶

Create a simple test function with just four lines of code:

# content of test_sample.py
def func(x):
    return x + 1


def test_answer():
    assert func(3) == 5

That’s it. You can now execute the test function:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_sample.py F                                                     [100%]

================================= FAILURES =================================
_______________________________ test_answer ________________________________

    def test_answer():
>       assert func(3) == 5
E       assert 4 == 5
E        +  where 4 = func(3)

test_sample.py:6: AssertionError
============================ 1 failed in 0.12s =============================

This test returns a failure report because func(3) does not return 5.

Note

You can use the assert statement to verify test expectations. pytest’s Advanced assertion introspection will intelligently report intermediate values of the assert expression so you can avoid the many names of JUnit legacy methods.

Run multiple tests¶

pytest will run all files of the form test_*.py or *_test.py in the current directory and its subdirectories. More generally, it follows standard test discovery rules.

Assert that a certain exception is raised¶

Use the raises helper to assert that some code raises an exception:

# content of test_sysexit.py
import pytest


def f():
    raise SystemExit(1)


def test_mytest():
    with pytest.raises(SystemExit):
        f()

Execute the test function with “quiet” reporting mode:

$ pytest -q test_sysexit.py
.                                                                    [100%]
1 passed in 0.12s

Group multiple tests in a class¶

Once you develop multiple tests, you may want to group them into a class. pytest makes it easy to create a class containing more than one test:

# content of test_class.py
class TestClass:
    def test_one(self):
        x = "this"
        assert "h" in x

    def test_two(self):
        x = "hello"
        assert hasattr(x, "check")

pytest discovers all tests following its Conventions for Python test discovery, so it finds both test_ prefixed functions. There is no need to subclass anything, but make sure to prefix your class with Test otherwise the class will be skipped. We can simply run the module by passing its filename:

$ pytest -q test_class.py
.F                                                                   [100%]
================================= FAILURES =================================
____________________________ TestClass.test_two ____________________________

self = <test_class.TestClass object at 0xdeadbeef>

    def test_two(self):
        x = "hello"
>       assert hasattr(x, "check")
E       AssertionError: assert False
E        +  where False = hasattr('hello', 'check')

test_class.py:8: AssertionError
1 failed, 1 passed in 0.12s

The first test passed and the second failed. You can easily see the intermediate values in the assertion to help you understand the reason for the failure.

Request a unique temporary directory for functional tests¶

pytest provides Builtin fixtures/function arguments to request arbitrary resources, like a unique temporary directory:

# content of test_tmpdir.py
def test_needsfiles(tmpdir):
    print(tmpdir)
    assert 0

List the name tmpdir in the test function signature and pytest will lookup and call a fixture factory to create the resource before performing the test function call. Before the test runs, pytest creates a unique-per-test-invocation temporary directory:

$ pytest -q test_tmpdir.py
F                                                                    [100%]
================================= FAILURES =================================
_____________________________ test_needsfiles ______________________________

tmpdir = local('PYTEST_TMPDIR/test_needsfiles0')

    def test_needsfiles(tmpdir):
        print(tmpdir)
>       assert 0
E       assert 0

test_tmpdir.py:3: AssertionError
--------------------------- Captured stdout call ---------------------------
PYTEST_TMPDIR/test_needsfiles0
1 failed in 0.12s

More info on tmpdir handling is available at Temporary directories and files.

Find out what kind of builtin pytest fixtures exist with the command:

pytest --fixtures   # shows builtin and custom fixtures

Note that this command omits fixtures with leading _ unless the -v option is added.

Continue reading¶

Check out additional pytest resources to help you customize tests for your unique workflow:

“Calling pytest through python -m pytest” for command line invocation examples
“Using pytest with an existing test suite” for working with pre-existing tests
“Marking test functions with attributes” for information on the pytest.mark mechanism
“pytest fixtures: explicit, modular, scalable” for providing a functional baseline to your tests
“Writing plugins” for managing and writing plugins
“Good Integration Practices” for virtualenv and test layouts

Usage and Invocations¶

Calling pytest through `python -m pytest`¶

You can invoke testing through the Python interpreter from the command line:

python -m pytest [...]

This is almost equivalent to invoking the command line script pytest [...] directly, except that calling via python will also add the current directory to sys.path.

Possible exit codes¶

Running pytest can result in six different exit codes:

Exit code 0:	All tests were collected and passed successfully
Exit code 1:	Tests were collected and run but some of the tests failed
Exit code 2:	Test execution was interrupted by the user
Exit code 3:	Internal error happened while executing tests
Exit code 4:	pytest command line usage error
Exit code 5:	No tests were collected

They are represented by the _pytest.config.ExitCode enum. The exit codes being a part of the public API can be imported and accessed directly using:

from pytest import ExitCode

Note

If you would like to customize the exit code in some scenarios, specially when no tests are collected, consider using the pytest-custom_exit_code plugin.

Getting help on version, option names, environment variables¶

pytest --version   # shows where pytest was imported from
pytest --fixtures  # show available builtin function arguments
pytest -h | --help # show help on command line and config file options

Stopping after the first (or N) failures¶

To stop the testing process after the first (N) failures:

pytest -x           # stop after first failure
pytest --maxfail=2  # stop after two failures

Specifying tests / selecting tests¶

Pytest supports several ways to run and select tests from the command-line.

Run tests in a module

pytest test_mod.py

Run tests in a directory

pytest testing/

Run tests by keyword expressions

pytest -k "MyClass and not method"

This will run tests which contain names that match the given string expression (case-insensitive), which can include Python operators that use filenames, class names and function names as variables. The example above will run TestMyClass.test_something but not TestMyClass.test_method_simple.

Run tests by node ids

Each collected test is assigned a unique nodeid which consist of the module filename followed by specifiers like class names, function names and parameters from parametrization, separated by :: characters.

To run a specific test within a module:

pytest test_mod.py::test_func

Another example specifying a test method in the command line:

pytest test_mod.py::TestClass::test_method

Run tests by marker expressions

pytest -m slow

Will run all tests which are decorated with the @pytest.mark.slow decorator.

For more information see marks.

Run tests from packages

pytest --pyargs pkg.testing

This will import pkg.testing and use its filesystem location to find and run tests from.

Modifying Python traceback printing¶

Examples for modifying traceback printing:

pytest --showlocals # show local variables in tracebacks
pytest -l           # show local variables (shortcut)

pytest --tb=auto    # (default) 'long' tracebacks for the first and last
                     # entry, but 'short' style for the other entries
pytest --tb=long    # exhaustive, informative traceback formatting
pytest --tb=short   # shorter traceback format
pytest --tb=line    # only one line per failure
pytest --tb=native  # Python standard library formatting
pytest --tb=no      # no traceback at all

The --full-trace causes very long traces to be printed on error (longer than --tb=long). It also ensures that a stack trace is printed on KeyboardInterrupt (Ctrl+C). This is very useful if the tests are taking too long and you interrupt them with Ctrl+C to find out where the tests are hanging. By default no output will be shown (because KeyboardInterrupt is caught by pytest). By using this option you make sure a trace is shown.

Detailed summary report¶

The -r flag can be used to display a “short test summary info” at the end of the test session, making it easy in large test suites to get a clear picture of all failures, skips, xfails, etc.

It defaults to fE to list failures and errors.

Example:

# content of test_example.py
import pytest

@pytest.fixture
def error_fixture():
    assert 0

def test_ok():
    print("ok")

def test_fail():
    assert 0

def test_error(error_fixture):
    pass

def test_skip():
    pytest.skip("skipping this test")

def test_xfail():
    pytest.xfail("xfailing this test")

@pytest.mark.xfail(reason="always xfail")
def test_xpass():
    pass

$ pytest -ra
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
========================= short test summary info ==========================
SKIPPED [1] $REGENDOC_TMPDIR/test_example.py:22: skipping this test
XFAIL test_example.py::test_xfail
  reason: xfailing this test
XPASS test_example.py::test_xpass always xfail
ERROR test_example.py::test_error - assert 0
FAILED test_example.py::test_fail - assert 0
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

The -r options accepts a number of characters after it, with a used above meaning “all except passes”.

Here is the full list of available characters that can be used:

f - failed

E - error

s - skipped

x - xfailed

X - xpassed

p - passed

P - passed with output

Special characters for (de)selection of groups:

a - all except pP

A - all

N - none, this can be used to display nothing (since fE is the default)

More than one character can be used, so for example to only see failed and skipped tests, you can execute:

$ pytest -rfs
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
========================= short test summary info ==========================
FAILED test_example.py::test_fail - assert 0
SKIPPED [1] $REGENDOC_TMPDIR/test_example.py:22: skipping this test
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

Using p lists the passing tests, whilst P adds an extra section “PASSES” with those tests that passed but had captured output:

$ pytest -rpP
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 6 items

test_example.py .FEsxX                                               [100%]

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError
================================== PASSES ==================================
_________________________________ test_ok __________________________________
--------------------------- Captured stdout call ---------------------------
ok
========================= short test summary info ==========================
PASSED test_example.py::test_ok
== 1 failed, 1 passed, 1 skipped, 1 xfailed, 1 xpassed, 1 error in 0.12s ===

Dropping to PDB (Python Debugger) on failures¶

Python comes with a builtin Python debugger called PDB. pytest allows one to drop into the PDB prompt via a command line option:

pytest --pdb

This will invoke the Python debugger on every failure (or KeyboardInterrupt). Often you might only want to do this for the first failing test to understand a certain failure situation:

pytest -x --pdb   # drop to PDB on first failure, then end test session
pytest --pdb --maxfail=3  # drop to PDB for first three failures

Note that on any failure the exception information is stored on sys.last_value, sys.last_type and sys.last_traceback. In interactive use, this allows one to drop into postmortem debugging with any debug tool. One can also manually access the exception information, for example:

>>> import sys
>>> sys.last_traceback.tb_lineno
42
>>> sys.last_value
AssertionError('assert result == "ok"',)

Dropping to PDB (Python Debugger) at the start of a test¶

pytest allows one to drop into the PDB prompt immediately at the start of each test via a command line option:

pytest --trace

This will invoke the Python debugger at the start of every test.

Setting breakpoints¶

To set a breakpoint in your code use the native Python import pdb;pdb.set_trace() call in your code and pytest automatically disables its output capture for that test:

Output capture in other tests is not affected.
Any prior test output that has already been captured and will be processed as such.
Output capture gets resumed when ending the debugger session (via the continue command).

Using the builtin breakpoint function¶

Python 3.7 introduces a builtin breakpoint() function. Pytest supports the use of breakpoint() with the following behaviours:

When breakpoint() is called and PYTHONBREAKPOINT is set to the default value, pytest will use the custom internal PDB trace UI instead of the system default Pdb.

When tests are complete, the system will default back to the system Pdb trace UI.

With --pdb passed to pytest, the custom internal Pdb trace UI is used with both breakpoint() and failed tests/unhandled exceptions.

--pdbcls can be used to specify a custom debugger class.

Profiling test execution duration¶

To get a list of the slowest 10 test durations:

pytest --durations=10

By default, pytest will not show test durations that are too small (<0.01s) unless -vv is passed on the command-line.

Fault Handler¶

New in version 5.0.

The faulthandler standard module can be used to dump Python tracebacks on a segfault or after a timeout.

The module is automatically enabled for pytest runs, unless the -p no:faulthandler is given on the command-line.

Also the faulthandler_timeout=X configuration option can be used to dump the traceback of all threads if a test takes longer than X seconds to finish (not available on Windows).

Note

This functionality has been integrated from the external pytest-faulthandler plugin, with two small differences:

To disable it, use -p no:faulthandler instead of --no-faulthandler: the former can be used with any plugin, so it saves one option.
The --faulthandler-timeout command-line option has become the faulthandler_timeout configuration option. It can still be configured from the command-line using -o faulthandler_timeout=X.

Creating JUnitXML format files¶

To create result files which can be read by Jenkins or other Continuous integration servers, use this invocation:

pytest --junitxml=path

to create an XML file at path.

To set the name of the root test suite xml item, you can configure the junit_suite_name option in your config file:

[pytest]
junit_suite_name = my_suite

New in version 4.0.

JUnit XML specification seems to indicate that "time" attribute should report total test execution times, including setup and teardown (1, 2). It is the default pytest behavior. To report just call durations instead, configure the junit_duration_report option like this:

[pytest]
junit_duration_report = call

record_property¶

If you want to log additional information for a test, you can use the record_property fixture:

def test_function(record_property):
    record_property("example_key", 1)
    assert True

This will add an extra property example_key="1" to the generated testcase tag:

<testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009">
  <properties>
    <property name="example_key" value="1" />
  </properties>
</testcase>

Alternatively, you can integrate this functionality with custom markers:

# content of conftest.py


def pytest_collection_modifyitems(session, config, items):
    for item in items:
        for marker in item.iter_markers(name="test_id"):
            test_id = marker.args[0]
            item.user_properties.append(("test_id", test_id))

And in your tests:

# content of test_function.py
import pytest


@pytest.mark.test_id(1501)
def test_function():
    assert True

Will result in:

<testcase classname="test_function" file="test_function.py" line="0" name="test_function" time="0.0009">
  <properties>
    <property name="test_id" value="1501" />
  </properties>
</testcase>

Warning

Please note that using this feature will break schema verifications for the latest JUnitXML schema. This might be a problem when used with some CI servers.

record_xml_attribute¶

To add an additional xml attribute to a testcase element, you can use record_xml_attribute fixture. This can also be used to override existing values:

def test_function(record_xml_attribute):
    record_xml_attribute("assertions", "REQ-1234")
    record_xml_attribute("classname", "custom_classname")
    print("hello world")
    assert True

Unlike record_property, this will not add a new child element. Instead, this will add an attribute assertions="REQ-1234" inside the generated testcase tag and override the default classname with "classname=custom_classname":

<testcase classname="custom_classname" file="test_function.py" line="0" name="test_function" time="0.003" assertions="REQ-1234">
    <system-out>
        hello world
    </system-out>
</testcase>

Warning

record_xml_attribute is an experimental feature, and its interface might be replaced by something more powerful and general in future versions. The functionality per-se will be kept, however.

Using this over record_xml_property can help when using ci tools to parse the xml report. However, some parsers are quite strict about the elements and attributes that are allowed. Many tools use an xsd schema (like the example below) to validate incoming xml. Make sure you are using attribute names that are allowed by your parser.

Below is the Scheme used by Jenkins to validate the XML report:

<xs:element name="testcase">
    <xs:complexType>
        <xs:sequence>
            <xs:element ref="skipped" minOccurs="0" maxOccurs="1"/>
            <xs:element ref="error" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="failure" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="system-out" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element ref="system-err" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute name="name" type="xs:string" use="required"/>
        <xs:attribute name="assertions" type="xs:string" use="optional"/>
        <xs:attribute name="time" type="xs:string" use="optional"/>
        <xs:attribute name="classname" type="xs:string" use="optional"/>
        <xs:attribute name="status" type="xs:string" use="optional"/>
    </xs:complexType>
</xs:element>

Warning

Please note that using this feature will break schema verifications for the latest JUnitXML schema. This might be a problem when used with some CI servers.

record_testsuite_property¶

New in version 4.5.

If you want to add a properties node at the test-suite level, which may contains properties that are relevant to all tests, you can use the record_testsuite_property session-scoped fixture:

The record_testsuite_property session-scoped fixture can be used to add properties relevant to all tests.

import pytest


@pytest.fixture(scope="session", autouse=True)
def log_global_env_facts(record_testsuite_property):
    record_testsuite_property("ARCH", "PPC")
    record_testsuite_property("STORAGE_TYPE", "CEPH")


class TestMe:
    def test_foo(self):
        assert True

The fixture is a callable which receives name and value of a <property> tag added at the test-suite level of the generated xml:

<testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="0.006">
  <properties>
    <property name="ARCH" value="PPC"/>
    <property name="STORAGE_TYPE" value="CEPH"/>
  </properties>
  <testcase classname="test_me.TestMe" file="test_me.py" line="16" name="test_foo" time="0.000243663787842"/>
</testsuite>

name must be a string, value will be converted to a string and properly xml-escaped.

The generated XML is compatible with the latest xunit standard, contrary to record_property and record_xml_attribute.

Creating resultlog format files¶

To create plain-text machine-readable result files you can issue:

pytest --resultlog=path

and look at the content at the path location. Such files are used e.g. by the PyPy-test web page to show test results over several revisions.

Warning

This option is rarely used and is scheduled for removal in pytest 6.0.

If you use this option, consider using the new pytest-reportlog plugin instead.

See the deprecation docs for more information.

Sending test report to online pastebin service¶

Creating a URL for each test failure:

pytest --pastebin=failed

This will submit test run information to a remote Paste service and provide a URL for each failure. You may select tests as usual or add for example -x if you only want to send one particular failure.

Creating a URL for a whole test session log:

pytest --pastebin=all

Currently only pasting to the http://bpaste.net service is implemented.

Changed in version 5.2.

If creating the URL fails for any reason, a warning is generated instead of failing the entire test suite.

Early loading plugins¶

You can early-load plugins (internal and external) explicitly in the command-line with the -p option:

pytest -p mypluginmodule

The option receives a name parameter, which can be:

A full module dotted name, for example myproject.plugins. This dotted name must be importable.
The entry-point name of a plugin. This is the name passed to setuptools when the plugin is registered. For example to early-load the pytest-cov plugin you can use:
```
pytest -p pytest_cov
```

Disabling plugins¶

To disable loading specific plugins at invocation time, use the -p option together with the prefix no:.

Example: to disable loading the plugin doctest, which is responsible for executing doctest tests from text files, invoke pytest like this:

pytest -p no:doctest

Calling pytest from Python code¶

You can invoke pytest from Python code directly:

pytest.main()

this acts as if you would call “pytest” from the command line. It will not raise SystemExit but return the exitcode instead. You can pass in options and arguments:

pytest.main(["-x", "mytestdir"])

You can specify additional plugins to pytest.main:

# content of myinvoke.py
import pytest


class MyPlugin:
    def pytest_sessionfinish(self):
        print("*** test run reporting finishing")


pytest.main(["-qq"], plugins=[MyPlugin()])

Running it will show that MyPlugin was added and its hook was invoked:

$ python myinvoke.py
.FEsxX.                                                              [100%]*** test run reporting finishing

================================== ERRORS ==================================
_______________________ ERROR at setup of test_error _______________________

    @pytest.fixture
    def error_fixture():
>       assert 0
E       assert 0

test_example.py:6: AssertionError
================================= FAILURES =================================
________________________________ test_fail _________________________________

    def test_fail():
>       assert 0
E       assert 0

test_example.py:14: AssertionError

Note

Calling pytest.main() will result in importing your tests and any modules that they import. Due to the caching mechanism of python’s import system, making subsequent calls to pytest.main() from the same process will not reflect changes to those files between the calls. For this reason, making multiple calls to pytest.main() from the same process (in order to re-run tests, for example) is not recommended.

Using pytest with an existing test suite¶

Pytest can be used with most existing test suites, but its behavior differs from other test runners such as nose or Python’s default unittest framework.

Before using this section you will want to install pytest.

Running an existing test suite with pytest¶

Say you want to contribute to an existing repository somewhere. After pulling the code into your development space using some flavor of version control and (optionally) setting up a virtualenv you will want to run:

cd <repository>
pip install -e .  # Environment dependent alternatives include
                  # 'python setup.py develop' and 'conda develop'

in your project root. This will set up a symlink to your code in site-packages, allowing you to edit your code while your tests run against it as if it were installed.

Setting up your project in development mode lets you avoid having to reinstall every time you want to run your tests, and is less brittle than mucking about with sys.path to point your tests at local code.

Also consider using tox.

The writing and reporting of assertions in tests¶

Asserting with the `assert` statement¶

pytest allows you to use the standard python assert for verifying expectations and values in Python tests. For example, you can write the following:

# content of test_assert1.py
def f():
    return 3


def test_function():
    assert f() == 4

to assert that your function returns a certain value. If this assertion fails you will see the return value of the function call:

$ pytest test_assert1.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_assert1.py F                                                    [100%]

================================= FAILURES =================================
______________________________ test_function _______________________________

    def test_function():
>       assert f() == 4
E       assert 3 == 4
E        +  where 3 = f()

test_assert1.py:6: AssertionError
============================ 1 failed in 0.12s =============================

pytest has support for showing the values of the most common subexpressions including calls, attributes, comparisons, and binary and unary operators. (See Demo of Python failure reports with pytest). This allows you to use the idiomatic python constructs without boilerplate code while not losing introspection information.

However, if you specify a message with the assertion like this:

assert a % 2 == 0, "value was odd, should be even"

then no assertion introspection takes places at all and the message will be simply shown in the traceback.

See Assertion introspection details for more information on assertion introspection.

Assertions about expected exceptions¶

In order to write assertions about raised exceptions, you can use pytest.raises as a context manager like this:

import pytest


def test_zero_division():
    with pytest.raises(ZeroDivisionError):
        1 / 0

and if you need to have access to the actual exception info you may use:

def test_recursion_depth():
    with pytest.raises(RuntimeError) as excinfo:

        def f():
            f()

        f()
    assert "maximum recursion" in str(excinfo.value)

excinfo is a ExceptionInfo instance, which is a wrapper around the actual exception raised. The main attributes of interest are .type, .value and .traceback.

You can pass a match keyword parameter to the context-manager to test that a regular expression matches on the string representation of an exception (similar to the TestCase.assertRaisesRegexp method from unittest):

import pytest


def myfunc():
    raise ValueError("Exception 123 raised")


def test_match():
    with pytest.raises(ValueError, match=r".* 123 .*"):
        myfunc()

The regexp parameter of the match method is matched with the re.search function, so in the above example match='123' would have worked as well.

There’s an alternate form of the pytest.raises function where you pass a function that will be executed with the given *args and **kwargs and assert that the given exception is raised:

pytest.raises(ExpectedException, func, *args, **kwargs)

The reporter will provide you with helpful output in case of failures such as no exception or wrong exception.

Note that it is also possible to specify a “raises” argument to pytest.mark.xfail, which checks that the test is failing in a more specific way than just having any exception raised:

@pytest.mark.xfail(raises=IndexError)
def test_f():
    f()

Using pytest.raises is likely to be better for cases where you are testing exceptions your own code is deliberately raising, whereas using @pytest.mark.xfail with a check function is probably better for something like documenting unfixed bugs (where the test describes what “should” happen) or bugs in dependencies.

Assertions about expected warnings¶

You can check that code raises a particular warning using pytest.warns.

Making use of context-sensitive comparisons¶

pytest has rich support for providing context-sensitive information when it encounters comparisons. For example:

# content of test_assert2.py


def test_set_comparison():
    set1 = set("1308")
    set2 = set("8035")
    assert set1 == set2

if you run this module:

$ pytest test_assert2.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_assert2.py F                                                    [100%]

================================= FAILURES =================================
___________________________ test_set_comparison ____________________________

    def test_set_comparison():
        set1 = set("1308")
        set2 = set("8035")
>       assert set1 == set2
E       AssertionError: assert {'0', '1', '3', '8'} == {'0', '3', '5', '8'}
E         Extra items in the left set:
E         '1'
E         Extra items in the right set:
E         '5'
E         Use -v to get the full diff

test_assert2.py:6: AssertionError
============================ 1 failed in 0.12s =============================

Special comparisons are done for a number of cases:

comparing long strings: a context diff is shown
comparing long sequences: first failing indices
comparing dicts: different entries

See the reporting demo for many more examples.

Defining your own explanation for failed assertions¶

It is possible to add your own detailed explanations by implementing the pytest_assertrepr_compare hook.

pytest_assertrepr_compare(config, op, left, right)[source]

return explanation for comparisons in failing assert expressions.

Return None for no custom explanation, otherwise return a list of strings. The strings will be joined by newlines but any newlines in a string will be escaped. Note that all but the first line will be indented slightly, the intention is for the first line to be a summary.

Parameters:	config (_pytest.config.Config) – pytest config object

As an example consider adding the following hook in a conftest.py file which provides an alternative explanation for Foo objects:

# content of conftest.py
from test_foocompare import Foo


def pytest_assertrepr_compare(op, left, right):
    if isinstance(left, Foo) and isinstance(right, Foo) and op == "==":
        return [
            "Comparing Foo instances:",
            "   vals: {} != {}".format(left.val, right.val),
        ]

now, given this test module:

# content of test_foocompare.py
class Foo:
    def __init__(self, val):
        self.val = val

    def __eq__(self, other):
        return self.val == other.val


def test_compare():
    f1 = Foo(1)
    f2 = Foo(2)
    assert f1 == f2

you can run the test module and get the custom output defined in the conftest file:

$ pytest -q test_foocompare.py
F                                                                    [100%]
================================= FAILURES =================================
_______________________________ test_compare _______________________________

    def test_compare():
        f1 = Foo(1)
        f2 = Foo(2)
>       assert f1 == f2
E       assert Comparing Foo instances:
E            vals: 1 != 2

test_foocompare.py:12: AssertionError
1 failed in 0.12s

Assertion introspection details¶

Reporting details about a failing assertion is achieved by rewriting assert statements before they are run. Rewritten assert statements put introspection information into the assertion failure message. pytest only rewrites test modules directly discovered by its test collection process, so asserts in supporting modules which are not themselves test modules will not be rewritten.

You can manually enable assertion rewriting for an imported module by calling register_assert_rewrite before you import it (a good place to do that is in your root conftest.py).

For further information, Benjamin Peterson wrote up Behind the scenes of pytest’s new assertion rewriting.

Assertion rewriting caches files on disk¶

pytest will write back the rewritten modules to disk for caching. You can disable this behavior (for example to avoid leaving stale .pyc files around in projects that move files around a lot) by adding this to the top of your conftest.py file:

import sys

sys.dont_write_bytecode = True

Note that you still get the benefits of assertion introspection, the only change is that the .pyc files won’t be cached on disk.

Additionally, rewriting will silently skip caching if it cannot write new .pyc files, i.e. in a read-only filesystem or a zipfile.

Disabling assert rewriting¶

pytest rewrites test modules on import by using an import hook to write new pyc files. Most of the time this works transparently. However, if you are working with the import machinery yourself, the import hook may interfere.

If this is the case you have two options:

Disable rewriting for a specific module by adding the string PYTEST_DONT_REWRITE to its docstring.
Disable rewriting for all modules by using --assert=plain.

Add assert rewriting as an alternate introspection technique.

Introduce the --assert option. Deprecate --no-assert and --nomagic.

Removes the --no-assert and --nomagic options. Removes the --assert=reinterp option.

pytest fixtures: explicit, modular, scalable¶

Software test fixtures initialize test functions. They provide a fixed baseline so that tests execute reliably and produce consistent, repeatable, results. Initialization may setup services, state, or other operating environments. These are accessed by test functions through arguments; for each fixture used by a test function there is typically a parameter (named after the fixture) in the test function’s definition.

pytest fixtures offer dramatic improvements over the classic xUnit style of setup/teardown functions:

fixtures have explicit names and are activated by declaring their use from test functions, modules, classes or whole projects.
fixtures are implemented in a modular manner, as each fixture name triggers a fixture function which can itself use other fixtures.
fixture management scales from simple unit to complex functional testing, allowing to parametrize fixtures and tests according to configuration and component options, or to re-use fixtures across function, class, module or whole test session scopes.

In addition, pytest continues to support classic xunit-style setup. You can mix both styles, moving incrementally from classic to new style, as you prefer. You can also start out from existing unittest.TestCase style or nose based projects.

Fixtures are defined using the @pytest.fixture decorator, described below. Pytest has useful built-in fixtures, listed here for reference:

capfd

Capture, as text, output to file descriptors 1 and 2.

capfdbinary

Capture, as bytes, output to file descriptors 1 and 2.

caplog

Control logging and access log entries.

capsys

Capture, as text, output to sys.stdout and sys.stderr.

capsysbinary

Capture, as bytes, output to sys.stdout and sys.stderr.

cache

Store and retrieve values across pytest runs.

doctest_namespace

Provide a dict injected into the docstests namespace.

monkeypatch

Temporarily modify classes, functions, dictionaries, os.environ, and other objects.

pytestconfig

Access to configuration values, pluginmanager and plugin hooks.

record_property

Add extra properties to the test.

record_testsuite_property

Add extra properties to the test suite.

recwarn

Record warnings emitted by test functions.

request

Provide information on the executing test function.

testdir

Provide a temporary test directory to aid in running, and testing, pytest plugins.

tmp_path

Provide a pathlib.Path object to a temporary directory which is unique to each test function.

tmp_path_factory

Make session-scoped temporary directories and return pathlib.Path objects.

tmpdir

Provide a py.path.local object to a temporary directory which is unique to each test function; replaced by tmp_path.

tmpdir_factory

Make session-scoped temporary directories and return py.path.local objects; replaced by tmp_path_factory.

Fixtures as Function arguments¶

Test functions can receive fixture objects by naming them as an input argument. For each argument name, a fixture function with that name provides the fixture object. Fixture functions are registered by marking them with @pytest.fixture. Let’s look at a simple self-contained test module containing a fixture and a test function using it:

# content of ./test_smtpsimple.py
import pytest


@pytest.fixture
def smtp_connection():
    import smtplib

    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)


def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250
    assert 0  # for demo purposes

Here, the test_ehlo needs the smtp_connection fixture value. pytest will discover and call the @pytest.fixture marked smtp_connection fixture function. Running the test looks like this:

$ pytest test_smtpsimple.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_smtpsimple.py F                                                 [100%]

================================= FAILURES =================================
________________________________ test_ehlo _________________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_smtpsimple.py:14: AssertionError
============================ 1 failed in 0.12s =============================

In the failure traceback we see that the test function was called with a smtp_connection argument, the smtplib.SMTP() instance created by the fixture function. The test function fails on our deliberate assert 0. Here is the exact protocol used by pytest to call the test function this way:

pytest finds the test_ehlo because of the test_ prefix. The test function needs a function argument named smtp_connection. A matching fixture function is discovered by looking for a fixture-marked function named smtp_connection.
smtp_connection() is called to create an instance.
test_ehlo(<smtp_connection instance>) is called and fails in the last line of the test function.

Note that if you misspell a function argument or want to use one that isn’t available, you’ll see an error with a list of available function arguments.

Note

You can always issue:

pytest --fixtures test_simplefactory.py

to see available fixtures (fixtures with leading _ are only shown if you add the -v option).

Fixtures: a prime example of dependency injection¶

Fixtures allow test functions to easily receive and work against specific pre-initialized application objects without having to care about import/setup/cleanup details. It’s a prime example of dependency injection where fixture functions take the role of the injector and test functions are the consumers of fixture objects.

Scope: sharing a fixture instance across tests in a class, module or session¶

Fixtures requiring network access depend on connectivity and are usually time-expensive to create. Extending the previous example, we can add a scope="module" parameter to the @pytest.fixture invocation to cause the decorated smtp_connection fixture function to only be invoked once per test module (the default is to invoke once per test function). Multiple test functions in a test module will thus each receive the same smtp_connection fixture instance, thus saving time. Possible values for scope are: function, class, module, package or session.

The next example puts the fixture function into a separate conftest.py file so that tests from multiple test modules in the directory can access the fixture function:

# content of conftest.py
import pytest
import smtplib


@pytest.fixture(scope="module")
def smtp_connection():
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

The name of the fixture again is smtp_connection and you can access its result by listing the name smtp_connection as an input parameter in any test or fixture function (in or below the directory where conftest.py is located):

# content of test_module.py


def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250
    assert b"smtp.gmail.com" in msg
    assert 0  # for demo purposes


def test_noop(smtp_connection):
    response, msg = smtp_connection.noop()
    assert response == 250
    assert 0  # for demo purposes

We deliberately insert failing assert 0 statements in order to inspect what is going on and can now run the tests:

$ pytest test_module.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py FF                                                    [100%]

================================= FAILURES =================================
________________________________ test_ehlo _________________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
        assert b"smtp.gmail.com" in msg
>       assert 0  # for demo purposes
E       assert 0

test_module.py:7: AssertionError
________________________________ test_noop _________________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_noop(smtp_connection):
        response, msg = smtp_connection.noop()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_module.py:13: AssertionError
============================ 2 failed in 0.12s =============================

You see the two assert 0 failing and more importantly you can also see that the same (module-scoped) smtp_connection object was passed into the two test functions because pytest shows the incoming argument values in the traceback. As a result, the two test functions using smtp_connection run as quick as a single one because they reuse the same instance.

If you decide that you rather want to have a session-scoped smtp_connection instance, you can simply declare it:

@pytest.fixture(scope="session")
def smtp_connection():
    # the returned fixture value will be shared for
    # all tests needing it
    ...

Finally, the class scope will invoke the fixture once per test class.

Note

Pytest will only cache one instance of a fixture at a time. This means that when using a parametrized fixture, pytest may invoke a fixture more than once in the given scope.

`package` scope (experimental)¶

In pytest 3.7 the package scope has been introduced. Package-scoped fixtures are finalized when the last test of a package finishes.

Warning

This functionality is considered experimental and may be removed in future versions if hidden corner-cases or serious problems with this functionality are discovered after it gets more usage in the wild.

Use this new feature sparingly and please make sure to report any issues you find.

Dynamic scope¶

New in version 5.2.

In some cases, you might want to change the scope of the fixture without changing the code. To do that, pass a callable to scope. The callable must return a string with a valid scope and will be executed only once - during the fixture definition. It will be called with two keyword arguments - fixture_name as a string and config with a configuration object.

This can be especially useful when dealing with fixtures that need time for setup, like spawning a docker container. You can use the command-line argument to control the scope of the spawned containers for different environments. See the example below.

def determine_scope(fixture_name, config):
    if config.getoption("--keep-containers", None):
        return "session"
    return "function"


@pytest.fixture(scope=determine_scope)
def docker_container():
    yield spawn_container()

Order: Higher-scoped fixtures are instantiated first¶

Within a function request for features, fixture of higher-scopes (such as session) are instantiated first than lower-scoped fixtures (such as function or class). The relative order of fixtures of same scope follows the declared order in the test function and honours dependencies between fixtures. Autouse fixtures will be instantiated before explicitly used fixtures.

Consider the code below:

import pytest

# fixtures documentation order example
order = []


@pytest.fixture(scope="session")
def s1():
    order.append("s1")


@pytest.fixture(scope="module")
def m1():
    order.append("m1")


@pytest.fixture
def f1(f3):
    order.append("f1")


@pytest.fixture
def f3():
    order.append("f3")


@pytest.fixture(autouse=True)
def a1():
    order.append("a1")


@pytest.fixture
def f2():
    order.append("f2")


def test_order(f1, m1, f2, s1):
    assert order == ["s1", "m1", "a1", "f3", "f1", "f2"]

The fixtures requested by test_order will be instantiated in the following order:

s1: is the highest-scoped fixture (session).
m1: is the second highest-scoped fixture (module).
a1: is a function-scoped autouse fixture: it will be instantiated before other fixtures within the same scope.
f3: is a function-scoped fixture, required by f1: it needs to be instantiated at this point
f1: is the first function-scoped fixture in test_order parameter list.
f2: is the last function-scoped fixture in test_order parameter list.

Fixture finalization / executing teardown code¶

pytest supports execution of fixture specific finalization code when the fixture goes out of scope. By using a yield statement instead of return, all the code after the yield statement serves as the teardown code:

# content of conftest.py

import smtplib
import pytest


@pytest.fixture(scope="module")
def smtp_connection():
    smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
    yield smtp_connection  # provide the fixture value
    print("teardown smtp")
    smtp_connection.close()

The print and smtp.close() statements will execute when the last test in the module has finished execution, regardless of the exception status of the tests.

Let’s execute it:

$ pytest -s -q --tb=no
FFteardown smtp

2 failed in 0.12s

We see that the smtp_connection instance is finalized after the two tests finished execution. Note that if we decorated our fixture function with scope='function' then fixture setup and cleanup would occur around each single test. In either case the test module itself does not need to change or know about these details of fixture setup.

Note that we can also seamlessly use the yield syntax with with statements:

# content of test_yield2.py

import smtplib
import pytest


@pytest.fixture(scope="module")
def smtp_connection():
    with smtplib.SMTP("smtp.gmail.com", 587, timeout=5) as smtp_connection:
        yield smtp_connection  # provide the fixture value

The smtp_connection connection will be closed after the test finished execution because the smtp_connection object automatically closes when the with statement ends.

Using the contextlib.ExitStack context manager finalizers will always be called regardless if the fixture setup code raises an exception. This is handy to properly close all resources created by a fixture even if one of them fails to be created/acquired:

# content of test_yield3.py

import contextlib

import pytest


@contextlib.contextmanager
def connect(port):
    ...  # create connection
    yield
    ...  # close connection


@pytest.fixture
def equipments():
    with contextlib.ExitStack() as stack:
        yield [stack.enter_context(connect(port)) for port in ("C1", "C3", "C28")]

In the example above, if "C28" fails with an exception, "C1" and "C3" will still be properly closed.

Note that if an exception happens during the setup code (before the yield keyword), the teardown code (after the yield) will not be called.

An alternative option for executing teardown code is to make use of the addfinalizer method of the request-context object to register finalization functions.

Here’s the smtp_connection fixture changed to use addfinalizer for cleanup:

# content of conftest.py
import smtplib
import pytest


@pytest.fixture(scope="module")
def smtp_connection(request):
    smtp_connection = smtplib.SMTP("smtp.gmail.com", 587, timeout=5)

    def fin():
        print("teardown smtp_connection")
        smtp_connection.close()

    request.addfinalizer(fin)
    return smtp_connection  # provide the fixture value

Here’s the equipments fixture changed to use addfinalizer for cleanup:

# content of test_yield3.py

import contextlib
import functools

import pytest


@contextlib.contextmanager
def connect(port):
    ...  # create connection
    yield
    ...  # close connection


@pytest.fixture
def equipments(request):
    r = []
    for port in ("C1", "C3", "C28"):
        cm = connect(port)
        equip = cm.__enter__()
        request.addfinalizer(functools.partial(cm.__exit__, None, None, None))
        r.append(equip)
    return r

Both yield and addfinalizer methods work similarly by calling their code after the test ends. Of course, if an exception happens before the finalize function is registered then it will not be executed.

Fixtures can introspect the requesting test context¶

Fixture functions can accept the request object to introspect the “requesting” test function, class or module context. Further extending the previous smtp_connection fixture example, let’s read an optional server URL from the test module which uses our fixture:

# content of conftest.py
import pytest
import smtplib


@pytest.fixture(scope="module")
def smtp_connection(request):
    server = getattr(request.module, "smtpserver", "smtp.gmail.com")
    smtp_connection = smtplib.SMTP(server, 587, timeout=5)
    yield smtp_connection
    print("finalizing {} ({})".format(smtp_connection, server))
    smtp_connection.close()

We use the request.module attribute to optionally obtain an smtpserver attribute from the test module. If we just execute again, nothing much has changed:

$ pytest -s -q --tb=no
FFfinalizing <smtplib.SMTP object at 0xdeadbeef> (smtp.gmail.com)

2 failed in 0.12s

Let’s quickly create another test module that actually sets the server URL in its module namespace:

# content of test_anothersmtp.py

smtpserver = "mail.python.org"  # will be read by smtp fixture

def test_showhelo(smtp_connection):
    assert 0, smtp_connection.helo()

Running it:

$ pytest -qq --tb=short test_anothersmtp.py
F                                                                    [100%]
================================= FAILURES =================================
______________________________ test_showhelo _______________________________
test_anothersmtp.py:6: in test_showhelo
    assert 0, smtp_connection.helo()
E   AssertionError: (250, b'mail.python.org')
E   assert 0
------------------------- Captured stdout teardown -------------------------
finalizing <smtplib.SMTP object at 0xdeadbeef> (mail.python.org)

voila! The smtp_connection fixture function picked up our mail server name from the module namespace.

Factories as fixtures¶

The “factory as fixture” pattern can help in situations where the result of a fixture is needed multiple times in a single test. Instead of returning data directly, the fixture instead returns a function which generates the data. This function can then be called multiple times in the test.

Factories can have parameters as needed:

@pytest.fixture
def make_customer_record():
    def _make_customer_record(name):
        return {"name": name, "orders": []}

    return _make_customer_record


def test_customer_records(make_customer_record):
    customer_1 = make_customer_record("Lisa")
    customer_2 = make_customer_record("Mike")
    customer_3 = make_customer_record("Meredith")

If the data created by the factory requires managing, the fixture can take care of that:

@pytest.fixture
def make_customer_record():

    created_records = []

    def _make_customer_record(name):
        record = models.Customer(name=name, orders=[])
        created_records.append(record)
        return record

    yield _make_customer_record

    for record in created_records:
        record.destroy()


def test_customer_records(make_customer_record):
    customer_1 = make_customer_record("Lisa")
    customer_2 = make_customer_record("Mike")
    customer_3 = make_customer_record("Meredith")

Parametrizing fixtures¶

Fixture functions can be parametrized in which case they will be called multiple times, each time executing the set of dependent tests, i. e. the tests that depend on this fixture. Test functions usually do not need to be aware of their re-running. Fixture parametrization helps to write exhaustive functional tests for components which themselves can be configured in multiple ways.

Extending the previous example, we can flag the fixture to create two smtp_connection fixture instances which will cause all tests using the fixture to run twice. The fixture function gets access to each parameter through the special request object:

# content of conftest.py
import pytest
import smtplib


@pytest.fixture(scope="module", params=["smtp.gmail.com", "mail.python.org"])
def smtp_connection(request):
    smtp_connection = smtplib.SMTP(request.param, 587, timeout=5)
    yield smtp_connection
    print("finalizing {}".format(smtp_connection))
    smtp_connection.close()

The main change is the declaration of params with @pytest.fixture, a list of values for each of which the fixture function will execute and can access a value via request.param. No test function code needs to change. So let’s just do another run:

$ pytest -q test_module.py
FFFF                                                                 [100%]
================================= FAILURES =================================
________________________ test_ehlo[smtp.gmail.com] _________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
        assert b"smtp.gmail.com" in msg
>       assert 0  # for demo purposes
E       assert 0

test_module.py:7: AssertionError
________________________ test_noop[smtp.gmail.com] _________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_noop(smtp_connection):
        response, msg = smtp_connection.noop()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_module.py:13: AssertionError
________________________ test_ehlo[mail.python.org] ________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_ehlo(smtp_connection):
        response, msg = smtp_connection.ehlo()
        assert response == 250
>       assert b"smtp.gmail.com" in msg
E       AssertionError: assert b'smtp.gmail.com' in b'mail.python.org\nPIPELINING\nSIZE 51200000\nETRN\nSTARTTLS\nAUTH DIGEST-MD5 NTLM CRAM-MD5\nENHANCEDSTATUSCODES\n8BITMIME\nDSN\nSMTPUTF8\nCHUNKING'

test_module.py:6: AssertionError
-------------------------- Captured stdout setup ---------------------------
finalizing <smtplib.SMTP object at 0xdeadbeef>
________________________ test_noop[mail.python.org] ________________________

smtp_connection = <smtplib.SMTP object at 0xdeadbeef>

    def test_noop(smtp_connection):
        response, msg = smtp_connection.noop()
        assert response == 250
>       assert 0  # for demo purposes
E       assert 0

test_module.py:13: AssertionError
------------------------- Captured stdout teardown -------------------------
finalizing <smtplib.SMTP object at 0xdeadbeef>
4 failed in 0.12s

We see that our two test functions each ran twice, against the different smtp_connection instances. Note also, that with the mail.python.org connection the second test fails in test_ehlo because a different server string is expected than what arrived.

pytest will build a string that is the test ID for each fixture value in a parametrized fixture, e.g. test_ehlo[smtp.gmail.com] and test_ehlo[mail.python.org] in the above examples. These IDs can be used with -k to select specific cases to run, and they will also identify the specific case when one is failing. Running pytest with --collect-only will show the generated IDs.

Numbers, strings, booleans and None will have their usual string representation used in the test ID. For other objects, pytest will make a string based on the argument name. It is possible to customise the string used in a test ID for a certain fixture value by using the ids keyword argument:

# content of test_ids.py
import pytest


@pytest.fixture(params=[0, 1], ids=["spam", "ham"])
def a(request):
    return request.param


def test_a(a):
    pass


def idfn(fixture_value):
    if fixture_value == 0:
        return "eggs"
    else:
        return None


@pytest.fixture(params=[0, 1], ids=idfn)
def b(request):
    return request.param


def test_b(b):
    pass

The above shows how ids can be either a list of strings to use or a function which will be called with the fixture value and then has to return a string to use. In the latter case if the function return None then pytest’s auto-generated ID will be used.

Running the above tests results in the following test IDs being used:

$ pytest --collect-only
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 10 items
<Module test_anothersmtp.py>
  <Function test_showhelo[smtp.gmail.com]>
  <Function test_showhelo[mail.python.org]>
<Module test_ids.py>
  <Function test_a[spam]>
  <Function test_a[ham]>
  <Function test_b[eggs]>
  <Function test_b[1]>
<Module test_module.py>
  <Function test_ehlo[smtp.gmail.com]>
  <Function test_noop[smtp.gmail.com]>
  <Function test_ehlo[mail.python.org]>
  <Function test_noop[mail.python.org]>

========================== no tests ran in 0.12s ===========================

Using marks with parametrized fixtures¶

pytest.param() can be used to apply marks in values sets of parametrized fixtures in the same way that they can be used with @pytest.mark.parametrize.

Example:

# content of test_fixture_marks.py
import pytest


@pytest.fixture(params=[0, 1, pytest.param(2, marks=pytest.mark.skip)])
def data_set(request):
    return request.param


def test_data(data_set):
    pass

Running this test will skip the invocation of data_set with value 2:

$ pytest test_fixture_marks.py -v
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 3 items

test_fixture_marks.py::test_data[0] PASSED                           [ 33%]
test_fixture_marks.py::test_data[1] PASSED                           [ 66%]
test_fixture_marks.py::test_data[2] SKIPPED                          [100%]

======================= 2 passed, 1 skipped in 0.12s =======================

Modularity: using fixtures from a fixture function¶

In addition to using fixtures in test functions, fixture functions can use other fixtures themselves. This contributes to a modular design of your fixtures and allows re-use of framework-specific fixtures across many projects. As a simple example, we can extend the previous example and instantiate an object app where we stick the already defined smtp_connection resource into it:

# content of test_appsetup.py

import pytest

class App:
    def __init__(self, smtp_connection):
        self.smtp_connection = smtp_connection

@pytest.fixture(scope="module")
def app(smtp_connection):
    return App(smtp_connection)

def test_smtp_connection_exists(app):
    assert app.smtp_connection

Here we declare an app fixture which receives the previously defined smtp_connection fixture and instantiates an App object with it. Let’s run it:

$ pytest -v test_appsetup.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 2 items

test_appsetup.py::test_smtp_connection_exists[smtp.gmail.com] PASSED [ 50%]
test_appsetup.py::test_smtp_connection_exists[mail.python.org] PASSED [100%]

============================ 2 passed in 0.12s =============================

Due to the parametrization of smtp_connection, the test will run twice with two different App instances and respective smtp servers. There is no need for the app fixture to be aware of the smtp_connection parametrization because pytest will fully analyse the fixture dependency graph.

Note that the app fixture has a scope of module and uses a module-scoped smtp_connection fixture. The example would still work if smtp_connection was cached on a session scope: it is fine for fixtures to use “broader” scoped fixtures but not the other way round: A session-scoped fixture could not use a module-scoped one in a meaningful way.

Automatic grouping of tests by fixture instances¶

pytest minimizes the number of active fixtures during test runs. If you have a parametrized fixture, then all the tests using it will first execute with one instance and then finalizers are called before the next fixture instance is created. Among other things, this eases testing of applications which create and use global state.

The following example uses two parametrized fixtures, one of which is scoped on a per-module basis, and all the functions perform print calls to show the setup/teardown flow:

# content of test_module.py
import pytest


@pytest.fixture(scope="module", params=["mod1", "mod2"])
def modarg(request):
    param = request.param
    print("  SETUP modarg", param)
    yield param
    print("  TEARDOWN modarg", param)


@pytest.fixture(scope="function", params=[1, 2])
def otherarg(request):
    param = request.param
    print("  SETUP otherarg", param)
    yield param
    print("  TEARDOWN otherarg", param)


def test_0(otherarg):
    print("  RUN test0 with otherarg", otherarg)


def test_1(modarg):
    print("  RUN test1 with modarg", modarg)


def test_2(otherarg, modarg):
    print("  RUN test2 with otherarg {} and modarg {}".format(otherarg, modarg))

Let’s run the tests in verbose mode and with looking at the print-output:

$ pytest -v -s test_module.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y -- $PYTHON_PREFIX/bin/python
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collecting ... collected 8 items

test_module.py::test_0[1]   SETUP otherarg 1
  RUN test0 with otherarg 1
PASSED  TEARDOWN otherarg 1

test_module.py::test_0[2]   SETUP otherarg 2
  RUN test0 with otherarg 2
PASSED  TEARDOWN otherarg 2

test_module.py::test_1[mod1]   SETUP modarg mod1
  RUN test1 with modarg mod1
PASSED
test_module.py::test_2[mod1-1]   SETUP otherarg 1
  RUN test2 with otherarg 1 and modarg mod1
PASSED  TEARDOWN otherarg 1

test_module.py::test_2[mod1-2]   SETUP otherarg 2
  RUN test2 with otherarg 2 and modarg mod1
PASSED  TEARDOWN otherarg 2

test_module.py::test_1[mod2]   TEARDOWN modarg mod1
  SETUP modarg mod2
  RUN test1 with modarg mod2
PASSED
test_module.py::test_2[mod2-1]   SETUP otherarg 1
  RUN test2 with otherarg 1 and modarg mod2
PASSED  TEARDOWN otherarg 1

test_module.py::test_2[mod2-2]   SETUP otherarg 2
  RUN test2 with otherarg 2 and modarg mod2
PASSED  TEARDOWN otherarg 2
  TEARDOWN modarg mod2


============================ 8 passed in 0.12s =============================

You can see that the parametrized module-scoped modarg resource caused an ordering of test execution that lead to the fewest possible “active” resources. The finalizer for the mod1 parametrized resource was executed before the mod2 resource was setup.

In particular notice that test_0 is completely independent and finishes first. Then test_1 is executed with mod1, then test_2 with mod1, then test_1 with mod2 and finally test_2 with mod2.

The otherarg parametrized resource (having function scope) was set up before and teared down after every test that used it.

Using fixtures from classes, modules or projects¶

Sometimes test functions do not directly need access to a fixture object. For example, tests may require to operate with an empty directory as the current working directory but otherwise do not care for the concrete directory. Here is how you can use the standard tempfile and pytest fixtures to achieve it. We separate the creation of the fixture into a conftest.py file:

# content of conftest.py

import os
import shutil
import tempfile

import pytest


@pytest.fixture
def cleandir():
    old_cwd = os.getcwd()
    newpath = tempfile.mkdtemp()
    os.chdir(newpath)
    yield
    os.chdir(old_cwd)
    shutil.rmtree(newpath)

and declare its use in a test module via a usefixtures marker:

# content of test_setenv.py
import os
import pytest


@pytest.mark.usefixtures("cleandir")
class TestDirectoryInit:
    def test_cwd_starts_empty(self):
        assert os.listdir(os.getcwd()) == []
        with open("myfile", "w") as f:
            f.write("hello")

    def test_cwd_again_starts_empty(self):
        assert os.listdir(os.getcwd()) == []

Due to the usefixtures marker, the cleandir fixture will be required for the execution of each test method, just as if you specified a “cleandir” function argument to each of them. Let’s run it to verify our fixture is activated and the tests pass:

$ pytest -q
..                                                                   [100%]
2 passed in 0.12s

You can specify multiple fixtures like this:

@pytest.mark.usefixtures("cleandir", "anotherfixture")
def test():
    ...

and you may specify fixture usage at the test module level, using a generic feature of the mark mechanism:

pytestmark = pytest.mark.usefixtures("cleandir")

Note that the assigned variable must be called pytestmark, assigning e.g. foomark will not activate the fixtures.

It is also possible to put fixtures required by all tests in your project into an ini-file:

# content of pytest.ini
[pytest]
usefixtures = cleandir

Warning

Note this mark has no effect in fixture functions. For example, this will not work as expected:

@pytest.mark.usefixtures("my_other_fixture")
@pytest.fixture
def my_fixture_that_sadly_wont_use_my_other_fixture():
    ...

Currently this will not generate any error or warning, but this is intended to be handled by #3664.

Autouse fixtures (xUnit setup on steroids)¶

Occasionally, you may want to have fixtures get invoked automatically without declaring a function argument explicitly or a usefixtures decorator. As a practical example, suppose we have a database fixture which has a begin/rollback/commit architecture and we want to automatically surround each test method by a transaction and a rollback. Here is a dummy self-contained implementation of this idea:

# content of test_db_transact.py

import pytest


class DB:
    def __init__(self):
        self.intransaction = []

    def begin(self, name):
        self.intransaction.append(name)

    def rollback(self):
        self.intransaction.pop()


@pytest.fixture(scope="module")
def db():
    return DB()


class TestClass:
    @pytest.fixture(autouse=True)
    def transact(self, request, db):
        db.begin(request.function.__name__)
        yield
        db.rollback()

    def test_method1(self, db):
        assert db.intransaction == ["test_method1"]

    def test_method2(self, db):
        assert db.intransaction == ["test_method2"]

The class-level transact fixture is marked with autouse=true which implies that all test methods in the class will use this fixture without a need to state it in the test function signature or with a class-level usefixtures decorator.

If we run it, we get two passing tests:

$ pytest -q
..                                                                   [100%]
2 passed in 0.12s

Here is how autouse fixtures work in other scopes:

autouse fixtures obey the scope= keyword-argument: if an autouse fixture has scope='session' it will only be run once, no matter where it is defined. scope='class' means it will be run once per class, etc.
if an autouse fixture is defined in a test module, all its test functions automatically use it.
if an autouse fixture is defined in a conftest.py file then all tests in all test modules below its directory will invoke the fixture.
lastly, and please use that with care: if you define an autouse fixture in a plugin, it will be invoked for all tests in all projects where the plugin is installed. This can be useful if a fixture only anyway works in the presence of certain settings e. g. in the ini-file. Such a global fixture should always quickly determine if it should do any work and avoid otherwise expensive imports or computation.

Note that the above transact fixture may very well be a fixture that you want to make available in your project without having it generally active. The canonical way to do that is to put the transact definition into a conftest.py file without using autouse:

# content of conftest.py
@pytest.fixture
def transact(request, db):
    db.begin()
    yield
    db.rollback()

and then e.g. have a TestClass using it by declaring the need:

@pytest.mark.usefixtures("transact")
class TestClass:
    def test_method1(self):
        ...

All test methods in this TestClass will use the transaction fixture while other test classes or functions in the module will not use it unless they also add a transact reference.

Overriding fixtures on various levels¶

In relatively large test suite, you most likely need to override a global or root fixture with a locally defined one, keeping the test code readable and maintainable.

Override a fixture on a folder (conftest) level¶

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture
        def username():
            return 'username'

    test_something.py
        # content of tests/test_something.py
        def test_username(username):
            assert username == 'username'

    subfolder/
        __init__.py

        conftest.py
            # content of tests/subfolder/conftest.py
            import pytest

            @pytest.fixture
            def username(username):
                return 'overridden-' + username

        test_something.py
            # content of tests/subfolder/test_something.py
            def test_username(username):
                assert username == 'overridden-username'

As you can see, a fixture with the same name can be overridden for certain test folder level. Note that the base or super fixture can be accessed from the overriding fixture easily - used in the example above.

Override a fixture on a test module level¶

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture
        def username():
            return 'username'

    test_something.py
        # content of tests/test_something.py
        import pytest

        @pytest.fixture
        def username(username):
            return 'overridden-' + username

        def test_username(username):
            assert username == 'overridden-username'

    test_something_else.py
        # content of tests/test_something_else.py
        import pytest

        @pytest.fixture
        def username(username):
            return 'overridden-else-' + username

        def test_username(username):
            assert username == 'overridden-else-username'

In the example above, a fixture with the same name can be overridden for certain test module.

Override a fixture with direct test parametrization¶

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture
        def username():
            return 'username'

        @pytest.fixture
        def other_username(username):
            return 'other-' + username

    test_something.py
        # content of tests/test_something.py
        import pytest

        @pytest.mark.parametrize('username', ['directly-overridden-username'])
        def test_username(username):
            assert username == 'directly-overridden-username'

        @pytest.mark.parametrize('username', ['directly-overridden-username-other'])
        def test_username_other(other_username):
            assert other_username == 'other-directly-overridden-username-other'

In the example above, a fixture value is overridden by the test parameter value. Note that the value of the fixture can be overridden this way even if the test doesn’t use it directly (doesn’t mention it in the function prototype).

Override a parametrized fixture with non-parametrized one and vice versa¶

Given the tests file structure is:

tests/
    __init__.py

    conftest.py
        # content of tests/conftest.py
        import pytest

        @pytest.fixture(params=['one', 'two', 'three'])
        def parametrized_username(request):
            return request.param

        @pytest.fixture
        def non_parametrized_username(request):
            return 'username'

    test_something.py
        # content of tests/test_something.py
        import pytest

        @pytest.fixture
        def parametrized_username():
            return 'overridden-username'

        @pytest.fixture(params=['one', 'two', 'three'])
        def non_parametrized_username(request):
            return request.param

        def test_username(parametrized_username):
            assert parametrized_username == 'overridden-username'

        def test_parametrized_username(non_parametrized_username):
            assert non_parametrized_username in ['one', 'two', 'three']

    test_something_else.py
        # content of tests/test_something_else.py
        def test_username(parametrized_username):
            assert parametrized_username in ['one', 'two', 'three']

        def test_username(non_parametrized_username):
            assert non_parametrized_username == 'username'

In the example above, a parametrized fixture is overridden with a non-parametrized version, and a non-parametrized fixture is overridden with a parametrized version for certain test module. The same applies for the test folder level obviously.

Marking test functions with attributes¶

By using the pytest.mark helper you can easily set metadata on your test functions. There are some builtin markers, for example:

skip - always skip a test function
skipif - skip a test function if a certain condition is met
xfail - produce an “expected failure” outcome if a certain condition is met
parametrize to perform multiple calls to the same test function.

It’s easy to create custom markers or to apply markers to whole test classes or modules. Those markers can be used by plugins, and also are commonly used to select tests on the command-line with the -m option.

See Working with custom markers for examples which also serve as documentation.

Note

Marks can only be applied to tests, having no effect on fixtures.

Registering marks¶

You can register custom marks in your pytest.ini file like this:

[pytest]
markers =
    slow: marks tests as slow (deselect with '-m "not slow"')
    serial

Note that everything after the : is an optional description.

Alternatively, you can register new markers programmatically in a pytest_configure hook:

def pytest_configure(config):
    config.addinivalue_line(
        "markers", "env(name): mark test to run only on named environment"
    )

Registered marks appear in pytest’s help text and do not emit warnings (see the next section). It is recommended that third-party plugins always register their markers.

Raising errors on unknown marks¶

Unregistered marks applied with the @pytest.mark.name_of_the_mark decorator will always emit a warning in order to avoid silently doing something surprising due to mis-typed names. As described in the previous section, you can disable the warning for custom marks by registering them in your pytest.ini file or using a custom pytest_configure hook.

When the --strict-markers command-line flag is passed, any unknown marks applied with the @pytest.mark.name_of_the_mark decorator will trigger an error. You can enforce this validation in your project by adding --strict-markers to addopts:

[pytest]
addopts = --strict-markers
markers =
    slow: marks tests as slow (deselect with '-m "not slow"')
    serial

Monkeypatching/mocking modules and environments¶

Sometimes tests need to invoke functionality which depends on global settings or which invokes code which cannot be easily tested such as network access. The monkeypatch fixture helps you to safely set/delete an attribute, dictionary item or environment variable, or to modify sys.path for importing.

The monkeypatch fixture provides these helper methods for safely patching and mocking functionality in tests:

monkeypatch.setattr(obj, name, value, raising=True)
monkeypatch.delattr(obj, name, raising=True)
monkeypatch.setitem(mapping, name, value)
monkeypatch.delitem(obj, name, raising=True)
monkeypatch.setenv(name, value, prepend=False)
monkeypatch.delenv(name, raising=True)
monkeypatch.syspath_prepend(path)
monkeypatch.chdir(path)

All modifications will be undone after the requesting test function or fixture has finished. The raising parameter determines if a KeyError or AttributeError will be raised if the target of the set/deletion operation does not exist.

Consider the following scenarios:

1. Modifying the behavior of a function or the property of a class for a test e.g. there is an API call or database connection you will not make for a test but you know what the expected output should be. Use monkeypatch.setattr() to patch the function or property with your desired testing behavior. This can include your own functions. Use monkeypatch.delattr() to remove the function or property for the test.

2. Modifying the values of dictionaries e.g. you have a global configuration that you want to modify for certain test cases. Use monkeypatch.setitem() to patch the dictionary for the test. monkeypatch.delitem() can be used to remove items.

3. Modifying environment variables for a test e.g. to test program behavior if an environment variable is missing, or to set multiple values to a known variable. monkeypatch.setenv() and monkeypatch.delenv() can be used for these patches.

4. Use monkeypatch.setenv("PATH", value, prepend=os.pathsep) to modify $PATH, and monkeypatch.chdir() to change the context of the current working directory during a test.

5. Use monkeypatch.syspath_prepend() to modify sys.path which will also call pkg_resources.fixup_namespace_packages() and importlib.invalidate_caches().

See the monkeypatch blog post for some introduction material and a discussion of its motivation.

Simple example: monkeypatching functions¶

Consider a scenario where you are working with user directories. In the context of testing, you do not want your test to depend on the running user. monkeypatch can be used to patch functions dependent on the user to always return a specific value.

In this example, monkeypatch.setattr() is used to patch Path.home so that the known testing path Path("/abc") is always used when the test is run. This removes any dependency on the running user for testing purposes. monkeypatch.setattr() must be called before the function which will use the patched function is called. After the test function finishes the Path.home modification will be undone.

# contents of test_module.py with source code and the test
from pathlib import Path


def getssh():
    """Simple function to return expanded homedir ssh path."""
    return Path.home() / ".ssh"


def test_getssh(monkeypatch):
    # mocked return function to replace Path.home
    # always return '/abc'
    def mockreturn():
        return Path("/abc")

    # Application of the monkeypatch to replace Path.home
    # with the behavior of mockreturn defined above.
    monkeypatch.setattr(Path, "home", mockreturn)

    # Calling getssh() will use mockreturn in place of Path.home
    # for this test with the monkeypatch.
    x = getssh()
    assert x == Path("/abc/.ssh")

Monkeypatching returned objects: building mock classes¶

monkeypatch.setattr() can be used in conjunction with classes to mock returned objects from functions instead of values. Imagine a simple function to take an API url and return the json response.

# contents of app.py, a simple API retrieval example
import requests


def get_json(url):
    """Takes a URL, and returns the JSON."""
    r = requests.get(url)
    return r.json()

We need to mock r, the returned response object for testing purposes. The mock of r needs a .json() method which returns a dictionary. This can be done in our test file by defining a class to represent r.

# contents of test_app.py, a simple test for our API retrieval
# import requests for the purposes of monkeypatching
import requests

# our app.py that includes the get_json() function
# this is the previous code block example
import app

# custom class to be the mock return value
# will override the requests.Response returned from requests.get
class MockResponse:

    # mock json() method always returns a specific testing dictionary
    @staticmethod
    def json():
        return {"mock_key": "mock_response"}


def test_get_json(monkeypatch):

    # Any arguments may be passed and mock_get() will always return our
    # mocked object, which only has the .json() method.
    def mock_get(*args, **kwargs):
        return MockResponse()

    # apply the monkeypatch for requests.get to mock_get
    monkeypatch.setattr(requests, "get", mock_get)

    # app.get_json, which contains requests.get, uses the monkeypatch
    result = app.get_json("https://fakeurl")
    assert result["mock_key"] == "mock_response"

monkeypatch applies the mock for requests.get with our mock_get function. The mock_get function returns an instance of the MockResponse class, which has a json() method defined to return a known testing dictionary and does not require any outside API connection.

You can build the MockResponse class with the appropriate degree of complexity for the scenario you are testing. For instance, it could include an ok property that always returns True, or return different values from the json() mocked method based on input strings.

This mock can be shared across tests using a fixture:

# contents of test_app.py, a simple test for our API retrieval
import pytest
import requests

# app.py that includes the get_json() function
import app

# custom class to be the mock return value of requests.get()
class MockResponse:
    @staticmethod
    def json():
        return {"mock_key": "mock_response"}


# monkeypatched requests.get moved to a fixture
@pytest.fixture
def mock_response(monkeypatch):
    """Requests.get() mocked to return {'mock_key':'mock_response'}."""

    def mock_get(*args, **kwargs):
        return MockResponse()

    monkeypatch.setattr(requests, "get", mock_get)


# notice our test uses the custom fixture instead of monkeypatch directly
def test_get_json(mock_response):
    result = app.get_json("https://fakeurl")
    assert result["mock_key"] == "mock_response"

Furthermore, if the mock was designed to be applied to all tests, the fixture could be moved to a conftest.py file and use the with autouse=True option.

Global patch example: preventing “requests” from remote operations¶

If you want to prevent the “requests” library from performing http requests in all your tests, you can do:

# contents of conftest.py
import pytest


@pytest.fixture(autouse=True)
def no_requests(monkeypatch):
    """Remove requests.sessions.Session.request for all tests."""
    monkeypatch.delattr("requests.sessions.Session.request")

This autouse fixture will be executed for each test function and it will delete the method request.session.Session.request so that any attempts within tests to create http requests will fail.

Note

Be advised that it is not recommended to patch builtin functions such as open, compile, etc., because it might break pytest’s internals. If that’s unavoidable, passing --tb=native, --assert=plain and --capture=no might help although there’s no guarantee.

Note

Mind that patching stdlib functions and some third-party libraries used by pytest might break pytest itself, therefore in those cases it is recommended to use MonkeyPatch.context() to limit the patching to the block you want tested:

import functools


def test_partial(monkeypatch):
    with monkeypatch.context() as m:
        m.setattr(functools, "partial", 3)
        assert functools.partial == 3

See issue #3290 for details.

Monkeypatching environment variables¶

If you are working with environment variables you often need to safely change the values or delete them from the system for testing purposes. monkeypatch provides a mechanism to do this using the setenv and delenv method. Our example code to test:

# contents of our original code file e.g. code.py
import os


def get_os_user_lower():
    """Simple retrieval function.
    Returns lowercase USER or raises EnvironmentError."""
    username = os.getenv("USER")

    if username is None:
        raise OSError("USER environment is not set.")

    return username.lower()

There are two potential paths. First, the USER environment variable is set to a value. Second, the USER environment variable does not exist. Using monkeypatch both paths can be safely tested without impacting the running environment:

# contents of our test file e.g. test_code.py
import pytest


def test_upper_to_lower(monkeypatch):
    """Set the USER env var to assert the behavior."""
    monkeypatch.setenv("USER", "TestingUser")
    assert get_os_user_lower() == "testinguser"


def test_raise_exception(monkeypatch):
    """Remove the USER env var and assert EnvironmentError is raised."""
    monkeypatch.delenv("USER", raising=False)

    with pytest.raises(OSError):
        _ = get_os_user_lower()

This behavior can be moved into fixture structures and shared across tests:

# contents of our test file e.g. test_code.py
import pytest


@pytest.fixture
def mock_env_user(monkeypatch):
    monkeypatch.setenv("USER", "TestingUser")


@pytest.fixture
def mock_env_missing(monkeypatch):
    monkeypatch.delenv("USER", raising=False)


# notice the tests reference the fixtures for mocks
def test_upper_to_lower(mock_env_user):
    assert get_os_user_lower() == "testinguser"


def test_raise_exception(mock_env_missing):
    with pytest.raises(OSError):
        _ = get_os_user_lower()

Monkeypatching dictionaries¶

monkeypatch.setitem() can be used to safely set the values of dictionaries to specific values during tests. Take this simplified connection string example:

# contents of app.py to generate a simple connection string
DEFAULT_CONFIG = {"user": "user1", "database": "db1"}


def create_connection_string(config=None):
    """Creates a connection string from input or defaults."""
    config = config or DEFAULT_CONFIG
    return f"User Id={config['user']}; Location={config['database']};"

For testing purposes we can patch the DEFAULT_CONFIG dictionary to specific values.

# contents of test_app.py
# app.py with the connection string function (prior code block)
import app


def test_connection(monkeypatch):

    # Patch the values of DEFAULT_CONFIG to specific
    # testing values only for this test.
    monkeypatch.setitem(app.DEFAULT_CONFIG, "user", "test_user")
    monkeypatch.setitem(app.DEFAULT_CONFIG, "database", "test_db")

    # expected result based on the mocks
    expected = "User Id=test_user; Location=test_db;"

    # the test uses the monkeypatched dictionary settings
    result = app.create_connection_string()
    assert result == expected

You can use the monkeypatch.delitem() to remove values.

# contents of test_app.py
import pytest

# app.py with the connection string function
import app


def test_missing_user(monkeypatch):

    # patch the DEFAULT_CONFIG t be missing the 'user' key
    monkeypatch.delitem(app.DEFAULT_CONFIG, "user", raising=False)

    # Key error expected because a config is not passed, and the
    # default is now missing the 'user' entry.
    with pytest.raises(KeyError):
        _ = app.create_connection_string()

The modularity of fixtures gives you the flexibility to define separate fixtures for each potential mock and reference them in the needed tests.

# contents of test_app.py
import pytest

# app.py with the connection string function
import app

# all of the mocks are moved into separated fixtures
@pytest.fixture
def mock_test_user(monkeypatch):
    """Set the DEFAULT_CONFIG user to test_user."""
    monkeypatch.setitem(app.DEFAULT_CONFIG, "user", "test_user")


@pytest.fixture
def mock_test_database(monkeypatch):
    """Set the DEFAULT_CONFIG database to test_db."""
    monkeypatch.setitem(app.DEFAULT_CONFIG, "database", "test_db")


@pytest.fixture
def mock_missing_default_user(monkeypatch):
    """Remove the user key from DEFAULT_CONFIG"""
    monkeypatch.delitem(app.DEFAULT_CONFIG, "user", raising=False)


# tests reference only the fixture mocks that are needed
def test_connection(mock_test_user, mock_test_database):

    expected = "User Id=test_user; Location=test_db;"

    result = app.create_connection_string()
    assert result == expected


def test_missing_user(mock_missing_default_user):

    with pytest.raises(KeyError):
        _ = app.create_connection_string()

API Reference¶

Consult the docs for the MonkeyPatch class.

Temporary directories and files¶

The `tmp_path` fixture¶

You can use the tmp_path fixture which will provide a temporary directory unique to the test invocation, created in the base temporary directory.

tmp_path is a pathlib/pathlib2.Path object. Here is an example test usage:

# content of test_tmp_path.py
import os

CONTENT = "content"


def test_create_file(tmp_path):
    d = tmp_path / "sub"
    d.mkdir()
    p = d / "hello.txt"
    p.write_text(CONTENT)
    assert p.read_text() == CONTENT
    assert len(list(tmp_path.iterdir())) == 1
    assert 0

Running this would result in a passed test except for the last assert 0 line which we use to look at values:

$ pytest test_tmp_path.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_tmp_path.py F                                                   [100%]

================================= FAILURES =================================
_____________________________ test_create_file _____________________________

tmp_path = PosixPath('PYTEST_TMPDIR/test_create_file0')

    def test_create_file(tmp_path):
        d = tmp_path / "sub"
        d.mkdir()
        p = d / "hello.txt"
        p.write_text(CONTENT)
        assert p.read_text() == CONTENT
        assert len(list(tmp_path.iterdir())) == 1
>       assert 0
E       assert 0

test_tmp_path.py:13: AssertionError
============================ 1 failed in 0.12s =============================

The `tmp_path_factory` fixture¶

The tmp_path_factory is a session-scoped fixture which can be used to create arbitrary temporary directories from any other fixture or test.

It is intended to replace tmpdir_factory, and returns pathlib.Path instances.

See tmp_path_factory API for details.

The ‘tmpdir’ fixture¶

You can use the tmpdir fixture which will provide a temporary directory unique to the test invocation, created in the base temporary directory.

tmpdir is a py.path.local object which offers os.path methods and more. Here is an example test usage:

# content of test_tmpdir.py
import os


def test_create_file(tmpdir):
    p = tmpdir.mkdir("sub").join("hello.txt")
    p.write("content")
    assert p.read() == "content"
    assert len(tmpdir.listdir()) == 1
    assert 0

Running this would result in a passed test except for the last assert 0 line which we use to look at values:

$ pytest test_tmpdir.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_tmpdir.py F                                                     [100%]

================================= FAILURES =================================
_____________________________ test_create_file _____________________________

tmpdir = local('PYTEST_TMPDIR/test_create_file0')

    def test_create_file(tmpdir):
        p = tmpdir.mkdir("sub").join("hello.txt")
        p.write("content")
        assert p.read() == "content"
        assert len(tmpdir.listdir()) == 1
>       assert 0
E       assert 0

test_tmpdir.py:9: AssertionError
============================ 1 failed in 0.12s =============================

The ‘tmpdir_factory’ fixture¶

The tmpdir_factory is a session-scoped fixture which can be used to create arbitrary temporary directories from any other fixture or test.

For example, suppose your test suite needs a large image on disk, which is generated procedurally. Instead of computing the same image for each test that uses it into its own tmpdir, you can generate it once per-session to save time:

# contents of conftest.py
import pytest


@pytest.fixture(scope="session")
def image_file(tmpdir_factory):
    img = compute_expensive_image()
    fn = tmpdir_factory.mktemp("data").join("img.png")
    img.save(str(fn))
    return fn


# contents of test_image.py
def test_histogram(image_file):
    img = load_image(image_file)
    # compute and test histogram

See tmpdir_factory API for details.

The default base temporary directory¶

Temporary directories are by default created as sub-directories of the system temporary directory. The base name will be pytest-NUM where NUM will be incremented with each test run. Moreover, entries older than 3 temporary directories will be removed.

You can override the default temporary directory setting like this:

pytest --basetemp=mydir

When distributing tests on the local machine, pytest takes care to configure a basetemp directory for the sub processes such that all temporary data lands below a single per-test run basetemp directory.

Capturing of the stdout/stderr output¶

Default stdout/stderr/stdin capturing behaviour¶

During test execution any output sent to stdout and stderr is captured. If a test or a setup method fails its according captured output will usually be shown along with the failure traceback. (this behavior can be configured by the --show-capture command-line option).

In addition, stdin is set to a “null” object which will fail on attempts to read from it because it is rarely desired to wait for interactive input when running automated tests.

By default capturing is done by intercepting writes to low level file descriptors. This allows to capture output from simple print statements as well as output from a subprocess started by a test.

Setting capturing methods or disabling capturing¶

There are three ways in which pytest can perform capturing:

fd (file descriptor) level capturing (default): All writes going to the operating system file descriptors 1 and 2 will be captured.
sys level capturing: Only writes to Python files sys.stdout and sys.stderr will be captured. No capturing of writes to filedescriptors is performed.
tee-sys capturing: Python writes to sys.stdout and sys.stderr will be captured, however the writes will also be passed-through to the actual sys.stdout and sys.stderr. This allows output to be ‘live printed’ and captured for plugin use, such as junitxml (new in pytest 5.4).

You can influence output capturing mechanisms from the command line:

pytest -s                  # disable all capturing
pytest --capture=sys       # replace sys.stdout/stderr with in-mem files
pytest --capture=fd        # also point filedescriptors 1 and 2 to temp file
pytest --capture=tee-sys   # combines 'sys' and '-s', capturing sys.stdout/stderr
                           # and passing it along to the actual sys.stdout/stderr

Using print statements for debugging¶

One primary benefit of the default capturing of stdout/stderr output is that you can use print statements for debugging:

# content of test_module.py

def setup_function(function):
    print("setting up", function)

def test_func1():
    assert True

def test_func2():
    assert False

and running this module will show you precisely the output of the failing function and hide the other one:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_module.py .F                                                    [100%]

================================= FAILURES =================================
________________________________ test_func2 ________________________________

    def test_func2():
>       assert False
E       assert False

test_module.py:12: AssertionError
-------------------------- Captured stdout setup ---------------------------
setting up <function test_func2 at 0xdeadbeef>
======================= 1 failed, 1 passed in 0.12s ========================

Accessing captured output from a test function¶

The capsys, capsysbinary, capfd, and capfdbinary fixtures allow access to stdout/stderr output created during test execution. Here is an example test function that performs some output related checks:

def test_myoutput(capsys):  # or use "capfd" for fd-level
    print("hello")
    sys.stderr.write("world\n")
    captured = capsys.readouterr()
    assert captured.out == "hello\n"
    assert captured.err == "world\n"
    print("next")
    captured = capsys.readouterr()
    assert captured.out == "next\n"

The readouterr() call snapshots the output so far - and capturing will be continued. After the test function finishes the original streams will be restored. Using capsys this way frees your test from having to care about setting/resetting output streams and also interacts well with pytest’s own per-test capturing.

If you want to capture on filedescriptor level you can use the capfd fixture which offers the exact same interface but allows to also capture output from libraries or subprocesses that directly write to operating system level output streams (FD1 and FD2).

The return value from readouterr changed to a namedtuple with two attributes, out and err.

If the code under test writes non-textual data, you can capture this using the capsysbinary fixture which instead returns bytes from the readouterr method. The capfsysbinary fixture is currently only available in python 3.

If the code under test writes non-textual data, you can capture this using the capfdbinary fixture which instead returns bytes from the readouterr method. The capfdbinary fixture operates on the filedescriptor level.

To temporarily disable capture within a test, both capsys and capfd have a disabled() method that can be used as a context manager, disabling capture inside the with block:

def test_disabling_capturing(capsys):
    print("this output is captured")
    with capsys.disabled():
        print("output not captured, going directly to sys.stdout")
    print("this output is also captured")

Warnings Capture¶

Starting from version 3.1, pytest now automatically catches warnings during test execution and displays them at the end of the session:

# content of test_show_warnings.py
import warnings


def api_v1():
    warnings.warn(UserWarning("api v1, should use functions from v2"))
    return 1


def test_one():
    assert api_v1() == 1

Running pytest now produces this output:

$ pytest test_show_warnings.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_show_warnings.py .                                              [100%]

============================= warnings summary =============================
test_show_warnings.py::test_one
  $REGENDOC_TMPDIR/test_show_warnings.py:5: UserWarning: api v1, should use functions from v2
    warnings.warn(UserWarning("api v1, should use functions from v2"))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================= 1 passed, 1 warning in 0.12s =======================

The -W flag can be passed to control which warnings will be displayed or even turn them into errors:

$ pytest -q test_show_warnings.py -W error::UserWarning
F                                                                    [100%]
================================= FAILURES =================================
_________________________________ test_one _________________________________

    def test_one():
>       assert api_v1() == 1

test_show_warnings.py:10:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def api_v1():
>       warnings.warn(UserWarning("api v1, should use functions from v2"))
E       UserWarning: api v1, should use functions from v2

test_show_warnings.py:5: UserWarning
1 failed in 0.12s

The same option can be set in the pytest.ini file using the filterwarnings ini option. For example, the configuration below will ignore all user warnings, but will transform all other warnings into errors.

[pytest]
filterwarnings =
    error
    ignore::UserWarning

When a warning matches more than one option in the list, the action for the last matching option is performed.

Both -W command-line option and filterwarnings ini option are based on Python’s own -W option and warnings.simplefilter, so please refer to those sections in the Python documentation for other examples and advanced usage.

`@pytest.mark.filterwarnings`¶

You can use the @pytest.mark.filterwarnings to add warning filters to specific test items, allowing you to have finer control of which warnings should be captured at test, class or even module level:

import warnings


def api_v1():
    warnings.warn(UserWarning("api v1, should use functions from v2"))
    return 1


@pytest.mark.filterwarnings("ignore:api v1")
def test_one():
    assert api_v1() == 1

Filters applied using a mark take precedence over filters passed on the command line or configured by the filterwarnings ini option.

You may apply a filter to all tests of a class by using the filterwarnings mark as a class decorator or to all tests in a module by setting the pytestmark variable:

# turns all warnings into errors for this module
pytestmark = pytest.mark.filterwarnings("error")

Credits go to Florian Schulze for the reference implementation in the pytest-warnings plugin.

Disabling warnings summary¶

Although not recommended, you can use the --disable-warnings command-line option to suppress the warning summary entirely from the test run output.

Disabling warning capture entirely¶

This plugin is enabled by default but can be disabled entirely in your pytest.ini file with:

[pytest]
addopts = -p no:warnings

Or passing -p no:warnings in the command-line. This might be useful if your test suites handles warnings using an external system.

DeprecationWarning and PendingDeprecationWarning¶

By default pytest will display DeprecationWarning and PendingDeprecationWarning warnings from user code and third-party libraries, as recommended by PEP-0565. This helps users keep their code modern and avoid breakages when deprecated warnings are effectively removed.

Sometimes it is useful to hide some specific deprecation warnings that happen in code that you have no control over (such as third-party libraries), in which case you might use the warning filters options (ini or marks) to ignore those warnings.

For example:

[pytest]
filterwarnings =
    ignore:.*U.*mode is deprecated:DeprecationWarning

This will ignore all warnings of type DeprecationWarning where the start of the message matches the regular expression ".*U.*mode is deprecated".

Note

If warnings are configured at the interpreter level, using the PYTHONWARNINGS environment variable or the -W command-line option, pytest will not configure any filters by default.

Also pytest doesn’t follow PEP-0506 suggestion of resetting all warning filters because it might break test suites that configure warning filters themselves by calling warnings.simplefilter (see issue #2430 for an example of that).

Ensuring code triggers a deprecation warning¶

You can also use pytest.deprecated_call() for checking that a certain function call triggers a DeprecationWarning or PendingDeprecationWarning:

import pytest


def test_myfunction_deprecated():
    with pytest.deprecated_call():
        myfunction(17)

This test will fail if myfunction does not issue a deprecation warning when called with a 17 argument.

By default, DeprecationWarning and PendingDeprecationWarning will not be caught when using pytest.warns() or recwarn because the default Python warnings filters hide them. If you wish to record them in your own code, use warnings.simplefilter('always'):

import warnings
import pytest


def test_deprecation(recwarn):
    warnings.simplefilter("always")
    myfunction(17)
    assert len(recwarn) == 1
    assert recwarn.pop(DeprecationWarning)

The recwarn fixture automatically ensures to reset the warnings filter at the end of the test, so no global state is leaked.

Asserting warnings with the warns function¶

You can check that code raises a particular warning using pytest.warns, which works in a similar manner to raises:

import warnings
import pytest


def test_warning():
    with pytest.warns(UserWarning):
        warnings.warn("my warning", UserWarning)

The test will fail if the warning in question is not raised. The keyword argument match to assert that the exception matches a text or regex:

>>> with warns(UserWarning, match='must be 0 or None'):
...     warnings.warn("value must be 0 or None", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("value must be 42", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("this is not here", UserWarning)
Traceback (most recent call last):
  ...
Failed: DID NOT WARN. No warnings of type ...UserWarning... was emitted...

You can also call pytest.warns on a function or code string:

pytest.warns(expected_warning, func, *args, **kwargs)
pytest.warns(expected_warning, "func(*args, **kwargs)")

The function also returns a list of all raised warnings (as warnings.WarningMessage objects), which you can query for additional information:

with pytest.warns(RuntimeWarning) as record:
    warnings.warn("another warning", RuntimeWarning)

# check that only one warning was raised
assert len(record) == 1
# check that the message matches
assert record[0].message.args[0] == "another warning"

Alternatively, you can examine raised warnings in detail using the recwarn fixture (see below).

Note

DeprecationWarning and PendingDeprecationWarning are treated differently; see Ensuring code triggers a deprecation warning.

Recording warnings¶

You can record raised warnings either using pytest.warns or with the recwarn fixture.

To record with pytest.warns without asserting anything about the warnings, pass None as the expected warning type:

with pytest.warns(None) as record:
    warnings.warn("user", UserWarning)
    warnings.warn("runtime", RuntimeWarning)

assert len(record) == 2
assert str(record[0].message) == "user"
assert str(record[1].message) == "runtime"

The recwarn fixture will record warnings for the whole function:

import warnings


def test_hello(recwarn):
    warnings.warn("hello", UserWarning)
    assert len(recwarn) == 1
    w = recwarn.pop(UserWarning)
    assert issubclass(w.category, UserWarning)
    assert str(w.message) == "hello"
    assert w.filename
    assert w.lineno

Both recwarn and pytest.warns return the same interface for recorded warnings: a WarningsRecorder instance. To view the recorded warnings, you can iterate over this instance, call len on it to get the number of recorded warnings, or index into it to get a particular recorded warning.

Full API: WarningsRecorder.

Custom failure messages¶

Recording warnings provides an opportunity to produce custom test failure messages for when no warnings are issued or other conditions are met.

def test():
    with pytest.warns(Warning) as record:
        f()
        if not record:
            pytest.fail("Expected a warning!")

If no warnings are issued when calling f, then not record will evaluate to True. You can then call pytest.fail with a custom error message.

Internal pytest warnings¶

pytest may generate its own warnings in some situations, such as improper usage or deprecated features.

For example, pytest will emit a warning if it encounters a class that matches python_classes but also defines an __init__ constructor, as this prevents the class from being instantiated:

# content of test_pytest_warnings.py
class Test:
    def __init__(self):
        pass

    def test_foo(self):
        assert 1 == 1

$ pytest test_pytest_warnings.py -q

============================= warnings summary =============================
test_pytest_warnings.py:1
  $REGENDOC_TMPDIR/test_pytest_warnings.py:1: PytestCollectionWarning: cannot collect test class 'Test' because it has a __init__ constructor (from: test_pytest_warnings.py)
    class Test:

-- Docs: https://docs.pytest.org/en/latest/warnings.html
1 warning in 0.12s

These warnings might be filtered using the same builtin mechanisms used to filter other types of warnings.

Please read our Backwards Compatibility Policy to learn how we proceed about deprecating and eventually removing features.

The following warning types are used by pytest and are part of the public API:

class PytestWarning¶

Bases: UserWarning.

Base class for all warnings emitted by pytest.

class PytestAssertRewriteWarning¶

Bases: PytestWarning.

Warning emitted by the pytest assert rewrite module.

class PytestCacheWarning¶

Bases: PytestWarning.

Warning emitted by the cache plugin in various situations.

class PytestCollectionWarning¶

Bases: PytestWarning.

Warning emitted when pytest is not able to collect a file or symbol in a module.

class PytestConfigWarning¶

Bases: PytestWarning.

Warning emitted for configuration issues.

class PytestDeprecationWarning¶

Bases: pytest.PytestWarning, DeprecationWarning.

Warning class for features that will be removed in a future version.

class PytestExperimentalApiWarning¶

Bases: pytest.PytestWarning, FutureWarning.

Warning category used to denote experiments in pytest. Use sparingly as the API might change or even be removed completely in future version

class PytestUnhandledCoroutineWarning¶

Bases: PytestWarning.

Warning emitted when pytest encounters a test function which is a coroutine, but it was not handled by any async-aware plugin. Coroutine test functions are not natively supported.

class PytestUnknownMarkWarning¶

Bases: PytestWarning.

Warning emitted on use of unknown markers. See https://docs.pytest.org/en/latest/mark.html for details.

Doctest integration for modules and test files¶

By default all files matching the test*.txt pattern will be run through the python standard doctest module. You can change the pattern by issuing:

pytest --doctest-glob='*.rst'

on the command line. --doctest-glob can be given multiple times in the command-line.

If you then have a text file like this:

# content of test_example.txt

hello this is a doctest
>>> x = 3
>>> x
3

then you can just invoke pytest directly:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 1 item

test_example.txt .                                                   [100%]

============================ 1 passed in 0.12s =============================

By default, pytest will collect test*.txt files looking for doctest directives, but you can pass additional globs using the --doctest-glob option (multi-allowed).

In addition to text files, you can also execute doctests directly from docstrings of your classes and functions, including from test modules:

# content of mymodule.py
def something():
    """ a doctest in a docstring
    >>> something()
    42
    """
    return 42

$ pytest --doctest-modules
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

mymodule.py .                                                        [ 50%]
test_example.txt .                                                   [100%]

============================ 2 passed in 0.12s =============================

You can make these changes permanent in your project by putting them into a pytest.ini file like this:

# content of pytest.ini
[pytest]
addopts = --doctest-modules

Note

The builtin pytest doctest supports only doctest blocks, but if you are looking for more advanced checking over all your documentation, including doctests, .. codeblock:: python Sphinx directive support, and any other examples your documentation may include, you may wish to consider Sybil. It provides pytest integration out of the box.

Encoding¶

The default encoding is UTF-8, but you can specify the encoding that will be used for those doctest files using the doctest_encoding ini option:

# content of pytest.ini
[pytest]
doctest_encoding = latin1

Using ‘doctest’ options¶

Python’s standard doctest module provides some options to configure the strictness of doctest tests. In pytest, you can enable those flags using the configuration file.

For example, to make pytest ignore trailing whitespaces and ignore lengthy exception stack traces you can just write:

[pytest]
doctest_optionflags= NORMALIZE_WHITESPACE IGNORE_EXCEPTION_DETAIL

Alternatively, options can be enabled by an inline comment in the doc test itself:

>>> something_that_raises()  # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
ValueError: ...

pytest also introduces new options:

ALLOW_UNICODE: when enabled, the u prefix is stripped from unicode strings in expected doctest output. This allows doctests to run in Python 2 and Python 3 unchanged.
ALLOW_BYTES: similarly, the b prefix is stripped from byte strings in expected doctest output.
NUMBER: when enabled, floating-point numbers only need to match as far as the precision you have written in the expected doctest output. For example, the following output would only need to match to 2 decimal places:
```
>>> math.pi
3.14
```
If you wrote 3.1416 then the actual output would need to match to 4 decimal places; and so on.

This avoids false positives caused by limited floating-point precision, like this:
```
Expected:
    0.233
Got:
    0.23300000000000001
```
NUMBER also supports lists of floating-point numbers – in fact, it matches floating-point numbers appearing anywhere in the output, even inside a string! This means that it may not be appropriate to enable globally in doctest_optionflags in your configuration file.

New in version 5.1.

Continue on failure¶

By default, pytest would report only the first failure for a given doctest. If you want to continue the test even when you have failures, do:

pytest --doctest-modules --doctest-continue-on-failure

Output format¶

You can change the diff output format on failure for your doctests by using one of standard doctest modules format in options (see doctest.REPORT_UDIFF, doctest.REPORT_CDIFF, doctest.REPORT_NDIFF, doctest.REPORT_ONLY_FIRST_FAILURE):

pytest --doctest-modules --doctest-report none
pytest --doctest-modules --doctest-report udiff
pytest --doctest-modules --doctest-report cdiff
pytest --doctest-modules --doctest-report ndiff
pytest --doctest-modules --doctest-report only_first_failure

pytest-specific features¶

Some features are provided to make writing doctests easier or with better integration with your existing test suite. Keep in mind however that by using those features you will make your doctests incompatible with the standard doctests module.

Using fixtures¶

It is possible to use fixtures using the getfixture helper:

# content of example.rst
>>> tmp = getfixture('tmpdir')
>>> ...
>>>

Also, Using fixtures from classes, modules or projects and Autouse fixtures (xUnit setup on steroids) fixtures are supported when executing text doctest files.

‘doctest_namespace’ fixture¶

The doctest_namespace fixture can be used to inject items into the namespace in which your doctests run. It is intended to be used within your own fixtures to provide the tests that use them with context.

doctest_namespace is a standard dict object into which you place the objects you want to appear in the doctest namespace:

# content of conftest.py
import numpy


@pytest.fixture(autouse=True)
def add_np(doctest_namespace):
    doctest_namespace["np"] = numpy

which can then be used in your doctests directly:

# content of numpy.py
def arange():
    """
    >>> a = np.arange(10)
    >>> len(a)
    10
    """
    pass

Note that like the normal conftest.py, the fixtures are discovered in the directory tree conftest is in. Meaning that if you put your doctest with your source code, the relevant conftest.py needs to be in the same directory tree. Fixtures will not be discovered in a sibling directory tree!

Skipping tests dynamically¶

New in version 4.4.

You can use pytest.skip to dynamically skip doctests. For example:

>>> import sys, pytest
>>> if sys.platform.startswith('win'):
...     pytest.skip('this doctest does not work on Windows')
...

Skip and xfail: dealing with tests that cannot succeed¶

You can mark test functions that cannot be run on certain platforms or that you expect to fail so pytest can deal with them accordingly and present a summary of the test session, while keeping the test suite green.

A skip means that you expect your test to pass only if some conditions are met, otherwise pytest should skip running the test altogether. Common examples are skipping windows-only tests on non-windows platforms, or skipping tests that depend on an external resource which is not available at the moment (for example a database).

A xfail means that you expect a test to fail for some reason. A common example is a test for a feature not yet implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with pytest.mark.xfail), it’s an xpass and will be reported in the test summary.

pytest counts and lists skip and xfail tests separately. Detailed information about skipped/xfailed tests is not shown by default to avoid cluttering the output. You can use the -r option to see details corresponding to the “short” letters shown in the test progress:

pytest -rxXs  # show extra info on xfailed, xpassed, and skipped tests

More details on the -r option can be found by running pytest -h.

(See How to change command line options defaults)

Skipping test functions¶

The simplest way to skip a test function is to mark it with the skip decorator which may be passed an optional reason:

@pytest.mark.skip(reason="no way of currently testing this")
def test_the_unknown():
    ...

Alternatively, it is also possible to skip imperatively during test execution or setup by calling the pytest.skip(reason) function:

def test_function():
    if not valid_config():
        pytest.skip("unsupported configuration")

The imperative method is useful when it is not possible to evaluate the skip condition during import time.

It is also possible to skip the whole module using pytest.skip(reason, allow_module_level=True) at the module level:

import sys
import pytest

if not sys.platform.startswith("win"):
    pytest.skip("skipping windows-only tests", allow_module_level=True)

Reference: pytest.mark.skip

`skipif`¶

If you wish to skip something conditionally then you can use skipif instead. Here is an example of marking a test function to be skipped when run on an interpreter earlier than Python3.6:

import sys


@pytest.mark.skipif(sys.version_info < (3, 6), reason="requires python3.6 or higher")
def test_function():
    ...

If the condition evaluates to True during collection, the test function will be skipped, with the specified reason appearing in the summary when using -rs.

You can share skipif markers between modules. Consider this test module:

# content of test_mymodule.py
import mymodule

minversion = pytest.mark.skipif(
    mymodule.__versioninfo__ < (1, 1), reason="at least mymodule-1.1 required"
)


@minversion
def test_function():
    ...

You can import the marker and reuse it in another test module:

# test_myothermodule.py
from test_mymodule import minversion


@minversion
def test_anotherfunction():
    ...

For larger test suites it’s usually a good idea to have one file where you define the markers which you then consistently apply throughout your test suite.

Alternatively, you can use condition strings instead of booleans, but they can’t be shared between modules easily so they are supported mainly for backward compatibility reasons.

Reference: pytest.mark.skipif

Skip all test functions of a class or module¶

You can use the skipif marker (as any other marker) on classes:

@pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
class TestPosixCalls:
    def test_function(self):
        "will not be setup or run under 'win32' platform"

If the condition is True, this marker will produce a skip result for each of the test methods of that class.

If you want to skip all test functions of a module, you may use the pytestmark name on the global level:

# test_module.py
pytestmark = pytest.mark.skipif(...)

If multiple skipif decorators are applied to a test function, it will be skipped if any of the skip conditions is true.

Skipping files or directories¶

Sometimes you may need to skip an entire file or directory, for example if the tests rely on Python version-specific features or contain code that you do not wish pytest to run. In this case, you must exclude the files and directories from collection. Refer to Customizing test collection for more information.

Skipping on a missing import dependency¶

You can skip tests on a missing import by using pytest.importorskip at module level, within a test, or test setup function.

docutils = pytest.importorskip("docutils")

If docutils cannot be imported here, this will lead to a skip outcome of the test. You can also skip based on the version number of a library:

docutils = pytest.importorskip("docutils", minversion="0.3")

The version will be read from the specified module’s __version__ attribute.

Summary¶

Here’s a quick guide on how to skip tests in a module in different situations:

Skip all tests in a module unconditionally:

pytestmark = pytest.mark.skip("all tests still WIP")

Skip all tests in a module based on some condition:

pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="tests for linux only")

Skip all tests in a module if some import is missing:

pexpect = pytest.importorskip("pexpect")

XFail: mark test functions as expected to fail¶

You can use the xfail marker to indicate that you expect a test to fail:

@pytest.mark.xfail
def test_function():
    ...

This test will run but no traceback will be reported when it fails. Instead, terminal reporting will list it in the “expected to fail” (XFAIL) or “unexpectedly passing” (XPASS) sections.

Alternatively, you can also mark a test as XFAIL from within the test or its setup function imperatively:

def test_function():
    if not valid_config():
        pytest.xfail("failing configuration (but should work)")

def test_function2():
    import slow_module

    if slow_module.slow_function():
        pytest.xfail("slow_module taking too long")

These two examples illustrate situations where you don’t want to check for a condition at the module level, which is when a condition would otherwise be evaluated for marks.

This will make test_function XFAIL. Note that no other code is executed after the pytest.xfail call, differently from the marker. That’s because it is implemented internally by raising a known exception.

Reference: pytest.mark.xfail

`strict` parameter¶

Both XFAIL and XPASS don’t fail the test suite by default. You can change this by setting the strict keyword-only parameter to True:

@pytest.mark.xfail(strict=True)
def test_function():
    ...

This will make XPASS (“unexpectedly passing”) results from this test to fail the test suite.

You can change the default value of the strict parameter using the xfail_strict ini option:

[pytest]
xfail_strict=true

`reason` parameter¶

As with skipif you can also mark your expectation of a failure on a particular platform:

@pytest.mark.xfail(sys.version_info >= (3, 6), reason="python3.6 api changes")
def test_function():
    ...

`raises` parameter¶

If you want to be more specific as to why the test is failing, you can specify a single exception, or a tuple of exceptions, in the raises argument.

@pytest.mark.xfail(raises=RuntimeError)
def test_function():
    ...

Then the test will be reported as a regular failure if it fails with an exception not mentioned in raises.

`run` parameter¶

If a test should be marked as xfail and reported as such but should not be even executed, use the run parameter as False:

@pytest.mark.xfail(run=False)
def test_function():
    ...

This is specially useful for xfailing tests that are crashing the interpreter and should be investigated later.

Ignoring xfail¶

By specifying on the commandline:

pytest --runxfail

you can force the running and reporting of an xfail marked test as if it weren’t marked at all. This also causes pytest.xfail to produce no effect.

Examples¶

Here is a simple test file with the several usages:

import pytest

xfail = pytest.mark.xfail


@xfail
def test_hello():
    assert 0


@xfail(run=False)
def test_hello2():
    assert 0


@xfail("hasattr(os, 'sep')")
def test_hello3():
    assert 0


@xfail(reason="bug 110")
def test_hello4():
    assert 0


@xfail('pytest.__version__[0] != "17"')
def test_hello5():
    assert 0


def test_hello6():
    pytest.xfail("reason")


@xfail(raises=IndexError)
def test_hello7():
    x = []
    x[1] = 1

Running it with the report-on-xfail option gives this output:

example $ pytest -rx xfail_demo.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR/example
collected 7 items

xfail_demo.py xxxxxxx                                                [100%]

========================= short test summary info ==========================
XFAIL xfail_demo.py::test_hello
XFAIL xfail_demo.py::test_hello2
  reason: [NOTRUN]
XFAIL xfail_demo.py::test_hello3
  condition: hasattr(os, 'sep')
XFAIL xfail_demo.py::test_hello4
  bug 110
XFAIL xfail_demo.py::test_hello5
  condition: pytest.__version__[0] != "17"
XFAIL xfail_demo.py::test_hello6
  reason: reason
XFAIL xfail_demo.py::test_hello7
============================ 7 xfailed in 0.12s ============================

Skip/xfail with parametrize¶

It is possible to apply markers like skip and xfail to individual test instances when using parametrize:

import pytest


@pytest.mark.parametrize(
    ("n", "expected"),
    [
        (1, 2),
        pytest.param(1, 0, marks=pytest.mark.xfail),
        pytest.param(1, 3, marks=pytest.mark.xfail(reason="some bug")),
        (2, 3),
        (3, 4),
        (4, 5),
        pytest.param(
            10, 11, marks=pytest.mark.skipif(sys.version_info >= (3, 0), reason="py2k")
        ),
    ],
)
def test_increment(n, expected):
    assert n + 1 == expected

Parametrizing fixtures and test functions¶

pytest enables test parametrization at several levels:

pytest.fixture() allows one to parametrize fixture functions.

@pytest.mark.parametrize allows one to define multiple sets of arguments and fixtures at the test function or class.
pytest_generate_tests allows one to define custom parametrization schemes or extensions.

`@pytest.mark.parametrize`: parametrizing test functions¶

The builtin pytest.mark.parametrize decorator enables parametrization of arguments for a test function. Here is a typical example of a test function that implements checking that a certain input leads to an expected output:

# content of test_expectation.py
import pytest


@pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Here, the @parametrize decorator defines three different (test_input,expected) tuples so that the test_eval function will run three times using them in turn:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 3 items

test_expectation.py ..F                                              [100%]

================================= FAILURES =================================
____________________________ test_eval[6*9-42] _____________________________

test_input = '6*9', expected = 42

    @pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
    def test_eval(test_input, expected):
>       assert eval(test_input) == expected
E       AssertionError: assert 54 == 42
E        +  where 54 = eval('6*9')

test_expectation.py:6: AssertionError
======================= 1 failed, 2 passed in 0.12s ========================

Note

pytest by default escapes any non-ascii characters used in unicode strings for the parametrization because it has several downsides. If however you would like to use unicode strings in parametrization and see them in the terminal as is (non-escaped), use this option in your pytest.ini:

[pytest]
disable_test_id_escaping_and_forfeit_all_rights_to_community_support = True

Keep in mind however that this might cause unwanted side effects and even bugs depending on the OS used and plugins currently installed, so use it at your own risk.

As designed in this example, only one pair of input/output values fails the simple test function. And as usual with test function arguments, you can see the input and output values in the traceback.

Note that you could also use the parametrize marker on a class or a module (see Marking test functions with attributes) which would invoke several functions with the argument sets.

It is also possible to mark individual test instances within parametrize, for example with the builtin mark.xfail:

# content of test_expectation.py
import pytest


@pytest.mark.parametrize(
    "test_input,expected",
    [("3+5", 8), ("2+4", 6), pytest.param("6*9", 42, marks=pytest.mark.xfail)],
)
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Let’s run this:

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 3 items

test_expectation.py ..x                                              [100%]

======================= 2 passed, 1 xfailed in 0.12s =======================

The one parameter set which caused a failure previously now shows up as an “xfailed (expected to fail)” test.

In case the values provided to parametrize result in an empty list - for example, if they’re dynamically generated by some function - the behaviour of pytest is defined by the empty_parameter_set_mark option.

To get all combinations of multiple parametrized arguments you can stack parametrize decorators:

import pytest


@pytest.mark.parametrize("x", [0, 1])
@pytest.mark.parametrize("y", [2, 3])
def test_foo(x, y):
    pass

This will run the test with the arguments set to x=0/y=2, x=1/y=2, x=0/y=3, and x=1/y=3 exhausting parameters in the order of the decorators.

Basic `pytest_generate_tests` example¶

Sometimes you may want to implement your own parametrization scheme or implement some dynamism for determining the parameters or scope of a fixture. For this, you can use the pytest_generate_tests hook which is called when collecting a test function. Through the passed in metafunc object you can inspect the requesting test context and, most importantly, you can call metafunc.parametrize() to cause parametrization.

For example, let’s say we want to run a test taking string inputs which we want to set via a new pytest command line option. Let’s first write a simple test accepting a stringinput fixture function argument:

# content of test_strings.py


def test_valid_string(stringinput):
    assert stringinput.isalpha()

Now we add a conftest.py file containing the addition of a command line option and the parametrization of our test function:

# content of conftest.py


def pytest_addoption(parser):
    parser.addoption(
        "--stringinput",
        action="append",
        default=[],
        help="list of stringinputs to pass to test functions",
    )


def pytest_generate_tests(metafunc):
    if "stringinput" in metafunc.fixturenames:
        metafunc.parametrize("stringinput", metafunc.config.getoption("stringinput"))

If we now pass two stringinput values, our test will run twice:

$ pytest -q --stringinput="hello" --stringinput="world" test_strings.py
..                                                                   [100%]
2 passed in 0.12s

Let’s also run with a stringinput that will lead to a failing test:

$ pytest -q --stringinput="!" test_strings.py
F                                                                    [100%]
================================= FAILURES =================================
___________________________ test_valid_string[!] ___________________________

stringinput = '!'

    def test_valid_string(stringinput):
>       assert stringinput.isalpha()
E       AssertionError: assert False
E        +  where False = <built-in method isalpha of str object at 0xdeadbeef>()
E        +    where <built-in method isalpha of str object at 0xdeadbeef> = '!'.isalpha

test_strings.py:4: AssertionError
1 failed in 0.12s

As expected our test function fails.

If you don’t specify a stringinput it will be skipped because metafunc.parametrize() will be called with an empty parameter list:

$ pytest -q -rs test_strings.py
s                                                                    [100%]
========================= short test summary info ==========================
SKIPPED [1] test_strings.py: got empty parameter set ['stringinput'], function test_valid_string at $REGENDOC_TMPDIR/test_strings.py:2
1 skipped in 0.12s

Note that when calling metafunc.parametrize multiple times with different parameter sets, all parameter names across those sets cannot be duplicated, otherwise an error will be raised.

More examples¶

For further examples, you might want to look at more parametrization examples.

Cache: working with cross-testrun state¶

Usage¶

The plugin provides two command line options to rerun failures from the last pytest invocation:

--lf, --last-failed - to only re-run the failures.
--ff, --failed-first - to run the failures first and then the rest of the tests.

For cleanup (usually not needed), a --cache-clear option allows to remove all cross-session cache contents ahead of a test run.

Other plugins may access the config.cache object to set/get json encodable values between pytest invocations.

Note

This plugin is enabled by default, but can be disabled if needed: see Deactivating / unregistering a plugin by name (the internal name for this plugin is cacheprovider).

Rerunning only failures or failures first¶

First, let’s create 50 test invocation of which only 2 fail:

# content of test_50.py
import pytest


@pytest.mark.parametrize("i", range(50))
def test_num(i):
    if i in (17, 25):
        pytest.fail("bad luck")

If you run this for the first time you will see two failures:

$ pytest -q
.................F.......F........................                   [100%]
================================= FAILURES =================================
_______________________________ test_num[17] _______________________________

i = 17

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
_______________________________ test_num[25] _______________________________

i = 25

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
2 failed, 48 passed in 0.12s

If you then run it with --lf:

$ pytest --lf
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 50 items / 48 deselected / 2 selected
run-last-failure: rerun previous 2 failures

test_50.py FF                                                        [100%]

================================= FAILURES =================================
_______________________________ test_num[17] _______________________________

i = 17

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
_______________________________ test_num[25] _______________________________

i = 25

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
===================== 2 failed, 48 deselected in 0.12s =====================

You have run only the two failing tests from the last run, while the 48 passing tests have not been run (“deselected”).

Now, if you run with the --ff option, all tests will be run but the first previous failures will be executed first (as can be seen from the series of FF and dots):

$ pytest --ff
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 50 items
run-last-failure: rerun previous 2 failures first

test_50.py FF................................................        [100%]

================================= FAILURES =================================
_______________________________ test_num[17] _______________________________

i = 17

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
_______________________________ test_num[25] _______________________________

i = 25

    @pytest.mark.parametrize("i", range(50))
    def test_num(i):
        if i in (17, 25):
>           pytest.fail("bad luck")
E           Failed: bad luck

test_50.py:7: Failed
======================= 2 failed, 48 passed in 0.12s =======================

New --nf, --new-first options: run new tests first followed by the rest of the tests, in both cases tests are also sorted by the file modified time, with more recent files coming first.

Behavior when no tests failed in the last run¶

When no tests failed in the last run, or when no cached lastfailed data was found, pytest can be configured either to run all of the tests or no tests, using the --last-failed-no-failures option, which takes one of the following values:

pytest --last-failed --last-failed-no-failures all    # run all tests (default behavior)
pytest --last-failed --last-failed-no-failures none   # run no tests and exit

The new config.cache object¶

Plugins or conftest.py support code can get a cached value using the pytest config object. Here is a basic example plugin which implements a pytest fixtures: explicit, modular, scalable which re-uses previously created state across pytest invocations:

# content of test_caching.py
import pytest
import time


def expensive_computation():
    print("running expensive computation...")


@pytest.fixture
def mydata(request):
    val = request.config.cache.get("example/value", None)
    if val is None:
        expensive_computation()
        val = 42
        request.config.cache.set("example/value", val)
    return val


def test_function(mydata):
    assert mydata == 23

If you run this command for the first time, you can see the print statement:

$ pytest -q
F                                                                    [100%]
================================= FAILURES =================================
______________________________ test_function _______________________________

mydata = 42

    def test_function(mydata):
>       assert mydata == 23
E       assert 42 == 23

test_caching.py:20: AssertionError
-------------------------- Captured stdout setup ---------------------------
running expensive computation...
1 failed in 0.12s

If you run it a second time, the value will be retrieved from the cache and nothing will be printed:

$ pytest -q
F                                                                    [100%]
================================= FAILURES =================================
______________________________ test_function _______________________________

mydata = 42

    def test_function(mydata):
>       assert mydata == 23
E       assert 42 == 23

test_caching.py:20: AssertionError
1 failed in 0.12s

See the config.cache fixture for more details.

Inspecting Cache content¶

You can always peek at the content of the cache using the --cache-show command line option:

$ pytest --cache-show
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
cachedir: $PYTHON_PREFIX/.pytest_cache
--------------------------- cache values for '*' ---------------------------
cache/lastfailed contains:
  {'test_50.py::test_num[17]': True,
   'test_50.py::test_num[25]': True,
   'test_assert1.py::test_function': True,
   'test_assert2.py::test_set_comparison': True,
   'test_caching.py::test_function': True,
   'test_foocompare.py::test_compare': True}
cache/nodeids contains:
  ['test_assert1.py::test_function',
   'test_assert2.py::test_set_comparison',
   'test_foocompare.py::test_compare',
   'test_50.py::test_num[0]',
   'test_50.py::test_num[1]',
   'test_50.py::test_num[2]',
   'test_50.py::test_num[3]',
   'test_50.py::test_num[4]',
   'test_50.py::test_num[5]',
   'test_50.py::test_num[6]',
   'test_50.py::test_num[7]',
   'test_50.py::test_num[8]',
   'test_50.py::test_num[9]',
   'test_50.py::test_num[10]',
   'test_50.py::test_num[11]',
   'test_50.py::test_num[12]',
   'test_50.py::test_num[13]',
   'test_50.py::test_num[14]',
   'test_50.py::test_num[15]',
   'test_50.py::test_num[16]',
   'test_50.py::test_num[17]',
   'test_50.py::test_num[18]',
   'test_50.py::test_num[19]',
   'test_50.py::test_num[20]',
   'test_50.py::test_num[21]',
   'test_50.py::test_num[22]',
   'test_50.py::test_num[23]',
   'test_50.py::test_num[24]',
   'test_50.py::test_num[25]',
   'test_50.py::test_num[26]',
   'test_50.py::test_num[27]',
   'test_50.py::test_num[28]',
   'test_50.py::test_num[29]',
   'test_50.py::test_num[30]',
   'test_50.py::test_num[31]',
   'test_50.py::test_num[32]',
   'test_50.py::test_num[33]',
   'test_50.py::test_num[34]',
   'test_50.py::test_num[35]',
   'test_50.py::test_num[36]',
   'test_50.py::test_num[37]',
   'test_50.py::test_num[38]',
   'test_50.py::test_num[39]',
   'test_50.py::test_num[40]',
   'test_50.py::test_num[41]',
   'test_50.py::test_num[42]',
   'test_50.py::test_num[43]',
   'test_50.py::test_num[44]',
   'test_50.py::test_num[45]',
   'test_50.py::test_num[46]',
   'test_50.py::test_num[47]',
   'test_50.py::test_num[48]',
   'test_50.py::test_num[49]',
   'test_caching.py::test_function']
cache/stepwise contains:
  []
example/value contains:
  42

========================== no tests ran in 0.12s ===========================

--cache-show takes an optional argument to specify a glob pattern for filtering:

$ pytest --cache-show example/*
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
cachedir: $PYTHON_PREFIX/.pytest_cache
----------------------- cache values for 'example/*' -----------------------
example/value contains:
  42

========================== no tests ran in 0.12s ===========================

Clearing Cache content¶

You can instruct pytest to clear all cache files and values by adding the --cache-clear option like this:

pytest --cache-clear

This is recommended for invocations from Continuous Integration servers where isolation and correctness is more important than speed.

Stepwise¶

As an alternative to --lf -x, especially for cases where you expect a large part of the test suite will fail, --sw, --stepwise allows you to fix them one at a time. The test suite will run until the first failure and then stop. At the next invocation, tests will continue from the last failing test and then run until the next failing test. You may use the --stepwise-skip option to ignore one failing test and stop the test execution on the second failing test instead. This is useful if you get stuck on a failing test and just want to ignore it until later.

unittest.TestCase Support¶

pytest supports running Python unittest-based tests out of the box. It’s meant for leveraging existing unittest-based test suites to use pytest as a test runner and also allow to incrementally adapt the test suite to take full advantage of pytest’s features.

To run an existing unittest-style test suite using pytest, type:

pytest tests

pytest will automatically collect unittest.TestCase subclasses and their test methods in test_*.py or *_test.py files.

Almost all unittest features are supported:

@unittest.skip style decorators;
setUp/tearDown;
setUpClass/tearDownClass;
setUpModule/tearDownModule;

Up to this point pytest does not have support for the following features:

Benefits out of the box¶

By running your test suite with pytest you can make use of several features, in most cases without having to modify existing code:

Obtain more informative tracebacks;
stdout and stderr capturing;
Test selection options using -k and -m flags;
Stopping after the first (or N) failures;
–pdb command-line option for debugging on test failures (see note below);
Distribute tests to multiple CPUs using the pytest-xdist plugin;
Use plain assert-statements instead of self.assert* functions (unittest2pytest is immensely helpful in this);

pytest features in `unittest.TestCase` subclasses¶

The following pytest features work in unittest.TestCase subclasses:

Marks: skip, skipif, xfail;
Auto-use fixtures;

The following pytest features do not work, and probably never will due to different design philosophies:

Fixtures (except for autouse fixtures, see below);
Parametrization;
Custom hooks;

Third party plugins may or may not work well, depending on the plugin and the test suite.

Mixing pytest fixtures into `unittest.TestCase` subclasses using marks¶

Running your unittest with pytest allows you to use its fixture mechanism with unittest.TestCase style tests. Assuming you have at least skimmed the pytest fixture features, let’s jump-start into an example that integrates a pytest db_class fixture, setting up a class-cached database object, and then reference it from a unittest-style test:

# content of conftest.py

# we define a fixture function below and it will be "used" by
# referencing its name from tests

import pytest


@pytest.fixture(scope="class")
def db_class(request):
    class DummyDB:
        pass

    # set a class attribute on the invoking test context
    request.cls.db = DummyDB()

This defines a fixture function db_class which - if used - is called once for each test class and which sets the class-level db attribute to a DummyDB instance. The fixture function achieves this by receiving a special request object which gives access to the requesting test context such as the cls attribute, denoting the class from which the fixture is used. This architecture de-couples fixture writing from actual test code and allows re-use of the fixture by a minimal reference, the fixture name. So let’s write an actual unittest.TestCase class using our fixture definition:

# content of test_unittest_db.py

import unittest
import pytest


@pytest.mark.usefixtures("db_class")
class MyTest(unittest.TestCase):
    def test_method1(self):
        assert hasattr(self, "db")
        assert 0, self.db  # fail for demo purposes

    def test_method2(self):
        assert 0, self.db  # fail for demo purposes

The @pytest.mark.usefixtures("db_class") class-decorator makes sure that the pytest fixture function db_class is called once per class. Due to the deliberately failing assert statements, we can take a look at the self.db values in the traceback:

$ pytest test_unittest_db.py
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR
collected 2 items

test_unittest_db.py FF                                               [100%]

================================= FAILURES =================================
___________________________ MyTest.test_method1 ____________________________

self = <test_unittest_db.MyTest testMethod=test_method1>

    def test_method1(self):
        assert hasattr(self, "db")
>       assert 0, self.db  # fail for demo purposes
E       AssertionError: <conftest.db_class.<locals>.DummyDB object at 0xdeadbeef>
E       assert 0

test_unittest_db.py:10: AssertionError
___________________________ MyTest.test_method2 ____________________________

self = <test_unittest_db.MyTest testMethod=test_method2>

    def test_method2(self):
>       assert 0, self.db  # fail for demo purposes
E       AssertionError: <conftest.db_class.<locals>.DummyDB object at 0xdeadbeef>
E       assert 0

test_unittest_db.py:13: AssertionError
============================ 2 failed in 0.12s =============================

This default pytest traceback shows that the two test methods share the same self.db instance which was our intention when writing the class-scoped fixture function above.

Using autouse fixtures and accessing other fixtures¶

Although it’s usually better to explicitly declare use of fixtures you need for a given test, you may sometimes want to have fixtures that are automatically used in a given context. After all, the traditional style of unittest-setup mandates the use of this implicit fixture writing and chances are, you are used to it or like it.

You can flag fixture functions with @pytest.fixture(autouse=True) and define the fixture function in the context where you want it used. Let’s look at an initdir fixture which makes all test methods of a TestCase class execute in a temporary directory with a pre-initialized samplefile.ini. Our initdir fixture itself uses the pytest builtin tmpdir fixture to delegate the creation of a per-test temporary directory:

# content of test_unittest_cleandir.py
import pytest
import unittest


class MyTest(unittest.TestCase):
    @pytest.fixture(autouse=True)
    def initdir(self, tmpdir):
        tmpdir.chdir()  # change to pytest-provided temporary directory
        tmpdir.join("samplefile.ini").write("# testdata")

    def test_method(self):
        with open("samplefile.ini") as f:
            s = f.read()
        assert "testdata" in s

Due to the autouse flag the initdir fixture function will be used for all methods of the class where it is defined. This is a shortcut for using a @pytest.mark.usefixtures("initdir") marker on the class like in the previous example.

Running this test module …:

$ pytest -q test_unittest_cleandir.py
.                                                                    [100%]
1 passed in 0.12s

… gives us one passed test because the initdir fixture function was executed ahead of the test_method.

Note

unittest.TestCase methods cannot directly receive fixture arguments as implementing that is likely to inflict on the ability to run general unittest.TestCase test suites.

The above usefixtures and autouse examples should help to mix in pytest fixtures into unittest suites.

You can also gradually move away from subclassing from unittest.TestCase to plain asserts and then start to benefit from the full pytest feature set step by step.

Note

Due to architectural differences between the two frameworks, setup and teardown for unittest-based tests is performed during the call phase of testing instead of in pytest’s standard setup and teardown stages. This can be important to understand in some situations, particularly when reasoning about errors. For example, if a unittest-based suite exhibits errors during setup, pytest will report no errors during its setup phase and will instead raise the error during call.

Running tests written for nose¶

pytest has basic support for running tests written for nose.

Usage¶

After Install pytest type:

python setup.py develop  # make sure tests can import our package
pytest  # instead of 'nosetests'

and you should be able to run your nose style tests and make use of pytest’s capabilities.

Supported nose Idioms¶

setup and teardown at module/class/method level
SkipTest exceptions and markers
setup/teardown decorators
yield-based tests and their setup (considered deprecated as of pytest 3.0)
__test__ attribute on modules/classes/functions
general usage of nose utilities

Unsupported idioms / known issues¶

unittest-style setUp, tearDown, setUpClass, tearDownClass are recognized only on unittest.TestCase classes but not on plain classes. nose supports these methods also on plain classes but pytest deliberately does not. As nose and pytest already both support setup_class, teardown_class, setup_method, teardown_method it doesn’t seem useful to duplicate the unittest-API like nose does. If you however rather think pytest should support the unittest-spelling on plain classes please post to this issue.
nose imports test modules with the same import path (e.g. tests.test_mode) but different file system paths (e.g. tests/test_mode.py and other/tests/test_mode.py) by extending sys.path/import semantics. pytest does not do that but there is discussion in #268 for adding some support. Note that nose2 choose to avoid this sys.path/import hackery.

If you place a conftest.py file in the root directory of your project (as determined by pytest) pytest will run tests “nose style” against the code below that directory by adding it to your sys.path instead of running against your installed code.

You may find yourself wanting to do this if you ran python setup.py install to set up your project, as opposed to python setup.py develop or any of the package manager equivalents. Installing with develop in a virtual environment like tox is recommended over this pattern.
nose-style doctests are not collected and executed correctly, also doctest fixtures don’t work.
no nose-configuration is recognized.
yield-based methods don’t support setup properly because the setup method is always called in the same class instance. There are no plans to fix this currently because yield-tests are deprecated in pytest 3.0, with pytest.mark.parametrize being the recommended alternative.

classic xunit-style setup¶

This section describes a classic and popular way how you can implement fixtures (setup and teardown test state) on a per-module/class/function basis.

Note

While these setup/teardown methods are simple and familiar to those coming from a unittest or nose background, you may also consider using pytest’s more powerful fixture mechanism which leverages the concept of dependency injection, allowing for a more modular and more scalable approach for managing test state, especially for larger projects and for functional testing. You can mix both fixture mechanisms in the same file but test methods of unittest.TestCase subclasses cannot receive fixture arguments.

Module level setup/teardown¶

If you have multiple test functions and test classes in a single module you can optionally implement the following fixture methods which will usually be called once for all the functions:

def setup_module(module):
    """ setup any state specific to the execution of the given module."""


def teardown_module(module):
    """ teardown any state that was previously setup with a setup_module
    method.
    """

As of pytest-3.0, the module parameter is optional.

Class level setup/teardown¶

Similarly, the following methods are called at class level before and after all test methods of the class are called:

@classmethod
def setup_class(cls):
    """ setup any state specific to the execution of the given class (which
    usually contains tests).
    """


@classmethod
def teardown_class(cls):
    """ teardown any state that was previously setup with a call to
    setup_class.
    """

Method and function level setup/teardown¶

Similarly, the following methods are called around each method invocation:

def setup_method(self, method):
    """ setup any state tied to the execution of the given method in a
    class.  setup_method is invoked for every test method of a class.
    """


def teardown_method(self, method):
    """ teardown any state that was previously setup with a setup_method
    call.
    """

As of pytest-3.0, the method parameter is optional.

If you would rather define test functions directly at module level you can also use the following functions to implement fixtures:

def setup_function(function):
    """ setup any state tied to the execution of the given function.
    Invoked for every test function in the module.
    """


def teardown_function(function):
    """ teardown any state that was previously setup with a setup_function
    call.
    """

As of pytest-3.0, the function parameter is optional.

Remarks:

It is possible for setup/teardown pairs to be invoked multiple times per testing process.
teardown functions are not called if the corresponding setup function existed and failed/was skipped.
Prior to pytest-4.2, xunit-style functions did not obey the scope rules of fixtures, so it was possible, for example, for a setup_method to be called before a session-scoped autouse fixture.

Now the xunit-style functions are integrated with the fixture mechanism and obey the proper scope rules of fixtures involved in the call.

Installing and Using plugins¶

This section talks about installing and using third party plugins. For writing your own plugins, please refer to Writing plugins.

Installing a third party plugin can be easily done with pip:

pip install pytest-NAME
pip uninstall pytest-NAME

If a plugin is installed, pytest automatically finds and integrates it, there is no need to activate it.

Here is a little annotated list for some popular plugins:

pytest-django: write tests for django apps, using pytest integration.
pytest-twisted: write tests for twisted apps, starting a reactor and processing deferreds from test functions.
pytest-cov: coverage reporting, compatible with distributed testing
pytest-xdist: to distribute tests to CPUs and remote hosts, to run in boxed mode which allows to survive segmentation faults, to run in looponfailing mode, automatically re-running failing tests on file changes.
pytest-instafail: to report failures while the test run is happening.
pytest-bdd: to write tests using behaviour-driven testing.
pytest-timeout: to timeout tests based on function marks or global definitions.
pytest-pep8: a --pep8 option to enable PEP8 compliance checking.
pytest-flakes: check source code with pyflakes.
oejskit: a plugin to run javascript unittests in live browsers.

To see a complete list of all plugins with their latest testing status against different pytest and Python versions, please visit plugincompat.

You may also discover more plugins through a pytest- pypi.org search.

Requiring/Loading plugins in a test module or conftest file¶

You can require plugins in a test module or a conftest file like this:

pytest_plugins = ("myapp.testsupport.myplugin",)

When the test module or conftest plugin is loaded the specified plugins will be loaded as well.

Note

Requiring plugins using a pytest_plugins variable in non-root conftest.py files is deprecated. See full explanation in the Writing plugins section.

Note

The name pytest_plugins is reserved and should not be used as a name for a custom plugin module.

Finding out which plugins are active¶

If you want to find out which plugins are active in your environment you can type:

pytest --trace-config

and will get an extended test header which shows activated plugins and their names. It will also print local plugins aka conftest.py files when they are loaded.

Deactivating / unregistering a plugin by name¶

You can prevent plugins from loading or unregister them:

pytest -p no:NAME

This means that any subsequent try to activate/load the named plugin will not work.

If you want to unconditionally disable a plugin for a project, you can add this option to your pytest.ini file:

[pytest]
addopts = -p no:NAME

Alternatively to disable it only in certain environments (for example in a CI server), you can set PYTEST_ADDOPTS environment variable to -p no:name.

See Finding out which plugins are active for how to obtain the name of a plugin.

Writing plugins¶

It is easy to implement local conftest plugins for your own project or pip-installable plugins that can be used throughout many projects, including third party projects. Please refer to Installing and Using plugins if you only want to use but not write plugins.

A plugin contains one or multiple hook functions. Writing hooks explains the basics and details of how you can write a hook function yourself. pytest implements all aspects of configuration, collection, running and reporting by calling well specified hooks of the following plugins:

builtin plugins: loaded from pytest’s internal _pytest directory.
external plugins: modules discovered through setuptools entry points
conftest.py plugins: modules auto-discovered in test directories

In principle, each hook call is a 1:N Python function call where N is the number of registered implementation functions for a given specification. All specifications and implementations follow the pytest_ prefix naming convention, making them easy to distinguish and find.

Plugin discovery order at tool startup¶

pytest loads plugin modules at tool startup in the following way:

by loading all builtin plugins
by loading all plugins registered through setuptools entry points.
by pre-scanning the command line for the -p name option and loading the specified plugin before actual command line parsing.
by loading all conftest.py files as inferred by the command line invocation:
- if no test paths are specified use current dir as a test path
- if exists, load conftest.py and test*/conftest.py relative to the directory part of the first test path.
Note that pytest does not find conftest.py files in deeper nested sub directories at tool startup. It is usually a good idea to keep your conftest.py file in the top level test or project root directory.
by recursively loading all plugins specified by the pytest_plugins variable in conftest.py files

conftest.py: local per-directory plugins¶

Local conftest.py plugins contain directory-specific hook implementations. Hook Session and test running activities will invoke all hooks defined in conftest.py files closer to the root of the filesystem. Example of implementing the pytest_runtest_setup hook so that is called for tests in the a sub directory but not for other directories:

a/conftest.py:
    def pytest_runtest_setup(item):
        # called for running each test in 'a' directory
        print("setting up", item)

a/test_sub.py:
    def test_sub():
        pass

test_flat.py:
    def test_flat():
        pass

Here is how you might run it:

pytest test_flat.py --capture=no  # will not show "setting up"
pytest a/test_sub.py --capture=no  # will show "setting up"

Note

If you have conftest.py files which do not reside in a python package directory (i.e. one containing an __init__.py) then “import conftest” can be ambiguous because there might be other conftest.py files as well on your PYTHONPATH or sys.path. It is thus good practice for projects to either put conftest.py under a package scope or to never import anything from a conftest.py file.

Writing your own plugin¶

If you want to write a plugin, there are many real-life examples you can copy from:

a custom collection example plugin: A basic example for specifying tests in Yaml files
builtin plugins which provide pytest’s own functionality
many external plugins providing additional features

All of these plugins implement hooks and/or fixtures to extend and add functionality.

Note

Make sure to check out the excellent cookiecutter-pytest-plugin project, which is a cookiecutter template for authoring plugins.

The template provides an excellent starting point with a working plugin, tests running with tox, a comprehensive README file as well as a pre-configured entry-point.

Also consider contributing your plugin to pytest-dev once it has some happy users other than yourself.

Making your plugin installable by others¶

If you want to make your plugin externally available, you may define a so-called entry point for your distribution so that pytest finds your plugin module. Entry points are a feature that is provided by setuptools. pytest looks up the pytest11 entrypoint to discover its plugins and you can thus make your plugin available by defining it in your setuptools-invocation:

# sample ./setup.py file
from setuptools import setup

setup(
    name="myproject",
    packages=["myproject"],
    # the following makes a plugin available to pytest
    entry_points={"pytest11": ["name_of_plugin = myproject.pluginmodule"]},
    # custom PyPI classifier for pytest plugins
    classifiers=["Framework :: Pytest"],
)

If a package is installed this way, pytest will load myproject.pluginmodule as a plugin which can define hooks.

Note

Make sure to include Framework :: Pytest in your list of PyPI classifiers to make it easy for users to find your plugin.

Assertion Rewriting¶

One of the main features of pytest is the use of plain assert statements and the detailed introspection of expressions upon assertion failures. This is provided by “assertion rewriting” which modifies the parsed AST before it gets compiled to bytecode. This is done via a PEP 302 import hook which gets installed early on when pytest starts up and will perform this rewriting when modules get imported. However, since we do not want to test different bytecode from what you will run in production, this hook only rewrites test modules themselves (as defined by the python_files configuration option), and any modules which are part of plugins. Any other imported module will not be rewritten and normal assertion behaviour will happen.

If you have assertion helpers in other modules where you would need assertion rewriting to be enabled you need to ask pytest explicitly to rewrite this module before it gets imported.

register_assert_rewrite(*names) → None[source]

This function will make sure that this module or all modules inside the package will get their assert statements rewritten. Thus you should make sure to call this before the module is actually imported, usually in your __init__.py if you are a plugin using a package.

Raises:	TypeError – if the given module names are not strings.

This is especially important when you write a pytest plugin which is created using a package. The import hook only treats conftest.py files and any modules which are listed in the pytest11 entrypoint as plugins. As an example consider the following package:

pytest_foo/__init__.py
pytest_foo/plugin.py
pytest_foo/helper.py

With the following typical setup.py extract:

setup(..., entry_points={"pytest11": ["foo = pytest_foo.plugin"]}, ...)

In this case only pytest_foo/plugin.py will be rewritten. If the helper module also contains assert statements which need to be rewritten it needs to be marked as such, before it gets imported. This is easiest by marking it for rewriting inside the __init__.py module, which will always be imported first when a module inside a package is imported. This way plugin.py can still import helper.py normally. The contents of pytest_foo/__init__.py will then need to look like this:

import pytest

pytest.register_assert_rewrite("pytest_foo.helper")

Requiring/Loading plugins in a test module or conftest file¶

You can require plugins in a test module or a conftest.py file like this:

pytest_plugins = ["name1", "name2"]

When the test module or conftest plugin is loaded the specified plugins will be loaded as well. Any module can be blessed as a plugin, including internal application modules:

pytest_plugins = "myapp.testsupport.myplugin"

pytest_plugins variables are processed recursively, so note that in the example above if myapp.testsupport.myplugin also declares pytest_plugins, the contents of the variable will also be loaded as plugins, and so on.

Note

Requiring plugins using a pytest_plugins variable in non-root conftest.py files is deprecated.

This is important because conftest.py files implement per-directory hook implementations, but once a plugin is imported, it will affect the entire directory tree. In order to avoid confusion, defining pytest_plugins in any conftest.py file which is not located in the tests root directory is deprecated, and will raise a warning.

This mechanism makes it easy to share fixtures within applications or even external applications without the need to create external plugins using the setuptools’s entry point technique.

Plugins imported by pytest_plugins will also automatically be marked for assertion rewriting (see pytest.register_assert_rewrite()). However for this to have any effect the module must not be imported already; if it was already imported at the time the pytest_plugins statement is processed, a warning will result and assertions inside the plugin will not be rewritten. To fix this you can either call pytest.register_assert_rewrite() yourself before the module is imported, or you can arrange the code to delay the importing until after the plugin is registered.

Accessing another plugin by name¶

If a plugin wants to collaborate with code from another plugin it can obtain a reference through the plugin manager like this:

plugin = config.pluginmanager.get_plugin("name_of_plugin")

If you want to look at the names of existing plugins, use the --trace-config option.

Registering custom markers¶

If your plugin uses any markers, you should register them so that they appear in pytest’s help text and do not cause spurious warnings. For example, the following plugin would register cool_marker and mark_with for all users:

def pytest_configure(config):
    config.addinivalue_line("markers", "cool_marker: this one is for cool tests.")
    config.addinivalue_line(
        "markers", "mark_with(arg, arg2): this marker takes arguments."
    )

Testing plugins¶

pytest comes with a plugin named pytester that helps you write tests for your plugin code. The plugin is disabled by default, so you will have to enable it before you can use it.

You can do so by adding the following line to a conftest.py file in your testing directory:

# content of conftest.py

pytest_plugins = ["pytester"]

Alternatively you can invoke pytest with the -p pytester command line option.

This will allow you to use the testdir fixture for testing your plugin code.

Let’s demonstrate what you can do with the plugin with an example. Imagine we developed a plugin that provides a fixture hello which yields a function and we can invoke this function with one optional parameter. It will return a string value of Hello World! if we do not supply a value or Hello {value}! if we do supply a string value.

import pytest


def pytest_addoption(parser):
    group = parser.getgroup("helloworld")
    group.addoption(
        "--name",
        action="store",
        dest="name",
        default="World",
        help='Default "name" for hello().',
    )


@pytest.fixture
def hello(request):
    name = request.config.getoption("name")

    def _hello(name=None):
        if not name:
            name = request.config.getoption("name")
        return "Hello {name}!".format(name=name)

    return _hello

Now the testdir fixture provides a convenient API for creating temporary conftest.py files and test files. It also allows us to run the tests and return a result object, with which we can assert the tests’ outcomes.

def test_hello(testdir):
    """Make sure that our plugin works."""

    # create a temporary conftest.py file
    testdir.makeconftest(
        """
        import pytest

        @pytest.fixture(params=[
            "Brianna",
            "Andreas",
            "Floris",
        ])
        def name(request):
            return request.param
    """
    )

    # create a temporary pytest test file
    testdir.makepyfile(
        """
        def test_hello_default(hello):
            assert hello() == "Hello World!"

        def test_hello_name(hello, name):
            assert hello(name) == "Hello {0}!".format(name)
    """
    )

    # run all tests with pytest
    result = testdir.runpytest()

    # check that all 4 tests passed
    result.assert_outcomes(passed=4)

additionally it is possible to copy examples for an example folder before running pytest on it

# content of pytest.ini
[pytest]
pytester_example_dir = .

# content of test_example.py


def test_plugin(testdir):
    testdir.copy_example("test_example.py")
    testdir.runpytest("-k", "test_example")


def test_example():
    pass

$ pytest
=========================== test session starts ============================
platform linux -- Python 3.x.y, pytest-5.x.y, py-1.x.y, pluggy-0.x.y
cachedir: $PYTHON_PREFIX/.pytest_cache
rootdir: $REGENDOC_TMPDIR, inifile: pytest.ini
collected 2 items

test_example.py ..                                                   [100%]

============================= warnings summary =============================
test_example.py::test_plugin
  $REGENDOC_TMPDIR/test_example.py:4: PytestExperimentalApiWarning: testdir.copy_example is an experimental api that may change over time
    testdir.copy_example("test_example.py")

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================= 2 passed, 1 warning in 0.12s =======================

For more information about the result object that runpytest() returns, and the methods that it provides please check out the RunResult documentation.

Writing hook functions¶

hook function validation and execution¶

pytest calls hook functions from registered plugins for any given hook specification. Let’s look at a typical hook function for the pytest_collection_modifyitems(session, config, items) hook which pytest calls after collection of all test items is completed.

When we implement a pytest_collection_modifyitems function in our plugin pytest will during registration verify that you use argument names which match the specification and bail out if not.

Let’s look at a possible implementation:

def pytest_collection_modifyitems(config, items):
    # called after collection is completed
    # you can modify the ``items`` list
    ...

Here, pytest will pass in config (the pytest config object) and items (the list of collected test items) but will not pass in the session argument because we didn’t list it in the function signature. This dynamic “pruning” of arguments allows pytest to be “future-compatible”: we can introduce new hook named parameters without breaking the signatures of existing hook implementations. It is one of the reasons for the general long-lived compatibility of pytest plugins.

Note that hook functions other than pytest_runtest_* are not allowed to raise exceptions. Doing so will break the pytest run.

firstresult: stop at first non-None result¶

Most calls to pytest hooks result in a list of results which contains all non-None results of the called hook functions.

Some hook specifications use the firstresult=True option so that the hook call only executes until the first of N registered functions returns a non-None result which is then taken as result of the overall hook call. The remaining hook functions will not be called in this case.

hookwrapper: executing around other hooks¶

pytest plugins can implement hook wrappers which wrap the execution of other hook implementations. A hook wrapper is a generator function which yields exactly once. When pytest invokes hooks it first executes hook wrappers and passes the same arguments as to the regular hooks.

At the yield point of the hook wrapper pytest will execute the next hook implementations and return their result to the yield point in the form of a Result instance which encapsulates a result or exception info. The yield point itself will thus typically not raise exceptions (unless there are bugs).

Here is an example definition of a hook wrapper:

import pytest


@pytest.hookimpl(hookwrapper=True)
def pytest_pyfunc_call(pyfuncitem):
    do_something_before_next_hook_executes()

    outcome = yield
    # outcome.excinfo may be None or a (cls, val, tb) tuple

    res = outcome.get_result()  # will raise if outcome was exception

    post_process_result(res)

    outcome.force_result(new_res)  # to override the return value to the plugin system

Note that hook wrappers don’t return results themselves, they merely perform tracing or other side effects around the actual hook implementations. If the result of the underlying hook is a mutable object, they may modify that result but it’s probably better to avoid it.

For more information, consult the pluggy documentation.

Hook function ordering / call example¶

For any given hook specification there may be more than one implementation and we thus generally view hook execution as a 1:N function call where N is the number of registered functions. There are ways to influence if a hook implementation comes before or after others, i.e. the position in the N-sized list of functions:

# Plugin 1
@pytest.hookimpl(tryfirst=True)
def pytest_collection_modifyitems(items):
    # will execute as early as possible
    ...


# Plugin 2
@pytest.hookimpl(trylast=True)
def pytest_collection_modifyitems(items):
    # will execute as late as possible
    ...


# Plugin 3
@pytest.hookimpl(hookwrapper=True)
def pytest_collection_modifyitems(items):
    # will execute even before the tryfirst one above!
    outcome = yield
    # will execute after all non-hookwrappers executed

Here is the order of execution:

Plugin3’s pytest_collection_modifyitems called until the yield point because it is a hook wrapper.
Plugin1’s pytest_collection_modifyitems is called because it is marked with tryfirst=True.
Plugin2’s pytest_collection_modifyitems is called because it is marked with trylast=True (but even without this mark it would come after Plugin1).
Plugin3’s pytest_collection_modifyitems then executing the code after the yield point. The yield receives a Result instance which encapsulates the result from calling the non-wrappers. Wrappers shall not modify the result.

It’s possible to use tryfirst and trylast also in conjunction with hookwrapper=True in which case it will influence the ordering of hookwrappers among each other.

Declaring new hooks¶

Plugins and conftest.py files may declare new hooks that can then be implemented by other plugins in order to alter behaviour or interact with the new plugin:

pytest_addhooks(pluginmanager)[source]

called at plugin registration time to allow adding new hooks via a call to pluginmanager.add_hookspecs(module_or_class, prefix).

Parameters:	pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager

Note

This hook is incompatible with hookwrapper=True.

Hooks are usually declared as do-nothing functions that contain only documentation describing when the hook will be called and what return values are expected. The names of the functions must start with pytest_ otherwise pytest won’t recognize them.

Here’s an example. Let’s assume this code is in the hooks.py module.

def pytest_my_hook(config):
    """
    Receives the pytest config and does things with it
    """

To register the hooks with pytest they need to be structured in their own module or class. This class or module can then be passed to the pluginmanager using the pytest_addhooks function (which itself is a hook exposed by pytest).

def pytest_addhooks(pluginmanager):
    """ This example assumes the hooks are grouped in the 'hooks' module. """
    from my_app.tests import hooks

    pluginmanager.add_hookspecs(hooks)

For a real world example, see newhooks.py from xdist.

Hooks may be called both from fixtures or from other hooks. In both cases, hooks are called through the hook object, available in the config object. Most hooks receive a config object directly, while fixtures may use the pytestconfig fixture which provides the same object.

@pytest.fixture()
def my_fixture(pytestconfig):
    # call the hook called "pytest_my_hook"
    # 'result' will be a list of return values from all registered functions.
    result = pytestconfig.hook.pytest_my_hook(config=pytestconfig)

Note

Hooks receive parameters using only keyword arguments.

Now your hook is ready to be used. To register a function at the hook, other plugins or users must now simply define the function pytest_my_hook with the correct signature in their conftest.py.

Example:

def pytest_my_hook(config):
    """
    Print all active hooks to the screen.
    """
    print(config.hook)

Using hooks in pytest_addoption¶

Occasionally, it is necessary to change the way in which command line options are defined by one plugin based on hooks in another plugin. For example, a plugin may expose a command line option for which another plugin needs to define the default value. The pluginmanager can be used to install and use hooks to accomplish this. The plugin would define and add the hooks and use pytest_addoption as follows:

# contents of hooks.py

# Use firstresult=True because we only want one plugin to define this
# default value
@hookspec(firstresult=True)
def pytest_config_file_default_value():
    """ Return the default value for the config file command line option. """


# contents of myplugin.py


def pytest_addhooks(pluginmanager):
    """ This example assumes the hooks are grouped in the 'hooks' module. """
    from . import hook

    pluginmanager.add_hookspecs(hook)


def pytest_addoption(parser, pluginmanager):
    default_value = pluginmanager.hook.pytest_config_file_default_value()
    parser.addoption(
        "--config-file",
        help="Config file to use, defaults to %(default)s",
        default=default_value,
    )

The conftest.py that is using myplugin would simply define the hook as follows:

def pytest_config_file_default_value():
    return "config.yaml"

Optionally using hooks from 3rd party plugins¶

Using new hooks from plugins as explained above might be a little tricky because of the standard validation mechanism: if you depend on a plugin that is not installed, validation will fail and the error message will not make much sense to your users.

One approach is to defer the hook implementation to a new plugin instead of declaring the hook functions directly in your plugin module, for example:

# contents of myplugin.py


class DeferPlugin:
    """Simple plugin to defer pytest-xdist hook functions."""

    def pytest_testnodedown(self, node, error):
        """standard xdist hook function.
        """


def pytest_configure(config):
    if config.pluginmanager.hasplugin("xdist"):
        config.pluginmanager.register(DeferPlugin())

This has the added benefit of allowing you to conditionally install hooks depending on which plugins are installed.

Logging¶

pytest captures log messages of level WARNING or above automatically and displays them in their own section for each failed test in the same manner as captured stdout and stderr.

Running without options:

pytest

Shows failed tests like so:

----------------------- Captured stdlog call ----------------------
test_reporting.py    26 WARNING  text going to logger
----------------------- Captured stdout call ----------------------
text going to stdout
----------------------- Captured stderr call ----------------------
text going to stderr
==================== 2 failed in 0.02 seconds =====================

By default each captured log message shows the module, line number, log level and message.

If desired the log and date format can be specified to anything that the logging module supports by passing specific formatting options:

pytest --log-format="%(asctime)s %(levelname)s %(message)s" \
        --log-date-format="%Y-%m-%d %H:%M:%S"

Shows failed tests like so:

----------------------- Captured stdlog call ----------------------
2010-04-10 14:48:44 WARNING text going to logger
----------------------- Captured stdout call ----------------------
text going to stdout
----------------------- Captured stderr call ----------------------
text going to stderr
==================== 2 failed in 0.02 seconds =====================

These options can also be customized through pytest.ini file:

[pytest]
log_format = %(asctime)s %(levelname)s %(message)s
log_date_format = %Y-%m-%d %H:%M:%S

Further it is possible to disable reporting of captured content (stdout, stderr and logs) on failed tests completely with:

pytest --show-capture=no

caplog fixture¶

Inside tests it is possible to change the log level for the captured log messages. This is supported by the caplog fixture:

def test_foo(caplog):
    caplog.set_level(logging.INFO)
    pass

By default the level is set on the root logger, however as a convenience it is also possible to set the log level of any logger:

def test_foo(caplog):
    caplog.set_level(logging.CRITICAL, logger="root.baz")
    pass

The log levels set are restored automatically at the end of the test.

It is also possible to use a context manager to temporarily change the log level inside a with block:

def test_bar(caplog):
    with caplog.at_level(logging.INFO):
        pass

Again, by default the level of the root logger is affected but the level of any logger can be changed instead with:

def test_bar(caplog):
    with caplog.at_level(logging.CRITICAL, logger="root.baz"):
        pass

Lastly all the logs sent to the logger during the test run are made available on the fixture in the form of both the logging.LogRecord instances and the final log text. This is useful for when you want to assert on the contents of a message:

def test_baz(caplog):
    func_under_test()
    for record in caplog.records:
        assert record.levelname != "CRITICAL"
    assert "wally" not in caplog.text

For all the available attributes of the log records see the logging.LogRecord class.

You can also resort to record_tuples if all you want to do is to ensure, that certain messages have been logged under a given logger name with a given severity and message:

def test_foo(caplog):
    logging.getLogger().info("boo %s", "arg")

    assert caplog.record_tuples == [("root", logging.INFO, "boo arg")]

You can call caplog.clear() to reset the captured log records in a test:

def test_something_with_clearing_records(caplog):
    some_method_that_creates_log_records()
    caplog.clear()
    your_test_method()
    assert ["Foo"] == [rec.message for rec in caplog.records]

The caplog.records attribute contains records from the current stage only, so inside the setup phase it contains only setup logs, same with the call and teardown phases.

To access logs from other stages, use the caplog.get_records(when) method. As an example, if you want to make sure that tests which use a certain fixture never log any warnings, you can inspect the records for the setup and call stages during teardown like so:

@pytest.fixture
def window(caplog):
    window = create_window()
    yield window
    for when in ("setup", "call"):
        messages = [
            x.message for x in caplog.get_records(when) if x.levelno == logging.WARNING
        ]
        if messages:
            pytest.fail(
                "warning messages encountered during testing: {}".format(messages)
            )

The full API is available at _pytest.logging.LogCaptureFixture.

Live Logs¶

By setting the log_cli configuration option to true, pytest will output logging records as they are emitted directly into the console.

You can specify the logging level for which log records with equal or higher level are printed to the console by passing --log-cli-level. This setting accepts the logging level names as seen in python’s documentation or an integer as the logging level num.

Additionally, you can also specify --log-cli-format and --log-cli-date-format which mirror and default to --log-format and --log-date-format if not provided, but are applied only to the console logging handler.

All of the CLI log options can also be set in the configuration INI file. The option names are:

log_cli_level
log_cli_format
log_cli_date_format

If you need to record the whole test suite logging calls to a file, you can pass --log-file=/path/to/log/file. This log file is opened in write mode which means that it will be overwritten at each run tests session.

You can also specify the logging level for the log file by passing --log-file-level. This setting accepts the logging level names as seen in python’s documentation(ie, uppercased level names) or an integer as the logging level num.

Additionally, you can also specify --log-file-format and --log-file-date-format which are equal to --log-format and --log-date-format but are applied to the log file logging handler.

All of the log file options can also be set in the configuration INI file. The option names are:

log_file
log_file_level
log_file_format
log_file_date_format

You can call set_log_path() to customize the log_file path dynamically. This functionality is considered experimental.

Release notes¶

This feature was introduced as a drop-in replacement for the pytest-catchlog plugin and they conflict with each other. The backward compatibility API with pytest-capturelog has been dropped when this feature was introduced, so if for that reason you still need pytest-catchlog you can disable the internal feature by adding to your pytest.ini:

[pytest]
    addopts=-p no:logging

Incompatible changes in pytest 3.4¶

This feature was introduced in 3.3 and some incompatible changes have been made in 3.4 after community feedback:

Log levels are no longer changed unless explicitly requested by the log_level configuration or --log-level command-line options. This allows users to configure logger objects themselves.
Live Logs is now disabled by default and can be enabled setting the log_cli configuration option to true. When enabled, the verbosity is increased so logging for each test is visible.
Live Logs are now sent to sys.stdout and no longer require the -s command-line option to work.

If you want to partially restore the logging behavior of version 3.3, you can add this options to your ini file:

[pytest]
log_cli=true
log_level=NOTSET

More details about the discussion that lead to this changes can be read in issue #3013.

API Reference¶

This page contains the full reference to pytest’s API.

Functions
Marks
Fixtures
Hooks
Objects
- CallInfo
- Class
- Collector
- Config
- ExceptionInfo
- pytest.ExitCode
- FixtureDef
- FSCollector
- Function
- Item
- MarkDecorator
- MarkGenerator
- Mark
- Metafunc
- Module
- Node
- Parser
- PluginManager
- PytestPluginManager
- Session
- TestReport
- _Result
Special Variables
Environment Variables
Exceptions
- UsageError
Configuration Options

Functions ¶

pytest.approx ¶

approx(expected, rel=None, abs=None, nan_ok=False)[source]¶

Assert that two numbers (or two sets of numbers) are equal to each other within some tolerance.

Due to the intricacies of floating-point arithmetic, numbers that we would intuitively expect to be equal are not always so:

>>> 0.1 + 0.2 == 0.3
False

This problem is commonly encountered when writing tests, e.g. when making sure that floating-point values are what you expect them to be. One way to deal with this problem is to assert that two floating-point numbers are equal to within some appropriate tolerance:

>>> abs((0.1 + 0.2) - 0.3) < 1e-6
True

However, comparisons like this are tedious to write and difficult to understand. Furthermore, absolute comparisons like the one above are usually discouraged because there’s no tolerance that works well for all situations. 1e-6 is good for numbers around 1, but too small for very big numbers and too big for very small ones. It’s better to express the tolerance as a fraction of the expected value, but relative comparisons like that are even more difficult to write correctly and concisely.

The approx class performs floating-point comparisons using a syntax that’s as intuitive as possible:

>>> from pytest import approx
>>> 0.1 + 0.2 == approx(0.3)
True

The same syntax also works for sequences of numbers:

>>> (0.1 + 0.2, 0.2 + 0.4) == approx((0.3, 0.6))
True

Dictionary values:

>>> {'a': 0.1 + 0.2, 'b': 0.2 + 0.4} == approx({'a': 0.3, 'b': 0.6})
True

numpy arrays:

>>> import numpy as np                                                          
>>> np.array([0.1, 0.2]) + np.array([0.2, 0.4]) == approx(np.array([0.3, 0.6])) 
True

And for a numpy array against a scalar:

>>> import numpy as np                                         
>>> np.array([0.1, 0.2]) + np.array([0.2, 0.1]) == approx(0.3) 
True

By default, approx considers numbers within a relative tolerance of 1e-6 (i.e. one part in a million) of its expected value to be equal. This treatment would lead to surprising results if the expected value was 0.0, because nothing but 0.0 itself is relatively close to 0.0. To handle this case less surprisingly, approx also considers numbers within an absolute tolerance of 1e-12 of its expected value to be equal. Infinity and NaN are special cases. Infinity is only considered equal to itself, regardless of the relative tolerance. NaN is not considered equal to anything by default, but you can make it be equal to itself by setting the nan_ok argument to True. (This is meant to facilitate comparing arrays that use NaN to mean “no data”.)

Both the relative and absolute tolerances can be changed by passing arguments to the approx constructor:

>>> 1.0001 == approx(1)
False
>>> 1.0001 == approx(1, rel=1e-3)
True
>>> 1.0001 == approx(1, abs=1e-3)
True

If you specify abs but not rel, the comparison will not consider the relative tolerance at all. In other words, two numbers that are within the default relative tolerance of 1e-6 will still be considered unequal if they exceed the specified absolute tolerance. If you specify both abs and rel, the numbers will be considered equal if either tolerance is met:

>>> 1 + 1e-8 == approx(1)
True
>>> 1 + 1e-8 == approx(1, abs=1e-12)
False
>>> 1 + 1e-8 == approx(1, rel=1e-6, abs=1e-12)
True

If you’re thinking about using approx, then you might want to know how it compares to other good ways of comparing floating-point numbers. All of these algorithms are based on relative and absolute tolerances and should agree for the most part, but they do have meaningful differences:

math.isclose(a, b, rel_tol=1e-9, abs_tol=0.0): True if the relative tolerance is met w.r.t. either a or b or if the absolute tolerance is met. Because the relative tolerance is calculated w.r.t. both a and b, this test is symmetric (i.e. neither a nor b is a “reference value”). You have to specify an absolute tolerance if you want to compare to 0.0 because there is no tolerance by default. Only available in python>=3.5. More information…
numpy.isclose(a, b, rtol=1e-5, atol=1e-8): True if the difference between a and b is less that the sum of the relative tolerance w.r.t. b and the absolute tolerance. Because the relative tolerance is only calculated w.r.t. b, this test is asymmetric and you can think of b as the reference value. Support for comparing sequences is provided by numpy.allclose. More information…
unittest.TestCase.assertAlmostEqual(a, b): True if a and b are within an absolute tolerance of 1e-7. No relative tolerance is considered and the absolute tolerance cannot be changed, so this function is not appropriate for very large or very small numbers. Also, it’s only available in subclasses of unittest.TestCase and it’s ugly because it doesn’t follow PEP8. More information…
a == pytest.approx(b, rel=1e-6, abs=1e-12): True if the relative tolerance is met w.r.t. b or if the absolute tolerance is met. Because the relative tolerance is only calculated w.r.t. b, this test is asymmetric and you can think of b as the reference value. In the special case that you explicitly specify an absolute tolerance but not a relative tolerance, only the absolute tolerance is considered.

Warning

Changed in version 3.2.

In order to avoid inconsistent behavior, TypeError is raised for >, >=, < and <= comparisons. The example below illustrates the problem:

assert approx(0.1) > 0.1 + 1e-10  # calls approx(0.1).__gt__(0.1 + 1e-10)
assert 0.1 + 1e-10 > approx(0.1)  # calls approx(0.1).__lt__(0.1 + 1e-10)

In the second example one expects approx(0.1).__le__(0.1 + 1e-10) to be called. But instead, approx(0.1).__lt__(0.1 + 1e-10) is used to comparison. This is because the call hierarchy of rich comparisons follows a fixed behavior. More information…

pytest.fail ¶

Tutorial: Skip and xfail: dealing with tests that cannot succeed

fail(msg: str = '', pytrace: bool = True) → NoReturn[source]¶

Explicitly fail an executing test with the given message.

Parameters:	msg (str) – the message to show the user as reason for the failure. pytrace (bool) – if false the msg represents the full failure information and no python traceback will be reported.

pytest.skip ¶

skip(msg[, allow_module_level=False])[source]¶

Skip an executing test with the given message.

This function should be called only during testing (setup, call or teardown) or during collection by using the allow_module_level flag. This function can be called in doctests as well.

Parameters:	allow_module_level (bool) – allows this function to be called at module level, skipping the rest of the module. Default to False.

Note

It is better to use the pytest.mark.skipif marker when possible to declare a test to be skipped under certain conditions like mismatching platforms or dependencies. Similarly, use the # doctest: +SKIP directive (see doctest.SKIP) to skip a doctest statically.

pytest.importorskip ¶

importorskip(modname: str, minversion: Optional[str] = None, reason: Optional[str] = None) → Any[source]¶

Imports and returns the requested module modname, or skip the current test if the module cannot be imported.

Parameters:	modname (str) – the name of the module to import minversion (str) – if given, the imported module’s `__version__` attribute must be at least this minimal version, otherwise the test is still skipped. reason (str) – if given, this reason is shown as the message when the module cannot be imported.
Returns:	The imported module. This should be assigned to its canonical name.

Example:

docutils = pytest.importorskip("docutils")

pytest.xfail ¶

xfail(reason: str = '') → NoReturn[source]¶

Imperatively xfail an executing test or setup functions with the given reason.

This function should be called only during testing (setup, call or teardown).

Note

It is better to use the pytest.mark.xfail marker when possible to declare a test to be xfailed under certain conditions like known bugs or missing features.

pytest.exit ¶

exit(msg: str, returncode: Optional[int] = None) → NoReturn[source]¶

Exit testing process.

Parameters:	msg (str) – message to display upon exit. returncode (int) – return code to be used when exiting pytest.

pytest.main ¶

main(args=None, plugins=None) → Union[int, _pytest.config.ExitCode][source]¶

return exit code, after performing an in-process test run.

Parameters:	args – list of command line arguments. plugins – list of plugin objects to be auto-registered during initialization.

pytest.param ¶

param(*values[, id][, marks])[source]¶

Specify a parameter in pytest.mark.parametrize calls or parametrized fixtures.

@pytest.mark.parametrize("test_input,expected", [
    ("3+5", 8),
    pytest.param("6*9", 42, marks=pytest.mark.xfail),
])
def test_eval(test_input, expected):
    assert eval(test_input) == expected

Parameters:	values – variable args of the values of the parameter set, in order. marks – a single mark or a list of marks to be applied to this parameter set. id (str) – the id to attribute to this parameter set.

pytest.raises ¶

Tutorial: Assertions about expected exceptions.

with raises(expected_exception: Exception[, *, match]) as excinfo[source]¶

Assert that a code block/function call raises expected_exception or raise a failure exception otherwise.

Parameters:

Parameters:	match – if specified, a string containing a regular expression, or a regular expression object, that is tested against the string representation of the exception using `re.search`. To match a literal string that may contain special characters, the pattern can first be escaped with `re.escape`. (This is only used when `pytest.raises` is used as a context manager, and passed through to the function otherwise. When using `pytest.raises` as a function, you can use: `pytest.raises(Exc, func, match="passed on").match("my pattern")`.)

match –

if specified, a string containing a regular expression, or a regular expression object, that is tested against the string representation of the exception using re.search. To match a literal string that may contain special characters, the pattern can first be escaped with re.escape.

(This is only used when pytest.raises is used as a context manager, and passed through to the function otherwise. When using pytest.raises as a function, you can use: pytest.raises(Exc, func, match="passed on").match("my pattern").)

Use pytest.raises as a context manager, which will capture the exception of the given type:

>>> with raises(ZeroDivisionError):
...    1/0

If the code block does not raise the expected exception (ZeroDivisionError in the example above), or no exception at all, the check will fail instead.

You can also use the keyword argument match to assert that the exception matches a text or regex:

>>> with raises(ValueError, match='must be 0 or None'):
...     raise ValueError("value must be 0 or None")

>>> with raises(ValueError, match=r'must be \d+$'):
...     raise ValueError("value must be 42")

The context manager produces an ExceptionInfo object which can be used to inspect the details of the captured exception:

>>> with raises(ValueError) as exc_info:
...     raise ValueError("value must be 42")
>>> assert exc_info.type is ValueError
>>> assert exc_info.value.args[0] == "value must be 42"

Note

When using pytest.raises as a context manager, it’s worthwhile to note that normal context manager rules apply and that the exception raised must be the final line in the scope of the context manager. Lines of code after that, within the scope of the context manager will not be executed. For example:

>>> value = 15
>>> with raises(ValueError) as exc_info:
...     if value > 10:
...         raise ValueError("value must be <= 10")
...     assert exc_info.type is ValueError  # this will not execute

Instead, the following approach must be taken (note the difference in scope):

>>> with raises(ValueError) as exc_info:
...     if value > 10:
...         raise ValueError("value must be <= 10")
...
>>> assert exc_info.type is ValueError

Using with pytest.mark.parametrize

When using pytest.mark.parametrize it is possible to parametrize tests such that some runs raise an exception and others do not.

See Parametrizing conditional raising for an example.

Legacy form

It is possible to specify a callable by passing a to-be-called lambda:

>>> raises(ZeroDivisionError, lambda: 1/0)
<ExceptionInfo ...>

or you can specify an arbitrary callable with arguments:

>>> def f(x): return 1/x
...
>>> raises(ZeroDivisionError, f, 0)
<ExceptionInfo ...>
>>> raises(ZeroDivisionError, f, x=0)
<ExceptionInfo ...>

The form above is fully supported but discouraged for new code because the context manager form is regarded as more readable and less error-prone.

Note

Similar to caught exception objects in Python, explicitly clearing local references to returned ExceptionInfo objects can help the Python interpreter speed up its garbage collection.

Clearing those references breaks a reference cycle (ExceptionInfo –> caught exception –> frame stack raising the exception –> current frame stack –> local variables –> ExceptionInfo) which makes Python keep all objects referenced from that cycle (including all local variables in the current frame) alive until the next cyclic garbage collection run. More detailed information can be found in the official Python documentation for the try statement.

pytest.deprecated_call ¶

Tutorial: Ensuring code triggers a deprecation warning.

with deprecated_call()[source]¶

context manager that can be used to ensure a block of code triggers a DeprecationWarning or PendingDeprecationWarning:

>>> import warnings
>>> def api_call_v2():
...     warnings.warn('use v3 of this api', DeprecationWarning)
...     return 200

>>> with deprecated_call():
...    assert api_call_v2() == 200

deprecated_call can also be used by passing a function and *args and *kwargs, in which case it will ensure calling func(*args, **kwargs) produces one of the warnings types above.

pytest.register_assert_rewrite ¶

Tutorial: Assertion Rewriting.

register_assert_rewrite(*names) → None[source]¶

Raises:	TypeError – if the given module names are not strings.

pytest.warns ¶

Tutorial: Asserting warnings with the warns function

with warns(expected_warning: Exception[, match])[source]¶

Assert that code raises a particular class of warning.

Specifically, the parameter expected_warning can be a warning class or sequence of warning classes, and the inside the with block must issue a warning of that class or classes.

This helper produces a list of warnings.WarningMessage objects, one for each warning raised.

This function can be used as a context manager, or any of the other ways pytest.raises can be used:

>>> with warns(RuntimeWarning):
...    warnings.warn("my warning", RuntimeWarning)

In the context manager form you may use the keyword argument match to assert that the exception matches a text or regex:

>>> with warns(UserWarning, match='must be 0 or None'):
...     warnings.warn("value must be 0 or None", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("value must be 42", UserWarning)

>>> with warns(UserWarning, match=r'must be \d+$'):
...     warnings.warn("this is not here", UserWarning)
Traceback (most recent call last):
  ...
Failed: DID NOT WARN. No warnings of type ...UserWarning... was emitted...

pytest.freeze_includes ¶

Tutorial: Freezing pytest.

freeze_includes()[source]¶: Returns a list of module names used by pytest that should be included by cx_freeze.

Marks ¶

Marks can be used apply meta data to test functions (but not fixtures), which can then be accessed by fixtures or plugins.

pytest.mark.filterwarnings ¶

Tutorial: @pytest.mark.filterwarnings.

Add warning filters to marked test items.

pytest.mark.filterwarnings(filter)¶

Parameters:

Parameters:	filter (str) – A warning specification string, which is composed of contents of the tuple `(action, message, category, module, lineno)` as specified in The Warnings filter section of the Python documentation, separated by `":"`. Optional fields can be omitted. Module names passed for filtering are not regex-escaped. For example: @pytest.mark.warnings("ignore:.usage will be deprecated.:DeprecationWarning") def test_foo(): ...

filter (str) –

A warning specification string, which is composed of contents of the tuple (action, message, category, module, lineno) as specified in The Warnings filter section of the Python documentation, separated by ":". Optional fields can be omitted. Module names passed for filtering are not regex-escaped.

For example:

@pytest.mark.warnings("ignore:.*usage will be deprecated.*:DeprecationWarning")
def test_foo():
    ...

pytest.mark.parametrize ¶

Tutorial: Parametrizing fixtures and test functions.

Metafunc.parametrize(argnames: Union[str, List[str], Tuple[str, ...]], argvalues: Iterable[Union[_pytest.mark.structures.ParameterSet, Sequence[object], object]], indirect: Union[bool, Sequence[str]] = False, ids: Union[Iterable[Union[None, str, float, int, bool]], Callable[[object], Optional[object]], None] = None, scope: Optional[str] = None, *, _param_mark: Optional[_pytest.mark.structures.Mark] = None) → None[source]¶

Add new invocations to the underlying test function using the list of argvalues for the given argnames. Parametrization is performed during the collection phase. If you need to setup expensive resources see about setting indirect to do it rather at test setup time.

Parameters:

Parameters:	argnames – a comma-separated string denoting one or more argument names, or a list/tuple of argument strings. argvalues – The list of argvalues determines how often a test is invoked with different argument values. If only one argname was specified argvalues is a list of values. If N argnames were specified, argvalues must be a list of N-tuples, where each tuple-element specifies a value for its respective argname. indirect – The list of argnames or boolean. A list of arguments’ names (subset of argnames). If True the list contains all names from the argnames. Each argvalue corresponding to an argname in this list will be passed as request.param to its respective argname fixture function so that it can perform more expensive setups during the setup phase of a test rather than at collection time. ids – sequence of (or generator for) ids for `argvalues`, or a callable to return part of the id for each argvalue. With sequences (and generators like `itertools.count()`) the returned ids should be of type `string`, `int`, `float`, `bool`, or `None`. They are mapped to the corresponding index in `argvalues`. `None` means to use the auto-generated id. If it is a callable it will be called for each entry in `argvalues`, and the return value is used as part of the auto-generated id for the whole set (where parts are joined with dashes (“-“)). This is useful to provide more specific ids for certain items, e.g. dates. Returning `None` will use an auto-generated id. If no ids are provided they will be generated automatically from the argvalues. scope – if specified it denotes the scope of the parameters. The scope is used for grouping tests by parameter instances. It will also override any fixture-function defined scope, allowing to set a dynamic scope using test context or configuration.

argnames – a comma-separated string denoting one or more argument names, or a list/tuple of argument strings.
argvalues – The list of argvalues determines how often a test is invoked with different argument values. If only one argname was specified argvalues is a list of values. If N argnames were specified, argvalues must be a list of N-tuples, where each tuple-element specifies a value for its respective argname.
indirect – The list of argnames or boolean. A list of arguments’ names (subset of argnames). If True the list contains all names from the argnames. Each argvalue corresponding to an argname in this list will be passed as request.param to its respective argname fixture function so that it can perform more expensive setups during the setup phase of a test rather than at collection time.
ids –

sequence of (or generator for) ids for argvalues,

or a callable to return part of the id for each argvalue.

With sequences (and generators like itertools.count()) the returned ids should be of type string, int, float, bool, or None. They are mapped to the corresponding index in argvalues. None means to use the auto-generated id.

If it is a callable it will be called for each entry in argvalues, and the return value is used as part of the auto-generated id for the whole set (where parts are joined with dashes (“-“)). This is useful to provide more specific ids for certain items, e.g. dates. Returning None will use an auto-generated id.

If no ids are provided they will be generated automatically from the argvalues.
scope – if specified it denotes the scope of the parameters. The scope is used for grouping tests by parameter instances. It will also override any fixture-function defined scope, allowing to set a dynamic scope using test context or configuration.

pytest.mark.skip ¶

Tutorial: Skipping test functions.

Unconditionally skip a test function.

pytest.mark.skip(*, reason=None)¶

Parameters:	reason (str) – Reason why the test function is being skipped.

pytest.mark.skipif ¶

Tutorial: Skipping test functions.

Skip a test function if a condition is True.

pytest.mark.skipif(condition, *, reason=None)¶

Parameters:	condition (bool or str) – `True/False` if the condition should be skipped or a condition string. reason (str) – Reason why the test function is being skipped.

pytest.mark.usefixtures ¶

Tutorial: Using fixtures from classes, modules or projects.

Mark a test function as using the given fixture names.

Warning

This mark has no effect when applied to a fixture function.

pytest.mark.usefixtures(*names)¶

Parameters:	args – the names of the fixture to use, as strings

pytest.mark.xfail ¶

Tutorial: XFail: mark test functions as expected to fail.

Marks a test function as expected to fail.

pytest.mark.xfail(condition=None, *, reason=None, raises=None, run=True, strict=False)¶

Parameters:

Parameters:	condition (bool or str) – Condition for marking the test function as xfail (`True/False` or a condition string). reason (str) – Reason why the test function is marked as xfail. raises (Exception) – Exception subclass expected to be raised by the test function; other exceptions will fail the test. run (bool) – If the test function should actually be executed. If `False`, the function will always xfail and will not be executed (useful if a function is segfaulting). strict (bool) – If `False` (the default) the function will be shown in the terminal output as `xfailed` if it fails and as `xpass` if it passes. In both cases this will not cause the test suite to fail as a whole. This is particularly useful to mark flaky tests (tests that fail at random) to be tackled later. If `True`, the function will be shown in the terminal output as `xfailed` if it fails, but if it unexpectedly passes then it will fail the test suite. This is particularly useful to mark functions that are always failing and there should be a clear indication if they unexpectedly start to pass (for example a new release of a library fixes a known bug).

condition (bool or str) – Condition for marking the test function as xfail (True/False or a condition string).
reason (str) – Reason why the test function is marked as xfail.
raises (Exception) – Exception subclass expected to be raised by the test function; other exceptions will fail the test.
run (bool) – If the test function should actually be executed. If False, the function will always xfail and will not be executed (useful if a function is segfaulting).
strict (bool) –
- If False (the default) the function will be shown in the terminal output as xfailed if it fails and as xpass if it passes. In both cases this will not cause the test suite to fail as a whole. This is particularly useful to mark flaky tests (tests that fail at random) to be tackled later.
- If True, the function will be shown in the terminal output as xfailed if it fails, but if it unexpectedly passes then it will fail the test suite. This is particularly useful to mark functions that are always failing and there should be a clear indication if they unexpectedly start to pass (for example a new release of a library fixes a known bug).

custom marks ¶

Marks are created dynamically using the factory object pytest.mark and applied as a decorator.

For example:

@pytest.mark.timeout(10, "slow", method="thread")
def test_function():
    ...

Will create and attach a Mark object to the collected Item, which can then be accessed by fixtures or hooks with Node.iter_markers. The mark object will have the following attributes:

mark.args == (10, "slow")
mark.kwargs == {"method": "thread"}

Fixtures ¶

Tutorial: pytest fixtures: explicit, modular, scalable.

Fixtures are requested by test functions or other fixtures by declaring them as argument names.

Example of a test requiring a fixture:

def test_output(capsys):
    print("hello")
    out, err = capsys.readouterr()
    assert out == "hello\n"

Example of a fixture requiring another fixture:

@pytest.fixture
def db_session(tmpdir):
    fn = tmpdir / "db.file"
    return connect(str(fn))

For more details, consult the full fixtures docs.

@pytest.fixture ¶

@fixture(callable_or_scope=None, *args, scope='function', params=None, autouse=False, ids=None, name=None)[source]¶

Decorator to mark a fixture factory function.

This decorator can be used, with or without parameters, to define a fixture function.

The name of the fixture function can later be referenced to cause its invocation ahead of running tests: test modules or classes can use the pytest.mark.usefixtures(fixturename) marker.

Test functions can directly use fixture names as input arguments in which case the fixture instance returned from the fixture function will be injected.

Fixtures can provide their values to test functions using return or yield statements. When using yield the code block after the yield statement is executed as teardown code regardless of the test outcome, and must yield exactly once.

Parameters:

Parameters:	scope – the scope for which this fixture is shared, one of `"function"` (default), `"class"`, `"module"`, `"package"` or `"session"` (`"package"` is considered experimental at this time). This parameter may also be a callable which receives `(fixture_name, config)` as parameters, and must return a `str` with one of the values mentioned above. See Dynamic scope in the docs for more information. params – an optional list of parameters which will cause multiple invocations of the fixture function and all of the tests using it. The current parameter is available in `request.param`. autouse – if True, the fixture func is activated for all tests that can see it. If False (the default) then an explicit reference is needed to activate the fixture. ids – list of string ids each corresponding to the params so that they are part of the test id. If no ids are provided they will be generated automatically from the params. name – the name of the fixture. This defaults to the name of the decorated function. If a fixture is used in the same module in which it is defined, the function name of the fixture will be shadowed by the function arg that requests the fixture; one way to resolve this is to name the decorated function `fixture_<fixturename>` and then use `@pytest.fixture(name='<fixturename>')`.

scope –
the scope for which this fixture is shared, one of "function" (default), "class", "module", "package" or "session" ("package" is considered experimental at this time).

This parameter may also be a callable which receives (fixture_name, config) as parameters, and must return a str with one of the values mentioned above.

See Dynamic scope in the docs for more information.
params – an optional list of parameters which will cause multiple invocations of the fixture function and all of the tests using it. The current parameter is available in request.param.
autouse – if True, the fixture func is activated for all tests that can see it. If False (the default) then an explicit reference is needed to activate the fixture.
ids – list of string ids each corresponding to the params so that they are part of the test id. If no ids are provided they will be generated automatically from the params.
name – the name of the fixture. This defaults to the name of the decorated function. If a fixture is used in the same module in which it is defined, the function name of the fixture will be shadowed by the function arg that requests the fixture; one way to resolve this is to name the decorated function fixture_<fixturename> and then use @pytest.fixture(name='<fixturename>').

config.cache ¶

Tutorial: Cache: working with cross-testrun state.

The config.cache object allows other plugins and fixtures to store and retrieve values across test runs. To access it from fixtures request pytestconfig into your fixture and get it with pytestconfig.cache.

Under the hood, the cache plugin uses the simple dumps/loads API of the json stdlib module.

Cache.get(key, default)[source]¶

return cached value for the given key. If no value was yet cached or the value cannot be read, the specified default is returned.

Parameters:	key – must be a `/` separated value. Usually the first name is the name of your plugin or your application. default – must be provided in case of a cache-miss or invalid cache values.

Cache.set(key, value)[source]¶

save value for the given key.

Parameters:	key – must be a `/` separated value. Usually the first name is the name of your plugin or your application. value – must be of any combination of basic python types, including nested types like e. g. lists of dictionaries.

Cache.makedir(name)[source]¶

return a directory path object with the given name. If the directory does not yet exist, it will be created. You can use it to manage files likes e. g. store/retrieve database dumps across test sessions.

Parameters:	name – must be a string not containing a `/` separator. Make sure the name contains your plugin or application identifiers to prevent clashes with other cache users.

capsys ¶

Tutorial: Capturing of the stdout/stderr output.

capsys()[source]¶

Enable text capturing of writes to sys.stdout and sys.stderr.

The captured output is made available via capsys.readouterr() method calls, which return a (out, err) namedtuple. out and err will be text objects.

Returns an instance of CaptureFixture.

Example:

def test_output(capsys):
    print("hello")
    captured = capsys.readouterr()
    assert captured.out == "hello\n"

class CaptureFixture[source]¶

Object returned by capsys(), capsysbinary(), capfd() and capfdbinary() fixtures.

readouterr()[source]¶

Read and return the captured output so far, resetting the internal buffer.

Returns:	captured content as a namedtuple with `out` and `err` string attributes

with disabled()[source]¶: Temporarily disables capture while inside the ‘with’ block.

capsysbinary ¶

Tutorial: Capturing of the stdout/stderr output.

capsysbinary()[source]¶

Enable bytes capturing of writes to sys.stdout and sys.stderr.

The captured output is made available via capsysbinary.readouterr() method calls, which return a (out, err) namedtuple. out and err will be bytes objects.

Returns an instance of CaptureFixture.

Example:

def test_output(capsysbinary):
    print("hello")
    captured = capsysbinary.readouterr()
    assert captured.out == b"hello\n"

capfd ¶

Tutorial: Capturing of the stdout/stderr output.

capfd()[source]¶

Enable text capturing of writes to file descriptors 1 and 2.

The captured output is made available via capfd.readouterr() method calls, which return a (out, err) namedtuple. out and err will be text objects.

Returns an instance of CaptureFixture.

Example:

def test_system_echo(capfd):
    os.system('echo "hello"')
    captured = capfd.readouterr()
    assert captured.out == "hello\n"

capfdbinary ¶

Tutorial: Capturing of the stdout/stderr output.

capfdbinary()[source]¶

Enable bytes capturing of writes to file descriptors 1 and 2.

The captured output is made available via capfd.readouterr() method calls, which return a (out, err) namedtuple. out and err will be byte objects.

Returns an instance of CaptureFixture.

Example:

def test_system_echo(capfdbinary):
    os.system('echo "hello"')
    captured = capfdbinary.readouterr()
    assert captured.out == b"hello\n"

doctest_namespace ¶

Tutorial: Doctest integration for modules and test files.

doctest_namespace()[source]¶

Fixture that returns a dict that will be injected into the namespace of doctests.

Usually this fixture is used in conjunction with another autouse fixture:

@pytest.fixture(autouse=True)
def add_np(doctest_namespace):
    doctest_namespace["np"] = numpy

For more details: ‘doctest_namespace’ fixture.

request ¶

Tutorial: Pass different values to a test function, depending on command line options.

The request fixture is a special fixture providing information of the requesting test function.

class FixtureRequest[source]¶

A request for a fixture from a test or fixture function.

A request object gives access to the requesting test context and has an optional param attribute in case the fixture is parametrized indirectly.

fixturename = None¶: fixture for which this request is being performed

scope = None¶: Scope string, one of “function”, “class”, “module”, “session”

fixturenames¶: names of all active fixtures in this request

funcargnames¶: alias attribute for fixturenames for pre-2.3 compatibility

node¶: underlying collection node (depends on current request scope)

config¶: the pytest config object associated with this request.

function¶: test function object if the request has a per-function scope.

cls¶: class (can be None) where the test function was collected.

instance¶: instance (can be None) on which test function was collected.

module¶: python module object where the test function was collected.

fspath¶: the file system path of the test module which collected this test.

keywords¶: keywords/markers dictionary for the underlying node.

session¶: pytest session object.

addfinalizer(finalizer)[source]¶: add finalizer/teardown function to be called after the last test within the requesting test context finished execution.

applymarker(marker)[source]¶

Apply a marker to a single test function invocation. This method is useful if you don’t want to have a keyword/marker on all function invocations.

Parameters:	marker – a `_pytest.mark.MarkDecorator` object created by a call to `pytest.mark.NAME(...)`.

raiseerror(msg)[source]¶: raise a FixtureLookupError with the given message.

getfixturevalue(argname)[source]¶

Dynamically run a named fixture function.

Declaring fixtures via function argument is recommended where possible. But if you can only decide whether to use another fixture at test setup time, you may use this function to retrieve it inside a fixture or test function body.

pytestconfig ¶

pytestconfig()[source]¶

Session-scoped fixture that returns the _pytest.config.Config object.

Example:

def test_foo(pytestconfig):
    if pytestconfig.getoption("verbose") > 0:
        ...

record_property ¶

Tutorial: record_property.

record_property()[source]¶

Add an extra properties the calling test. User properties become part of the test report and are available to the configured reporters, like JUnit XML. The fixture is callable with (name, value), with value being automatically xml-encoded.

Example:

def test_function(record_property):
    record_property("example_key", 1)

record_testsuite_property ¶

Tutorial: record_testsuite_property.

record_testsuite_property()[source]¶

Records a new <property> tag as child of the root <testsuite>. This is suitable to writing global information regarding the entire test suite, and is compatible with xunit2 JUnit family.

This is a session-scoped fixture which is called with (name, value). Example:

def test_foo(record_testsuite_property):
    record_testsuite_property("ARCH", "PPC")
    record_testsuite_property("STORAGE_TYPE", "CEPH")

name must be a string, value will be converted to a string and properly xml-escaped.

caplog ¶

Tutorial: Logging.

caplog()[source]¶

Access and control log capturing.

Captured logs are available through the following properties/methods:

* caplog.messages        -> list of format-interpolated log messages
* caplog.text            -> string containing formatted log output
* caplog.records         -> list of logging.LogRecord instances
* caplog.record_tuples   -> list of (logger_name, level, message) tuples
* caplog.clear()         -> clear captured records and formatted log output string

This returns a _pytest.logging.LogCaptureFixture instance.

class LogCaptureFixture(item)[source]¶

Provides access and control of log capturing.

handler¶

Return type:	LogCaptureHandler

get_records(when: str) → List[logging.LogRecord][source]¶

Get the logging records for one of the possible test phases.

Parameters:	when (str) – Which test phase to obtain the records from. Valid values are: “setup”, “call” and “teardown”.
Return type:	List[logging.LogRecord]
Returns:	the list of captured records at the given stage

New in version 3.4.

text¶: Returns the formatted log text.

records¶: Returns the list of log records.

record_tuples¶

Returns a list of a stripped down version of log records intended for use in assertion comparison.

The format of the tuple is:

(logger_name, log_level, message)

messages¶

Returns a list of format-interpolated log messages.

Unlike ‘records’, which contains the format string and parameters for interpolation, log messages in this list are all interpolated. Unlike ‘text’, which contains the output from the handler, log messages in this list are unadorned with levels, timestamps, etc, making exact comparisons more reliable.

Note that traceback or stack info (from logging.exception() or the exc_info or stack_info arguments to the logging functions) is not included, as this is added by the formatter in the handler.

New in version 3.7.

clear()[source]¶: Reset the list of log records and the captured log text.

set_level(level, logger=None)[source]¶

Sets the level for capturing of logs. The level will be restored to its previous value at the end of the test.

Parameters:	level (int) – the logger to level. logger (str) – the logger to update the level. If not given, the root logger level is updated.

Changed in version 3.4: The levels of the loggers changed by this function will be restored to their initial values at the end of the test.

with at_level(level, logger=None)[source]¶

Context manager that sets the level for capturing of logs. After the end of the ‘with’ statement the level is restored to its original value.

Parameters:	level (int) – the logger to level. logger (str) – the logger to update the level. If not given, the root logger level is updated.

monkeypatch ¶

Tutorial: Monkeypatching/mocking modules and environments.

monkeypatch()[source]¶

The returned monkeypatch fixture provides these helper methods to modify objects, dictionaries or os.environ:

monkeypatch.setattr(obj, name, value, raising=True)
monkeypatch.delattr(obj, name, raising=True)
monkeypatch.setitem(mapping, name, value)
monkeypatch.delitem(obj, name, raising=True)
monkeypatch.setenv(name, value, prepend=False)
monkeypatch.delenv(name, raising=True)
monkeypatch.syspath_prepend(path)
monkeypatch.chdir(path)

All modifications will be undone after the requesting test function or fixture has finished. The raising parameter determines if a KeyError or AttributeError will be raised if the set/deletion operation has no target.

This returns a MonkeyPatch instance.

class MonkeyPatch[source]¶

Object returned by the monkeypatch fixture keeping a record of setattr/item/env/syspath changes.

with context() → Generator[MonkeyPatch, None, None][source]¶

Context manager that returns a new MonkeyPatch object which undoes any patching done inside the with block upon exit:

import functools
def test_partial(monkeypatch):
    with monkeypatch.context() as m:
        m.setattr(functools, "partial", 3)

Useful in situations where it is desired to undo some patches before the test ends, such as mocking stdlib functions that might break pytest itself if mocked (for examples of this see #3290.

setattr(target, name, value=<notset>, raising=True)[source]¶

Set attribute value on target, memorizing the old value. By default raise AttributeError if the attribute did not exist.

For convenience you can specify a string as target which will be interpreted as a dotted import path, with the last part being the attribute name. Example: monkeypatch.setattr("os.getcwd", lambda: "/") would set the getcwd function of the os module.

The raising value determines if the setattr should fail if the attribute is not already present (defaults to True which means it will raise).

delattr(target, name=<notset>, raising=True)[source]¶

Delete attribute name from target, by default raise AttributeError it the attribute did not previously exist.

If no name is specified and target is a string it will be interpreted as a dotted import path with the last part being the attribute name.

If raising is set to False, no exception will be raised if the attribute is missing.

setitem(dic, name, value)[source]¶: Set dictionary entry name to value.

delitem(dic, name, raising=True)[source]¶

Delete name from dict. Raise KeyError if it doesn’t exist.

If raising is set to False, no exception will be raised if the key is missing.

setenv(name, value, prepend=None)[source]¶: Set environment variable name to value. If prepend is a character, read the current environment variable value and prepend the value adjoined with the prepend character.

delenv(name, raising=True)[source]¶

Delete name from the environment. Raise KeyError if it does not exist.

If raising is set to False, no exception will be raised if the environment variable is missing.

syspath_prepend(path)[source]¶: Prepend path to sys.path list of import locations.

chdir(path)[source]¶: Change the current working directory to the specified path. Path can be a string or a py.path.local object.

undo()[source]¶

Undo previous changes. This call consumes the undo stack. Calling it a second time has no effect unless you do more monkeypatching after the undo call.

There is generally no need to call undo(), since it is called automatically during tear-down.

Note that the same monkeypatch fixture is used across a single test function invocation. If monkeypatch is used both by the test function itself and one of the test fixtures, calling undo() will undo all of the changes made in both functions.

testdir ¶

This fixture provides a Testdir instance useful for black-box testing of test files, making it ideal to test plugins.

To use it, include in your top-most conftest.py file:

pytest_plugins = "pytester"

class Testdir[source]¶

Temporary test directory with tools to test/run pytest itself.

This is based on the tmpdir fixture but provides a number of methods which aid with testing pytest itself. Unless chdir() is used all methods will use tmpdir as their current working directory.

Attributes:

Variables:	tmpdir – The `py.path.local` instance of the temporary directory. plugins – A list of plugins to use with `parseconfig()` and `runpytest()`. Initially this is an empty list but plugins can be added to the list. The type of items to add to the list depends on the method using them so refer to them for details.

CLOSE_STDIN¶: alias of builtins.object

exception TimeoutExpired[source]¶

finalize()[source]¶

Clean up global state artifacts.

Some methods modify the global interpreter state and this tries to clean this up. It does not remove the temporary directory however so it can be looked at after the test run has finished.

make_hook_recorder(pluginmanager)[source]¶: Create a new HookRecorder for a PluginManager.

chdir()[source]¶

Cd into the temporary directory.

This is done automatically upon instantiation.

makefile(ext, *args, **kwargs)[source]¶

Create new file(s) in the testdir.

Parameters:

Parameters:	ext (str) – The extension the file(s) should use, including the dot, e.g. `.py`. args (list[str]) – All args will be treated as strings and joined using newlines. The result will be written as contents to the file. The name of the file will be based on the test function requesting this fixture. kwargs – Each keyword is the name of a file, while the value of it will be written as contents of the file.

ext (str) – The extension the file(s) should use, including the dot, e.g. .py.
args (list[str]) – All args will be treated as strings and joined using newlines. The result will be written as contents to the file. The name of the file will be based on the test function requesting this fixture.
kwargs – Each keyword is the name of a file, while the value of it will be written as contents of the file.

Examples:

testdir.makefile(".txt", "line1", "line2")

testdir.makefile(".ini", pytest="[pytest]\naddopts=-rs\n")

makeconftest(source)[source]¶: Write a contest.py file with ‘source’ as contents.

makeini(source)[source]¶: Write a tox.ini file with ‘source’ as contents.

getinicfg(source)[source]¶: Return the pytest section from the tox.ini config file.

makepyfile(*args, **kwargs)[source]¶: Shortcut for .makefile() with a .py extension.

maketxtfile(*args, **kwargs)[source]¶: Shortcut for .makefile() with a .txt extension.

syspathinsert(path=None)[source]¶

Prepend a directory to sys.path, defaults to tmpdir.

This is undone automatically when this object dies at the end of each test.

mkdir(name)[source]¶: Create a new (sub)directory.

mkpydir(name)[source]¶

Create a new python package.

This creates a (sub)directory with an empty __init__.py file so it gets recognised as a python package.

copy_example(name=None)[source]¶

Copy file from project’s directory into the testdir.

Parameters:	name (str) – The name of the file to copy.
Returns:	path to the copied directory (inside `self.tmpdir`).

class Session(config: _pytest.config.Config)¶

exception Failed¶: signals a stop as failed test run.

exception Interrupted¶: signals an interrupted test run.

for ... in collect()¶: returns a list of children (items and collectors) for this collection node.

getnode(config, arg)[source]¶

Return the collection node of a file.

Parameters:	config – `_pytest.config.Config` instance, see `parseconfig()` and `parseconfigure()` to create the configuration arg – a `py.path.local` instance of the file

getpathnode(path)[source]¶

Return the collection node of a file.

This is like getnode() but uses parseconfigure() to create the (configured) pytest Config instance.

Parameters:	path – a `py.path.local` instance of the file

genitems(colitems)[source]¶

Generate all test items from a collection node.

This recurses into the collection node and returns a list of all the test items contained within.

runitem(source)[source]¶

Run the “test_func” Item.

The calling test instance (class containing the test method) must provide a .getrunner() method which should return a runner which can run the test protocol for a single item, e.g. _pytest.runner.runtestprotocol().

inline_runsource(source, *cmdlineargs)[source]¶

Run a test module in process using pytest.main().

This run writes “source” into a temporary file and runs pytest.main() on it, returning a HookRecorder instance for the result.

Parameters:	source – the source code of the test module cmdlineargs – any extra command line arguments to use
Returns:	`HookRecorder` instance of the result

inline_genitems(*args)[source]¶

Run pytest.main(['--collectonly']) in-process.

Runs the pytest.main() function to run all of pytest inside the test process itself like inline_run(), but returns a tuple of the collected items and a HookRecorder instance.

inline_run(*args, plugins=(), no_reraise_ctrlc: bool = False)[source]¶

Run pytest.main() in-process, returning a HookRecorder.

Runs the pytest.main() function to run all of pytest inside the test process itself. This means it can return a HookRecorder instance which gives more detailed results from that run than can be done by matching stdout/stderr from runpytest().

Parameters:	args – command line arguments to pass to `pytest.main()` plugins – extra plugin instances the `pytest.main()` instance should use. no_reraise_ctrlc – typically we reraise keyboard interrupts from the child run. If True, the KeyboardInterrupt exception is captured.
Returns:	a `HookRecorder` instance

runpytest_inprocess(*args, **kwargs) → _pytest.pytester.RunResult[source]¶: Return result of running pytest in-process, providing a similar interface to what self.runpytest() provides.

runpytest(*args, **kwargs) → _pytest.pytester.RunResult[source]¶: Run pytest inline or in a subprocess, depending on the command line option “–runpytest” and return a RunResult.

parseconfig(*args)[source]¶

Return a new pytest Config instance from given commandline args.

This invokes the pytest bootstrapping code in _pytest.config to create a new _pytest.core.PluginManager and call the pytest_cmdline_parse hook to create a new _pytest.config.Config instance.

If plugins has been populated they should be plugin modules to be registered with the PluginManager.

parseconfigure(*args)[source]¶

Return a new pytest configured Config instance.

This returns a new _pytest.config.Config instance like parseconfig(), but also calls the pytest_configure hook.

getitem(source, funcname='test_func')[source]¶

Return the test item for a test function.

This writes the source to a python file and runs pytest’s collection on the resulting module, returning the test item for the requested function name.

Parameters:	source – the module source funcname – the name of the test function for which to return a test item

getitems(source)[source]¶

Return all test items collected from the module.

This writes the source to a python file and runs pytest’s collection on the resulting module, returning all test items contained within.

getmodulecol(source, configargs=(), withinit=False)[source]¶

Return the module collection node for source.

This writes source to a file using makepyfile() and then runs the pytest collection on it, returning the collection node for the test module.

Parameters:	source – the source code of the module to collect configargs – any extra arguments to pass to `parseconfigure()` withinit – whether to also write an `__init__.py` file to the same directory to ensure it is a package

collect_by_name(modcol: _pytest.python.Module, name: str) → Union[_pytest.nodes.Item, _pytest.nodes.Collector, None][source]¶

Return the collection node for name from the module collection.

This will search a module collection node for a collection node matching the given name.

Parameters:	modcol – a module collection node; see `getmodulecol()` name – the name of the node to return

popen(cmdargs, stdout=-1, stderr=-1, stdin=<class 'object'>, **kw)[source]¶

Invoke subprocess.Popen.

This calls subprocess.Popen making sure the current working directory is in the PYTHONPATH.

You probably want to use run() instead.

run(*cmdargs, timeout=None, stdin=<class 'object'>) → _pytest.pytester.RunResult[source]¶

Run a command with arguments.

Run a process using subprocess.Popen saving the stdout and stderr.

Parameters:	args – the sequence of arguments to pass to `subprocess.Popen()` timeout – the period in seconds after which to timeout and raise `Testdir.TimeoutExpired` stdin – optional standard input. Bytes are being send, closing the pipe, otherwise it is passed through to `popen`. Defaults to `CLOSE_STDIN`, which translates to using a pipe (`subprocess.PIPE`) that gets closed.

Returns a RunResult.

runpython(script) → _pytest.pytester.RunResult[source]¶

Run a python script using sys.executable as interpreter.

Returns a RunResult.

runpython_c(command)[source]¶: Run python -c “command”, return a RunResult.

runpytest_subprocess(*args, timeout=None) → _pytest.pytester.RunResult[source]¶

Run pytest as a subprocess with given arguments.

Any plugins added to the plugins list will be added using the -p command line option. Additionally --basetemp is used to put any temporary files and directories in a numbered directory prefixed with “runpytest-” to not conflict with the normal numbered pytest location for temporary files and directories.

Parameters:	args – the sequence of arguments to pass to the pytest subprocess timeout – the period in seconds after which to timeout and raise `Testdir.TimeoutExpired`

Returns a RunResult.

spawn_pytest(string: str, expect_timeout: float = 10.0) → pexpect.spawn[source]¶

Run pytest using pexpect.

This makes sure to use the right pytest and sets up the temporary directory locations.

The pexpect child is returned.

spawn(cmd: str, expect_timeout: float = 10.0) → pexpect.spawn[source]¶

Run a command using pexpect.

The pexpect child is returned.

class RunResult[source]¶

The result of running a command.

Attributes:

Variables:	ret – the return value outlines – list of lines captured from stdout errlines – list of lines captured from stderr stdout – `LineMatcher` of stdout, use `stdout.str()` to reconstruct stdout or the commonly used `stdout.fnmatch_lines()` method stderr – `LineMatcher` of stderr duration – duration in seconds

parseoutcomes() → Dict[str, int][source]¶: Return a dictionary of outcomestring->num from parsing the terminal output that the test process produced.

assert_outcomes(passed: int = 0, skipped: int = 0, failed: int = 0, error: int = 0, xpassed: int = 0, xfailed: int = 0) → None[source]¶: Assert that the specified outcomes appear with the respective numbers (0 means it didn’t occur) in the text output from a test run.

class LineMatcher[source]¶

Flexible matching of text.

This is a convenience class to test large texts like the output of commands.

The constructor takes a list of lines without their trailing newlines, i.e. text.splitlines().

fnmatch_lines_random(lines2: Sequence[str]) → None[source]¶: Check lines exist in the output in any order (using fnmatch.fnmatch()).

re_match_lines_random(lines2: Sequence[str]) → None[source]¶: Check lines exist in the output in any order (using re.match()).

get_lines_after(fnline: str) → Sequence[str][source]¶

Return all lines following the given line in the text.

The given line can contain glob wildcards.

fnmatch_lines(lines2: Sequence[str], *, consecutive: bool = False) → None[source]¶

Check lines exist in the output (using fnmatch.fnmatch()).

The argument is a list of lines which have to match and can use glob wildcards. If they do not match a pytest.fail() is called. The matches and non-matches are also shown as part of the error message.

Parameters:	lines2 – string patterns to match. consecutive – match lines consecutive?

re_match_lines(lines2: Sequence[str], *, consecutive: bool = False) → None[source]¶

Check lines exist in the output (using re.match()).

The argument is a list of lines which have to match using re.match. If they do not match a pytest.fail() is called.

The matches and non-matches are also shown as part of the error message.

Parameters:	lines2 – string patterns to match. consecutive – match lines consecutively?

no_fnmatch_line(pat: str) → None[source]¶

Ensure captured lines do not match the given pattern, using fnmatch.fnmatch.

Parameters:	pat (str) – the pattern to match lines.

no_re_match_line(pat: str) → None[source]¶

Ensure captured lines do not match the given pattern, using re.match.

Parameters:	pat (str) – the regular expression to match lines.

str() → str[source]¶: Return the entire original text.

recwarn ¶

Tutorial: Asserting warnings with the warns function

recwarn()[source]¶

Return a WarningsRecorder instance that records all warnings emitted by test functions.

See http://docs.python.org/library/warnings.html for information on warning categories.

class WarningsRecorder[source]¶

A context manager to record raised warnings.

Adapted from warnings.catch_warnings.

list¶: The list of recorded warnings.

pop(cls: Type[Warning] = <class 'Warning'>) → warnings._Record[source]¶: Pop the first recorded warning, raise exception if not exists.

clear() → None[source]¶: Clear the list of recorded warnings.

Each recorded warning is an instance of warnings.WarningMessage.

Note

RecordedWarning was changed from a plain class to a namedtuple in pytest 3.1

Note

DeprecationWarning and PendingDeprecationWarning are treated differently; see Ensuring code triggers a deprecation warning.

tmp_path ¶

Tutorial: Temporary directories and files

tmp_path()[source]¶: Return a temporary directory path object which is unique to each test function invocation, created as a sub directory of the base temporary directory. The returned object is a pathlib.Path object.

Note

in python < 3.6 this is a pathlib2.Path

tmp_path_factory ¶

Tutorial: The tmp_path_factory fixture

tmp_path_factory instances have the following methods:

TempPathFactory.mktemp(basename: str, numbered: bool = True) → pathlib.Path[source]¶

Creates a new temporary directory managed by the factory.

Parameters:	basename – Directory base name, must be a relative path. numbered – If True, ensure the directory is unique by adding a number prefix greater than any existing one: `basename="foo"` and `numbered=True` means that this function will create directories named `"foo-0"`, `"foo-1"`, `"foo-2"` and so on.
Returns:	The path to the new directory.

TempPathFactory.getbasetemp() → pathlib.Path[source]¶: return base temporary directory.

tmpdir ¶

Tutorial: Temporary directories and files

tmpdir()[source]¶: Return a temporary directory path object which is unique to each test function invocation, created as a sub directory of the base temporary directory. The returned object is a py.path.local path object.

tmpdir_factory ¶

Tutorial: The ‘tmpdir_factory’ fixture

tmpdir_factory instances have the following methods:

TempdirFactory.mktemp(basename: str, numbered: bool = True) → py._path.local.LocalPath[source]¶: Same as TempPathFactory.mkdir(), but returns a py.path.local object.

TempdirFactory.getbasetemp()[source]¶: backward compat wrapper for _tmppath_factory.getbasetemp

Hooks ¶

Tutorial: Writing plugins.

Reference to all hooks which can be implemented by conftest.py files and plugins.

Bootstrapping hooks ¶

Bootstrapping hooks called for plugins registered early enough (internal and setuptools plugins).

pytest_load_initial_conftests(early_config, parser, args)[source]¶

implements the loading of initial conftest files ahead of command line option parsing.

Note

This hook will not be called for conftest.py files, only for setuptools plugins.

Parameters:	early_config (_pytest.config.Config) – pytest config object args (list[str]) – list of arguments passed on the command line parser (_pytest.config.argparsing.Parser) – to add command line options

pytest_cmdline_preparse(config, args)[source]¶

(Deprecated) modify command line arguments before option parsing.

This hook is considered deprecated and will be removed in a future pytest version. Consider using pytest_load_initial_conftests() instead.

Note

This hook will not be called for conftest.py files, only for setuptools plugins.

Parameters:	config (_pytest.config.Config) – pytest config object args (list[str]) – list of arguments passed on the command line

pytest_cmdline_parse(pluginmanager, args)[source]¶

return initialized config object, parsing the specified args.

Stops at first non-None result, see firstresult: stop at first non-None result

Note

This hook will only be called for plugin classes passed to the plugins arg when using pytest.main to perform an in-process test run.

Parameters:	pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager args (list[str]) – list of arguments passed on the command line

pytest_cmdline_main(config)[source]¶

called for performing the main command line action. The default implementation will invoke the configure hooks and runtest_mainloop.

Note

This hook will not be called for conftest.py files, only for setuptools plugins.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:	config (_pytest.config.Config) – pytest config object

Initialization hooks ¶

Initialization hooks called for plugins and conftest.py files.

pytest_addoption(parser, pluginmanager)[source]¶

Note

This function should be implemented only in plugins or conftest.py files situated at the tests root directory due to how pytest discovers plugins during startup.

Parameters:

parser (_pytest.config.argparsing.Parser) – To add command line options, call parser.addoption(...). To add ini-file values call parser.addini(...).
pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager, which can be used to install hookspec()’s or hookimpl()’s and allow one plugin to call another plugin’s hooks to change how command line options are added.

Options can later be accessed through the config object, respectively:

config.getoption(name) to retrieve the value of a command line option.
config.getini(name) to retrieve a value read from an ini-style file.

The config object is passed around on many internal objects via the .config attribute or can be retrieved as the pytestconfig fixture.

Note

This hook is incompatible with hookwrapper=True.

pytest_addhooks(pluginmanager)[source]¶

called at plugin registration time to allow adding new hooks via a call to pluginmanager.add_hookspecs(module_or_class, prefix).

Parameters:	pluginmanager (_pytest.config.PytestPluginManager) – pytest plugin manager

Note

This hook is incompatible with hookwrapper=True.

pytest_configure(config)[source]¶

Allows plugins and conftest files to perform initial configuration.

This hook is called for every plugin and initial conftest file after command line options have been parsed.

After that, the hook is called for other conftest files as they are imported.

Note

This hook is incompatible with hookwrapper=True.

Parameters:	config (_pytest.config.Config) – pytest config object

pytest_unconfigure(config)[source]¶

called before test process is exited.

Parameters:	config (_pytest.config.Config) – pytest config object

pytest_sessionstart(session)[source]¶

called after the Session object has been created and before performing collection and entering the run test loop.

Parameters:	session (_pytest.main.Session) – the pytest session object

pytest_sessionfinish(session, exitstatus)[source]¶

called after whole test run finished, right before returning the exit status to the system.

Parameters:	session (_pytest.main.Session) – the pytest session object exitstatus (int) – the status which pytest will return to the system

pytest_plugin_registered(plugin, manager)[source]¶

a new pytest plugin got registered.

Parameters:	plugin – the plugin module or instance manager (_pytest.config.PytestPluginManager) – pytest plugin manager

Note

This hook is incompatible with hookwrapper=True.

Test running hooks ¶

All runtest related hooks receive a pytest.Item object.

pytest_runtestloop(session)[source]¶

called for performing the main runtest loop (after collection finished).

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:	session (_pytest.main.Session) – the pytest session object

pytest_runtest_protocol(item, nextitem)[source]¶

implements the runtest_setup/call/teardown protocol for the given test item, including capturing exceptions and calling reporting hooks.

Parameters:	item – test item for which the runtest protocol is performed. nextitem – the scheduled-to-be-next test item (or None if this is the end my friend). This argument is passed on to `pytest_runtest_teardown()`.
Return boolean:	True if no further hook implementations should be invoked.

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_runtest_logstart(nodeid, location)[source]¶

signal the start of running a single test item.

This hook will be called before pytest_runtest_setup(), pytest_runtest_call() and pytest_runtest_teardown() hooks.

Parameters:	nodeid (str) – full id of the item location – a triple of `(filename, linenum, testname)`

pytest_runtest_logfinish(nodeid, location)[source]¶

signal the complete finish of running a single test item.

This hook will be called after pytest_runtest_setup(), pytest_runtest_call() and pytest_runtest_teardown() hooks.

Parameters:	nodeid (str) – full id of the item location – a triple of `(filename, linenum, testname)`

pytest_runtest_setup(item)[source]¶: called before pytest_runtest_call(item).

pytest_runtest_call(item)[source]¶: called to execute the test item.

pytest_runtest_teardown(item, nextitem)[source]¶

called after pytest_runtest_call.

Parameters:	nextitem – the scheduled-to-be-next test item (None if no further test item is scheduled). This argument can be used to perform exact teardowns, i.e. calling just enough finalizers so that nextitem only needs to call setup-functions.

pytest_runtest_makereport(item, call)[source]¶

return a _pytest.runner.TestReport object for the given pytest.Item and _pytest.runner.CallInfo.

Stops at first non-None result, see firstresult: stop at first non-None result

For deeper understanding you may look at the default implementation of these hooks in _pytest.runner and maybe also in _pytest.pdb which interacts with _pytest.capture and its input/output capturing in order to immediately drop into interactive debugging when a test failure occurs.

The _pytest.terminal reported specifically uses the reporting hook to print information about a test run.

pytest_pyfunc_call(pyfuncitem)[source]¶

call underlying test function.

Stops at first non-None result, see firstresult: stop at first non-None result

Collection hooks ¶

pytest calls the following hooks for collecting files and directories:

pytest_collection(session: Session) → Optional[Any][source]¶

Perform the collection protocol for the given session.

Stops at first non-None result, see firstresult: stop at first non-None result.

Parameters:	session (_pytest.main.Session) – the pytest session object

pytest_ignore_collect(path, config)[source]¶

return True to prevent considering this path for collection. This hook is consulted for all files and directories prior to calling more specific hooks.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:	path – a `py.path.local` - the path to analyze config (_pytest.config.Config) – pytest config object

pytest_collect_directory(path, parent)[source]¶

called before traversing a directory for collection files.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:	path – a `py.path.local` - the path to analyze

pytest_collect_file(path, parent)[source]¶

return collection Node or None for the given path. Any new node needs to have the specified parent as a parent.

Parameters:	path – a `py.path.local` - the path to collect

pytest_pycollect_makemodule(path, parent)[source]¶

return a Module collector or None for the given path. This hook will be called for each matching test module path. The pytest_collect_file hook needs to be used if you want to create test modules for files that do not match as a test module.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:	path – a `py.path.local` - the path of module to collect

For influencing the collection of objects in Python modules you can use the following hook:

pytest_pycollect_makeitem(collector, name, obj)[source]¶

return custom item/collector for a python object in a module, or None.

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_generate_tests(metafunc)[source]¶: generate (multiple) parametrized calls to a test function.

pytest_make_parametrize_id(config, val, argname)[source]¶

Return a user-friendly string representation of the given val that will be used by @pytest.mark.parametrize calls. Return None if the hook doesn’t know about val. The parameter name is available as argname, if required.

Stops at first non-None result, see firstresult: stop at first non-None result

Parameters:	config (_pytest.config.Config) – pytest config object val – the parametrized value argname (str) – the automatic parameter name produced by pytest

After collection is complete, you can modify the order of items, delete or otherwise amend the test items:

pytest_collection_modifyitems(session, config, items)[source]¶

called after collection has been performed, may filter or re-order the items in-place.

Parameters:	session (_pytest.main.Session) – the pytest session object config (_pytest.config.Config) – pytest config object items (List[_pytest.nodes.Item]) – list of item objects

pytest_collection_finish(session)[source]¶

called after collection has been performed and modified.

Parameters:	session (_pytest.main.Session) – the pytest session object

Reporting hooks ¶

Session related reporting hooks:

pytest_collectstart(collector)[source]¶: collector starts collecting.

pytest_make_collect_report(collector)[source]¶

perform collector.collect() and return a CollectReport.

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_itemcollected(item)[source]¶: we just collected a test item.

pytest_collectreport(report)[source]¶: collector finished collecting.

pytest_deselected(items)[source]¶: called for test items deselected, e.g. by keyword.

pytest_report_header(config, startdir)[source]¶

return a string or list of strings to be displayed as header info for terminal reporting.

Parameters:	config (_pytest.config.Config) – pytest config object startdir – py.path object with the starting dir

Note

This function should be implemented only in plugins or conftest.py files situated at the tests root directory due to how pytest discovers plugins during startup.

pytest_report_collectionfinish(config, startdir, items)[source]¶

New in version 3.2.

return a string or list of strings to be displayed after collection has finished successfully.

This strings will be displayed after the standard “collected X items” message.

Parameters:	config (_pytest.config.Config) – pytest config object startdir – py.path object with the starting dir items – list of pytest items that are going to be executed; this list should not be modified.

pytest_report_teststatus(report, config)[source]¶

return result-category, shortletter and verbose word for reporting.

Parameters:	config (_pytest.config.Config) – pytest config object

Stops at first non-None result, see firstresult: stop at first non-None result

pytest_terminal_summary(terminalreporter, exitstatus, config)[source]¶

Add a section to terminal summary reporting.

Parameters:	terminalreporter (_pytest.terminal.TerminalReporter) – the internal terminal reporter object exitstatus (int) – the exit status that will be reported back to the OS config (_pytest.config.Config) – pytest config object

New in version 4.2: The config parameter.

pytest_fixture_setup(fixturedef, request)[source]¶

performs fixture setup execution.

Returns:	The return value of the call to the fixture function

Stops at first non-None result, see firstresult: stop at first non-None result

Note

If the fixture function returns None, other implementations of this hook function will continue to be called, according to the behavior of the firstresult: stop at first non-None result option.

pytest_fixture_post_finalizer(fixturedef, request)[source]¶: Called after fixture teardown, but before the cache is cleared, so the fixture result fixturedef.cached_result is still available (not None).

pytest_warning_captured(warning_message, when, item, location)[source]¶

Process a warning captured by the internal pytest warnings plugin.

Parameters:

warning_message (warnings.WarningMessage) – The captured warning. This is the same object produced by warnings.catch_warnings(), and contains the same attributes as the parameters of warnings.showwarning().
when (str) –
Indicates when the warning was captured. Possible values:
- "config": during pytest configuration/initialization stage.
- "collect": during test collection.
- "runtest": during test execution.
item (pytest.Item|None) –
DEPRECATED: This parameter is incompatible with pytest-xdist, and will always receive None in a future release.

The item being executed if when is "runtest", otherwise None.
location (tuple) – Holds information about the execution context of the captured warning (filename, linenumber, function). function evaluates to <module> when the execution context is at the module level.

Central hook for reporting about test execution:

pytest_runtest_logreport(report)[source]¶: process a test setup/call/teardown report relating to the respective phase of executing a test.

Assertion related hooks:

pytest_assertrepr_compare(config, op, left, right)[source]¶

return explanation for comparisons in failing assert expressions.

Parameters:	config (_pytest.config.Config) – pytest config object

pytest_assertion_pass(item, lineno, orig, expl)[source]¶

(Experimental)

New in version 5.0.

Hook called whenever an assertion passes.

Use this hook to do some processing after a passing assertion. The original assertion information is available in the orig string and the pytest introspected assertion information is available in the expl string.

This hook must be explicitly enabled by the enable_assertion_pass_hook ini-file option:

[pytest]
enable_assertion_pass_hook=true

You need to clean the .pyc files in your project directory and interpreter libraries when enabling this option, as assertions will require to be re-written.

Parameters:	item (_pytest.nodes.Item) – pytest item object of current test lineno (int) – line number of the assert statement orig (string) – string with original assertion expl (string) – string with assert explanation

Note

This hook is experimental, so its parameters or even the hook itself might be changed/removed without warning in any future pytest release.

If you find this hook useful, please share your feedback opening an issue.

Debugging/Interaction hooks ¶

There are few hooks which can be used for special reporting or interaction with exceptions:

pytest_internalerror(excrepr, excinfo)[source]¶: called for internal errors.

pytest_keyboard_interrupt(excinfo)[source]¶: called for keyboard interrupt.

pytest_exception_interact(node, call, report)[source]¶

called when an exception was raised which can potentially be interactively handled.

This hook is only called if an exception was raised that is not an internal exception like skip.Exception.

pytest_enter_pdb(config, pdb)[source]¶

called upon pdb.set_trace(), can be used by plugins to take special action just before the python debugger enters in interactive mode.

Parameters:	config (_pytest.config.Config) – pytest config object pdb (pdb.Pdb) – Pdb instance

Objects ¶

Full reference to objects accessible from fixtures or hooks.

CallInfo ¶

class CallInfo[source]¶: Result/Exception info a function invocation.

Class ¶

class Class[source]¶

Bases: _pytest.python.PyCollector

Collector for test methods.

classmethod from_parent(parent, *, name, obj=None)[source]¶: The public constructor

collect()[source]¶: returns a list of children (items and collectors) for this collection node.

Collector ¶

class Collector[source]¶

Bases: _pytest.nodes.Node

Collector instances create children through collect() and thus iteratively build a tree.

exception CollectError[source]¶

Bases: Exception

an error during collection, contains a custom message.

collect()[source]¶: returns a list of children (items and collectors) for this collection node.

repr_failure(excinfo)[source]¶: represent a collection failure.

Config ¶

class Config[source]¶

Access to configuration values, pluginmanager and plugin hooks.

Variables:

pluginmanager (PytestPluginManager) – the plugin manager handles plugin registration and hook invocation.
option (argparse.Namespace) – access to command line option as attributes.
invocation_params (InvocationParams) –
Object containing the parameters regarding the pytest.main invocation.

Contains the following read-only attributes:
- args: tuple of command-line arguments as passed to pytest.main().
- plugins: list of extra plugins, might be None.
- dir: directory where pytest.main() was invoked from.

class InvocationParams(args, plugins, dir: pathlib.Path)[source]¶: Holds parameters passed during pytest.main()

New in version 5.1.

Note

Note that the environment variable PYTEST_ADDOPTS and the addopts ini option are handled by pytest, not being included in the args attribute.

Plugins accessing InvocationParams must be aware of that.

invocation_dir¶: Backward compatibility

add_cleanup(func)[source]¶: Add a function to be called when the config object gets out of use (usually coninciding with pytest_unconfigure).

classmethod fromdictargs(option_dict, args)[source]¶: constructor usable for subprocesses.

addinivalue_line(name, line)[source]¶: add a line to an ini-file option. The option must have been declared but might not yet be set in which case the line becomes the the first line in its value.

getini(name: str)[source]¶: return configuration value from an ini file. If the specified name hasn’t been registered through a prior parser.addini call (usually from a plugin), a ValueError is raised.

getoption(name: str, default=<NOTSET>, skip: bool = False)[source]¶

return command line option value.

Parameters:	name – name of the option. You may also specify the literal `--OPT` option instead of the “dest” option name. default – default value if no option of that name exists. skip – if True raise pytest.skip if option does not exists or has a None value.

getvalue(name, path=None)[source]¶: (deprecated, use getoption())

getvalueorskip(name, path=None)[source]¶: (deprecated, use getoption(skip=True))

ExceptionInfo ¶

class ExceptionInfo(excinfo: Optional[Tuple[Type[_E], _E, traceback]], striptext: str = '', traceback: Optional[_pytest._code.code.Traceback] = None)[source]¶

wraps sys.exc_info() objects and offers help for navigating the traceback.

classmethod from_exc_info(exc_info: Tuple[Type[_E], _E, traceback], exprinfo: Optional[str] = None) → ExceptionInfo[_E][source]¶

returns an ExceptionInfo for an existing exc_info tuple.

Warning

Experimental API

Parameters:	exprinfo – a text string helping to determine if we should strip `AssertionError` from the output, defaults to the exception message/`__str__()`

classmethod from_current(exprinfo: Optional[str] = None) → _pytest._code.code.ExceptionInfo[BaseException][BaseException][source]¶

returns an ExceptionInfo matching the current traceback

Warning

Experimental API

Parameters:	exprinfo – a text string helping to determine if we should strip `AssertionError` from the output, defaults to the exception message/`__str__()`

classmethod for_later() → _pytest._code.code.ExceptionInfo[~_E][_E][source]¶: return an unfilled ExceptionInfo

fill_unfilled(exc_info: Tuple[Type[_E], _E, traceback]) → None[source]¶: fill an unfilled ExceptionInfo created with for_later()

type¶: the exception class

value¶: the exception value

tb¶: the exception raw traceback

typename¶: the type name of the exception

traceback¶: the traceback

exconly(tryshort: bool = False) → str[source]¶

return the exception as a string

when ‘tryshort’ resolves to True, and the exception is a _pytest._code._AssertionError, only the actual exception part of the exception representation is returned (so ‘AssertionError: ‘ is removed from the beginning)

errisinstance(exc: Union[Type[BaseException], Tuple[Type[BaseException], ...]]) → bool[source]¶: return True if the exception is an instance of exc

getrepr(showlocals: bool = False, style: _TracebackStyle = 'long', abspath: bool = False, tbfilter: bool = True, funcargs: bool = False, truncate_locals: bool = True, chain: bool = True) → Union[ReprExceptionInfo, ExceptionChainRepr][source]¶

Return str()able representation of this exception info.

Parameters:

showlocals (bool) – Show locals per traceback entry. Ignored if style=="native".
style (str) – long|short|no|native traceback style
abspath (bool) – If paths should be changed to absolute or left unchanged.
tbfilter (bool) – Hide entries that contain a local variable __tracebackhide__==True. Ignored if style=="native".
funcargs (bool) – Show fixtures (“funcargs” for legacy purposes) per traceback entry.
truncate_locals (bool) – With showlocals==True, make sure locals can be safely represented as strings.
chain (bool) – if chained exceptions in Python 3 should be shown.

Changed in version 3.9: Added the chain parameter.

match(regexp: Union[str, Pattern]) → Literal[True][source]¶: Check whether the regular expression regexp matches the string representation of the exception using re.search(). If it matches True is returned. If it doesn’t match an AssertionError is raised.

pytest.ExitCode ¶

class ExitCode[source]¶

New in version 5.0.

Encodes the valid exit codes by pytest.

Currently users and plugins may supply other exit codes as well.

OK = 0¶: tests passed

TESTS_FAILED = 1¶: tests failed

INTERRUPTED = 2¶: pytest was interrupted

INTERNAL_ERROR = 3¶: an internal error got in the way

USAGE_ERROR = 4¶: pytest was misused

NO_TESTS_COLLECTED = 5¶: pytest couldn’t find tests

FixtureDef ¶

class FixtureDef[source]¶

Bases: object

A container for a factory definition.

FSCollector ¶

class FSCollector[source]¶

Bases: _pytest.nodes.Collector

classmethod from_parent(parent, *, fspath)[source]¶: The public constructor

Function ¶

class Function[source]¶

Bases: _pytest.python.PyobjMixin, _pytest.nodes.Item

a Function Item is responsible for setting up and executing a Python test function.

originalname = None¶: original function name, without any decorations (for example parametrization adds a "[...]" suffix to function names).

New in version 3.0.

classmethod from_parent(parent, **kw)[source]¶: The public constructor

function¶: underlying python ‘function’ object

funcargnames¶: alias attribute for fixturenames for pre-2.3 compatibility

runtest() → None[source]¶: execute the underlying test function.

Item ¶

class Item[source]¶

Bases: _pytest.nodes.Node

a basic test invocation item. Note that for a single function there might be multiple test invocation items.

user_properties = None¶: user properties is a list of tuples (name, value) that holds user defined properties for this test.

add_report_section(when: str, key: str, content: str) → None[source]¶

Adds a new report section, similar to what’s done internally to add stdout and stderr captured output:

item.add_report_section("call", "stdout", "report section contents")

Parameters:	when (str) – One of the possible capture states, `"setup"`, `"call"`, `"teardown"`. key (str) – Name of the section, can be customized at will. Pytest uses `"stdout"` and `"stderr"` internally. content (str) – The full contents as a string.

MarkDecorator ¶

class MarkDecorator(mark)[source]¶

A decorator for test functions and test classes. When applied it will create Mark objects which are often created like this:

mark1 = pytest.mark.NAME              # simple MarkDecorator
mark2 = pytest.mark.NAME(name1=value) # parametrized MarkDecorator

and can then be applied as decorators to test functions:

@mark2
def test_function():
    pass

When a MarkDecorator instance is called it does the following:

If called with a single class as its only positional argument and no additional keyword arguments, it attaches itself to the class so it gets applied automatically to all test cases found in that class.
If called with a single function as its only positional argument and no additional keyword arguments, it attaches a MarkInfo object to the function, containing all the arguments already stored internally in the MarkDecorator.
When called in any other case, it performs a ‘fake construction’ call, i.e. it returns a new MarkDecorator instance with the original MarkDecorator’s content updated with the arguments passed to this call.

Note: The rules above prevent MarkDecorator objects from storing only a single function or class reference as their positional argument with no additional keyword or positional arguments.

name¶: alias for mark.name

args¶: alias for mark.args

kwargs¶: alias for mark.kwargs

with_args(*args, **kwargs)[source]¶

return a MarkDecorator with extra arguments added

unlike call this can be used even if the sole argument is a callable/class

Returns:	MarkDecorator

MarkGenerator ¶

class MarkGenerator[source]¶

Factory for MarkDecorator objects - exposed as a pytest.mark singleton instance. Example:

import pytest
@pytest.mark.slowtest
def test_function():
   pass

will set a ‘slowtest’ MarkInfo object on the test_function object.

Mark ¶

class Mark(name: str, args, kwargs, param_ids_from: Optional[Mark] = None, param_ids_generated: Optional[List[str]] = None)[source]¶

name = None¶: name of the mark

args = None¶: positional arguments of the mark decorator

kwargs = None¶: keyword arguments of the mark decorator

combined_with(other: _pytest.mark.structures.Mark) → _pytest.mark.structures.Mark[source]¶

Parameters:	other (Mark) – the mark to combine with
Return type:	Mark

combines by appending args and merging the mappings

Metafunc ¶

class Metafunc(definition: _pytest.python.FunctionDefinition, fixtureinfo: _pytest.fixtures.FuncFixtureInfo, config: _pytest.config.Config, cls=None, module=None)[source]¶

Metafunc objects are passed to the pytest_generate_tests hook. They help to inspect a test function and to generate tests according to test configuration or values specified in the class or module where a test function is defined.

config = None¶: access to the _pytest.config.Config object for the test session

module = None¶: the module object where the test function is defined in.

function = None¶: underlying python test function

fixturenames = None¶: set of fixture names required by the test function

cls = None¶: class object where the test function is defined in or None.

funcargnames¶: alias attribute for fixturenames for pre-2.3 compatibility

parametrize(argnames: Union[str, List[str], Tuple[str, ...]], argvalues: Iterable[Union[_pytest.mark.structures.ParameterSet, Sequence[object], object]], indirect: Union[bool, Sequence[str]] = False, ids: Union[Iterable[Union[None, str, float, int, bool]], Callable[[object], Optional[object]], None] = None, scope: Optional[str] = None, *, _param_mark: Optional[_pytest.mark.structures.Mark] = None) → None[source]

Parameters:

argnames – a comma-separated string denoting one or more argument names, or a list/tuple of argument strings.
argvalues – The list of argvalues determines how often a test is invoked with different argument values. If only one argname was specified argvalues is a list of values. If N argnames were specified, argvalues must be a list of N-tuples, where each tuple-element specifies a value for its respective argname.
indirect – The list of argnames or boolean. A list of arguments’ names (subset of argnames). If True the list contains all names from the argnames. Each argvalue corresponding to an argname in this list will be passed as request.param to its respective argname fixture function so that it can perform more expensive setups during the setup phase of a test rather than at collection time.
ids –

sequence of (or generator for) ids for argvalues,

or a callable to return part of the id for each argvalue.

With sequences (and generators like itertools.count()) the returned ids should be of type string, int, float, bool, or None. They are mapped to the corresponding index in argvalues. None means to use the auto-generated id.

If it is a callable it will be called for each entry in argvalues, and the return value is used as part of the auto-generated id for the whole set (where parts are joined with dashes (“-“)). This is useful to provide more specific ids for certain items, e.g. dates. Returning None will use an auto-generated id.

If no ids are provided they will be generated automatically from the argvalues.
scope – if specified it denotes the scope of the parameters. The scope is used for grouping tests by parameter instances. It will also override any fixture-function defined scope, allowing to set a dynamic scope using test context or configuration.

Module ¶

class Module[source]¶

Bases: _pytest.nodes.File, _pytest.python.PyCollector

Collector for test classes and functions.

collect()[source]¶: returns a list of children (items and collectors) for this collection node.

Node ¶

class Node[source]¶

base class for Collector and Item the test collection tree. Collector subclasses have children, Items are terminal nodes.

name = None¶: a unique name within the scope of the parent node

parent = None¶: the parent collector node.

fspath = None¶: filesystem path where this node was collected from (can be None)

keywords = None¶: keywords/markers collected from all scopes

own_markers = None¶: the marker objects belonging to this node

extra_keyword_matches = None¶: allow adding of extra keywords to use for matching

classmethod from_parent(parent: _pytest.nodes.Node, **kw)[source]¶

Public Constructor for Nodes

This indirection got introduced in order to enable removing the fragile logic from the node constructors.

Subclasses can use super().from_parent(...) when overriding the construction

Parameters:	parent – the parent node of this test Node

ihook¶: fspath sensitive hook proxy used to call pytest hooks

warn(warning)[source]¶

Issue a warning for this item.

Warnings will be displayed after the test session, unless explicitly suppressed

Parameters:	warning (Warning) – the warning instance to issue. Must be a subclass of PytestWarning.
Raises:	ValueError – if `warning` instance is not a subclass of PytestWarning.

Example usage:

node.warn(PytestWarning("some message"))

nodeid¶: a ::-separated string denoting its collection tree address.

listchain()[source]¶: return list of all parent collectors up to self, starting from root of collection tree.

add_marker(marker: Union[str, _pytest.mark.structures.MarkDecorator], append: bool = True) → None[source]¶

dynamically add a marker object to the node.

Parameters:	marker (`str` or `pytest.mark.*` object) – `append=True` whether to append the marker, if `False` insert at position `0`.

iter_markers(name=None)[source]¶

Parameters:	name – if given, filter the results by the name attribute

iterate over all markers of the node

for ... in iter_markers_with_node(name=None)[source]¶

Parameters:	name – if given, filter the results by the name attribute

iterate over all markers of the node returns sequence of tuples (node, mark)

get_closest_marker(name, default=None)[source]¶

return the first marker matching the name, from closest (for example function) to farther level (for example module level).

Parameters:	default – fallback return value of no marker was found name – name to filter by

listextrakeywords()[source]¶: Return a set of all extra keywords in self and any parents.

addfinalizer(fin)[source]¶

This method can only be called when this node is active in a setup chain, for example during self.setup().

getparent(cls)[source]¶: get the next parent node (including ourself) which is an instance of the given class

Parser ¶

class Parser[source]¶

Parser for command line arguments and ini-file values.

Variables:	extra_info – dict of generic param -> value to display in case there’s an error processing the command line arguments.

getgroup(name: str, description: str = '', after: Optional[str] = None) → _pytest.config.argparsing.OptionGroup[source]¶

get (or create) a named option Group.

Name:	name of the option group.
Description:	long description for –help output.
After:	name of other group, used for ordering –help output.

The returned group object has an addoption method with the same signature as parser.addoption but will be shown in the respective group in the output of pytest. --help.

addoption(*opts, **attrs) → None[source]¶

Opts:	option names, can be short or long options.
Attrs:	same attributes which the `add_argument()` function of the argparse library accepts.

After command line parsing options are available on the pytest config object via config.option.NAME where NAME is usually set by passing a dest attribute, for example addoption("--long", dest="NAME", ...).

parse_known_args(args: Sequence[Union[str, py._path.local.LocalPath]], namespace: Optional[argparse.Namespace] = None) → argparse.Namespace[source]¶: parses and returns a namespace object with known arguments at this point.

parse_known_and_unknown_args(args: Sequence[Union[str, py._path.local.LocalPath]], namespace: Optional[argparse.Namespace] = None) → Tuple[argparse.Namespace, List[str]][source]¶: parses and returns a namespace object with known arguments, and the remaining arguments unknown at this point.

addini(name: str, help: str, type: Optional[Literal['pathlist', 'args', 'linelist', 'bool']] = None, default=None) → None[source]¶

Name:	name of the ini-variable
Type:	type of the variable, can be `pathlist`, `args`, `linelist` or `bool`.
Default:	default value if no ini-file option exists but is queried.

The value of ini-variables can be retrieved via a call to config.getini(name).

PluginManager ¶

class PluginManager[source]¶

Core PluginManager class which manages registration of plugin objects and 1:N hook calling.

You can register new hooks by calling add_hookspecs(module_or_class). You can register plugin objects (which contain hooks) by calling register(plugin). The PluginManager is initialized with a prefix that is searched for in the names of the dict of registered plugin objects.

For debugging purposes you can call PluginManager.enable_tracing() which will subsequently send debug information to the trace helper.

register(plugin, name=None)[source]¶: Register a plugin and return its canonical name or None if the name is blocked from registering. Raise a ValueError if the plugin is already registered.

unregister(plugin=None, name=None)[source]¶: unregister a plugin object and all its contained hook implementations from internal data structures.

set_blocked(name)[source]¶: block registrations of the given name, unregister if already registered.

is_blocked(name)[source]¶: return True if the given plugin name is blocked.

add_hookspecs(module_or_class)[source]¶: add new hook specifications defined in the given module_or_class. Functions are recognized if they have been decorated accordingly.

get_plugins()[source]¶: return the set of registered plugins.

is_registered(plugin)[source]¶: Return True if the plugin is already registered.

get_canonical_name(plugin)[source]¶: Return canonical name for a plugin object. Note that a plugin may be registered under a different name which was specified by the caller of register(plugin, name). To obtain the name of an registered plugin use get_name(plugin) instead.

get_plugin(name)[source]¶: Return a plugin or None for the given name.

has_plugin(name)[source]¶: Return True if a plugin with the given name is registered.

get_name(plugin)[source]¶: Return name for registered plugin or None if not registered.

check_pending()[source]¶: Verify that all hooks which have not been verified against a hook specification are optional, otherwise raise PluginValidationError.

load_setuptools_entrypoints(group, name=None)[source]¶

Load modules from querying the specified setuptools group.

Parameters:	group (str) – entry point group to load plugins name (str) – if given, loads only plugins with the given `name`.
Return type:	int
Returns:	return the number of loaded plugins by this call.

list_plugin_distinfo()[source]¶: return list of distinfo/plugin tuples for all setuptools registered plugins.

list_name_plugin()[source]¶: return list of name/plugin pairs.

get_hookcallers(plugin)[source]¶: get all hook callers for the specified plugin.

add_hookcall_monitoring(before, after)[source]¶

add before/after tracing functions for all hooks and return an undo function which, when called, will remove the added tracers.

before(hook_name, hook_impls, kwargs) will be called ahead of all hook calls and receive a hookcaller instance, a list of HookImpl instances and the keyword arguments for the hook call.

after(outcome, hook_name, hook_impls, kwargs) receives the same arguments as before but also a pluggy.callers._Result object which represents the result of the overall hook call.

enable_tracing()[source]¶: enable tracing of hook calls and return an undo function.

subset_hook_caller(name, remove_plugins)[source]¶: Return a new hooks._HookCaller instance for the named method which manages calls to all registered plugins except the ones from remove_plugins.

PytestPluginManager ¶

class PytestPluginManager[source]¶

Bases: pluggy.manager.PluginManager

Overwrites pluggy.PluginManager to add pytest-specific functionality:

loading plugins from the command line, PYTEST_PLUGINS env variable and pytest_plugins global variables found in plugins being loaded;
conftest.py loading during start-up;

parse_hookimpl_opts(plugin, name)[source]¶

parse_hookspec_opts(module_or_class, name)[source]¶

register(plugin, name=None)[source]¶: Register a plugin and return its canonical name or None if the name is blocked from registering. Raise a ValueError if the plugin is already registered.

getplugin(name)[source]¶

hasplugin(name)[source]¶: Return True if the plugin with the given name is registered.

pytest_configure(config)[source]¶

consider_preparse(args, *, exclude_only=False)[source]¶

consider_pluginarg(arg)[source]¶

consider_conftest(conftestmodule)[source]¶

consider_env()[source]¶

consider_module(mod)[source]¶

import_plugin(modname, consider_entry_points=False)[source]¶: Imports a plugin with modname. If consider_entry_points is True, entry point names are also considered to find a plugin.

Session ¶

class Session[source]¶

Bases: _pytest.nodes.FSCollector

exception Interrupted¶

Bases: KeyboardInterrupt

signals an interrupted test run.

exception Failed¶

Bases: Exception

signals a stop as failed test run.

for ... in collect()[source]¶: returns a list of children (items and collectors) for this collection node.

TestReport ¶

class TestReport[source]¶

Basic test report object (also used for setup and teardown calls if they fail).

nodeid = None¶: normalized collection node id

location = None¶: a (filesystempath, lineno, domaininfo) tuple indicating the actual location of a test item - it might be different from the collected one e.g. if a method is inherited from a different module.

keywords = None¶: a name -> value dictionary containing all keywords and markers associated with a test invocation.

outcome = None¶: test outcome, always one of “passed”, “failed”, “skipped”.

longrepr = None¶: None or a failure representation.

when = None¶: one of ‘setup’, ‘call’, ‘teardown’ to indicate runtest phase.

user_properties = None¶: user properties is a list of tuples (name, value) that holds user defined properties of the test

sections = []¶: list of pairs (str, str) of extra information which needs to marshallable. Used by pytest to add captured text from stdout and stderr, but may be used by other plugins to add arbitrary information to reports.

duration = None¶: time it took to run just the test

classmethod from_item_and_call(item, call) → _pytest.reports.TestReport[source]¶: Factory method to create and fill a TestReport with standard item and call info.

caplog¶: Return captured log lines, if log capturing is enabled

New in version 3.5.

capstderr¶: Return captured text from stderr, if capturing is enabled

New in version 3.0.

capstdout¶: Return captured text from stdout, if capturing is enabled

New in version 3.0.

count_towards_summary¶

Experimental

Returns True if this report should be counted towards the totals shown at the end of the test session: “1 passed, 1 failure, etc”.

Note

This function is considered experimental, so beware that it is subject to changes even in patch releases.

head_line¶

Experimental

Returns the head line shown with longrepr output for this report, more commonly during traceback representation during failures:

________ Test.foo ________

In the example above, the head_line is “Test.foo”.

Note

This function is considered experimental, so beware that it is subject to changes even in patch releases.

longreprtext¶: Read-only property that returns the full string representation of longrepr.

New in version 3.0.

_Result ¶

class _Result(result, excinfo)[source]¶

result¶: Get the result(s) for this hook call (DEPRECATED in favor of get_result()).

force_result(result)[source]¶

Force the result(s) to result.

If the hook was marked as a firstresult a single value should be set otherwise set a (modified) list of results. Any exceptions found during invocation will be deleted.

get_result()[source]¶

Get the result(s) for this hook call.

If the hook was marked as a firstresult only a single value will be returned otherwise a list of results.

Special Variables ¶

pytest treats some global variables in a special manner when defined in a test module.

collect_ignore ¶

Tutorial: Customizing test collection

Can be declared in conftest.py files to exclude test directories or modules. Needs to be list[str].

collect_ignore = ["setup.py"]

collect_ignore_glob ¶

Tutorial: Customizing test collection

Can be declared in conftest.py files to exclude test directories or modules with Unix shell-style wildcards. Needs to be list[str] where str can contain glob patterns.

collect_ignore_glob = ["*_ignore.py"]

pytest_plugins ¶

Tutorial: Requiring/Loading plugins in a test module or conftest file

Can be declared at the global level in test modules and conftest.py files to register additional plugins. Can be either a str or Sequence[str].

pytest_plugins = "myapp.testsupport.myplugin"

pytest_plugins = ("myapp.testsupport.tools", "myapp.testsupport.regression")

pytestmark ¶

Tutorial: Marking whole classes or modules

Can be declared at the global level in test modules to apply one or more marks to all test functions and methods. Can be either a single mark or a list of marks.

import pytest

pytestmark = pytest.mark.webtest

import pytest

pytestmark = [pytest.mark.integration, pytest.mark.slow]

PYTEST_DONT_REWRITE (module docstring)¶

The text PYTEST_DONT_REWRITE can be add to any module docstring to disable assertion rewriting for that module.

Environment Variables ¶

Environment variables that can be used to change pytest’s behavior.

PYTEST_ADDOPTS ¶

This contains a command-line (parsed by the py:mod:shlex module) that will be prepended to the command line given by the user, see How to change command line options defaults for more information.

PYTEST_DEBUG ¶

When set, pytest will print tracing and debug information.

PYTEST_PLUGINS ¶

Contains comma-separated list of modules that should be loaded as plugins:

export PYTEST_PLUGINS=mymodule.plugin,xdist

PYTEST_DISABLE_PLUGIN_AUTOLOAD ¶

When set, disables plugin auto-loading through setuptools entrypoints. Only explicitly specified plugins will be loaded.

PYTEST_CURRENT_TEST ¶

This is not meant to be set by users, but is set by pytest internally with the name of the current test so other processes can inspect it, see PYTEST_CURRENT_TEST environment variable for more information.

Exceptions ¶

UsageError ¶

class UsageError[source]¶: error in pytest usage or invocation

Configuration Options ¶

Here is a list of builtin configuration options that may be written in a pytest.ini, tox.ini or setup.cfg file, usually located at the root of your repository. All options must be under a [pytest] section ([tool:pytest] for setup.cfg files).

Warning

Usage of setup.cfg is not recommended unless for very simple use cases. .cfg files use a different parser than pytest.ini and tox.ini which might cause hard to track down problems. When possible, it is recommended to use the latter files to hold your pytest configuration.

Configuration file options may be overwritten in the command-line by using -o/--override, which can also be passed multiple times. The expected format is name=value. For example:

pytest -o console_output_style=classic -o cache_dir=/tmp/mycache

addopts¶

Add the specified OPTS to the set of command line arguments as if they had been specified by the user. Example: if you have this ini file content:

# content of pytest.ini
[pytest]
addopts = --maxfail=2 -rf  # exit after 2 failures, report fail info

issuing pytest test_hello.py actually means:

pytest --maxfail=2 -rf test_hello.py

Default is to add no options.

cache_dir¶: Sets a directory where stores content of cache plugin. Default directory is .pytest_cache which is created in rootdir. Directory may be relative or absolute path. If setting relative path, then directory is created relative to rootdir. Additionally path may contain environment variables, that will be expanded. For more information about cache plugin please refer to Cache: working with cross-testrun state.

confcutdir¶: Sets a directory where search upwards for conftest.py files stops. By default, pytest will stop searching for conftest.py files upwards from pytest.ini/tox.ini/setup.cfg of the project if any, or up to the file-system root.

console_output_style¶

Sets the console output style while running tests:

classic: classic pytest output.
progress: like classic pytest output, but with a progress indicator.
count: like progress, but shows progress as the number of tests completed instead of a percent.

The default is progress, but you can fallback to classic if you prefer or the new mode is causing unexpected problems:

# content of pytest.ini
[pytest]
console_output_style = classic

doctest_encoding¶: Default encoding to use to decode text files with docstrings. See how pytest handles doctests.

doctest_optionflags¶: One or more doctest flag names from the standard doctest module. See how pytest handles doctests.

empty_parameter_set_mark¶

Allows to pick the action for empty parametersets in parameterization

skip skips tests with an empty parameterset (default)
xfail marks tests with an empty parameterset as xfail(run=False)
fail_at_collect raises an exception if parametrize collects an empty parameter set

# content of pytest.ini
[pytest]
empty_parameter_set_mark = xfail

Note

The default value of this option is planned to change to xfail in future releases as this is considered less error prone, see #3155 for more details.

faulthandler_timeout¶

Dumps the tracebacks of all threads if a test takes longer than X seconds to run (including fixture setup and teardown). Implemented using the faulthandler.dump_traceback_later function, so all caveats there apply.

# content of pytest.ini
[pytest]
faulthandler_timeout=5

For more information please refer to Fault Handler.

filterwarnings¶

Sets a list of filters and actions that should be taken for matched warnings. By default all warnings emitted during the test session will be displayed in a summary at the end of the test session.

# content of pytest.ini
[pytest]
filterwarnings =
    error
    ignore::DeprecationWarning

This tells pytest to ignore deprecation warnings and turn all other warnings into errors. For more information please refer to Warnings Capture.

junit_duration_report¶

New in version 4.1.

Configures how durations are recorded into the JUnit XML report:

total (the default): duration times reported include setup, call, and teardown times.
call: duration times reported include only call times, excluding setup and teardown.

[pytest]
junit_duration_report = call

junit_family¶

New in version 4.2.

Configures the format of the generated JUnit XML file. The possible options are:

xunit1 (or legacy): produces old style output, compatible with the xunit 1.0 format. This is the default.
xunit2: produces xunit 2.0 style output,

which should be more compatible with latest Jenkins versions.

[pytest]
junit_family = xunit2

junit_logging¶

New in version 3.5.

Changed in version 5.4: log, all, out-err options added.

Configures if captured output should be written to the JUnit XML file. Valid values are:

log: write only logging captured output.
system-out: write captured stdout contents.
system-err: write captured stderr contents.
out-err: write both captured stdout and stderr contents.
all: write captured logging, stdout and stderr contents.
no (the default): no captured output is written.

[pytest]
junit_logging = system-out

junit_log_passing_tests¶

New in version 4.6.

If junit_logging != "no", configures if the captured output should be written to the JUnit XML file for passing tests. Default is True.

[pytest]
junit_log_passing_tests = False

junit_suite_name¶

To set the name of the root test suite xml item, you can configure the junit_suite_name option in your config file:

[pytest]
junit_suite_name = my_suite

log_auto_indent¶

Allow selective auto-indentation of multiline log messages.

Supports command line option --log-auto-indent [value] and config option log_auto_indent = [value] to set the auto-indentation behavior for all logging.

[value] can be:

True or “On” - Dynamically auto-indent multiline log messages
False or “Off” or 0 - Do not auto-indent multiline log messages (the default behavior)
[positive integer] - auto-indent multiline log messages by [value] spaces

[pytest]
log_auto_indent = False

Supports passing kwarg extra={"auto_indent": [value]} to calls to logging.log() to specify auto-indentation behavior for a specific entry in the log. extra kwarg overrides the value specified on the command line or in the config.

log_cli¶

Enable log display during test run (also known as “live logging”). The default is False.

[pytest]
log_cli = True

log_cli_date_format¶

Sets a time.strftime()-compatible string that will be used when formatting dates for live logging.

[pytest]
log_cli_date_format = %Y-%m-%d %H:%M:%S

For more information, see Live Logs.

log_cli_format¶

Sets a logging-compatible string used to format live logging messages.

[pytest]
log_cli_format = %(asctime)s %(levelname)s %(message)s

For more information, see Live Logs.

log_cli_level¶

Sets the minimum log message level that should be captured for live logging. The integer value or the names of the levels can be used.

[pytest]
log_cli_level = INFO

For more information, see Live Logs.

log_date_format¶

Sets a time.strftime()-compatible string that will be used when formatting dates for logging capture.

[pytest]
log_date_format = %Y-%m-%d %H:%M:%S