runbot/runbot_merge/tests
Xavier Morel 3cc87051dd [FIX] runbot_merge: avoid repeatedly warning about the same failures
The mergebot has a feature to ping users when an approved PR or
forward-port suffers from a CI failure, as those PRs might be somewhat
unattended (so the author needs to be warned explicitly).

Because the runbot can send the same failure information multiple
times, the mergebot also has a *deduplication* feature, however this
deduplication feature was too weak to handle the case where the PR has
2+ failures e.g. ci and linting as it only stores the last-seen
failure, and there would be two different failures here.

Worse, because the validation step looks at all required statuses, in
that case it would send a failure ping message for each failed
status *on each inbound status*: first it'd notify about the ci
failure and store that, then it'd see the linting failure, check
against the previous (ci), consider it a new failure, notify, and
store that. Rinse and repeat every time runbot sends a ci *or* lint
failure, leading to a lot of dumb and useless spam.

Fix by storing the entire current failure state (a map of context:
status) instead of just the last-seen status data.

Note: includes a backwards-compatibility shim where we just convert a
stored status into a full `{context: status}` map. This uses the
"current context" because we don't have the original, but if it was a
different context it's not going to match anyway (the target_url
should be different) and if it was the same context then there's a
chance we skip sending a redundant notification.

Fixes #435
2021-01-13 16:12:35 +01:00
..
conftest.py [IMP] runbot_merge: split staging cron in two 2020-02-11 08:07:57 +01:00
README.rst [IMP] runbot_merge: test readme & remove leftover pass 2018-09-03 13:16:36 +02:00
test_basic.py [FIX] runbot_merge: avoid repeatedly warning about the same failures 2021-01-13 16:12:35 +01:00
test_by_branch.py [CHG] runbot_merge: branch_ids -> branch_filter 2020-10-02 15:28:36 +02:00
test_disabled_branch.py [CHG] runbot_merge, forwardport: remove labelling 2020-11-20 07:41:54 +01:00
test_multirepo.py [CHG] runbot_merge, forwardport: remove labelling 2020-11-20 07:41:54 +01:00
test_oddities.py [CHG] runbot_merge, forwardport: remove labelling 2020-11-20 07:41:54 +01:00
test_status_overrides.py [FIX] mergebot: handle multiples of each command 2021-01-12 09:54:45 +01:00

Execute this test suite using pytest.

The default mode is to run tests locally using a mock github.com.

See the docstring of remote.py for instructions to run against github "actual"
(including remote-specific options) and the end of this file for a sample.

Shared properties running tests, regardless of the github implementation:

* test should be run from the root of the runbot repository providing the
  name of this module aka ``pytest runbot_merge`` or
  ``python -mpytest runbot_merge``
* a database name to use must be provided using ``--db``, the database should
  not exist beforehand
* the addons path must be specified using ``--addons-path``, both "runbot" and
  the standard addons (odoo/addons) must be provided explicitly

See pytest's documentation for other options, I would recommend ``-rXs``,
``-v`` and ``--showlocals``.

When running "remote" tests as they take a very long time (hours) ``-x``
(aka ``--maxfail=1``) and ``--ff`` (run previously failed first) is also
recommended unless e.g. you run the tests overnight.

``pytest.ini`` sample
---------------------

.. code:: ini

    [github]
    owner = test-org
    token = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

    [role_reviewer]
    name = Dick Bong
    user = loginb
    token = bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

    [role_self_reviewer]
    name = Fanny Chmelar
    user = loginc
    token = cccccccccccccccccccccccccccccccccccccccc

    [role_other]
    name = Harry Baals
    user = logind
    token = dddddddddddddddddddddddddddddddddddddddd