runbot/runbot_merge/models
Xavier Morel 391a323efd [IMP] runbot_merge: avoid updating every PR and staging on status change
Because the only condition between PRs and statuses is sharing a
repository (and kinda not even that for stagings), adding or removing
a status on a repository would try to recompute the statuses/state of
essentially every staging in the history of the project, and a
significant fraction of the PRs, leading to tens of thousands of
queries, minutes of computation, and even OOMs of the HTTP workers as
we'd load the PRs, their batches, and the stagings into memory to
update them, then load more things because of what depends on PR
statuses, etc...

But there is no reason to touch any of the closed or merged PRs, or
the completed (deactivated) stagings. And in fact there is every
reason *not* to.

Implementing a search-only m2m on each object in order to restrict the
set of PRs/stagings relevant to a change reduces the number of queries
to a few hundreds, the run time to seconds, and the memory increase to
unnoticeable. The change still takes some time because the RD project
currently has about 7000 open PRs, 4500 of which target
odoo/odoo (which is our test case here), but that is nothing compared
to the 164000 total PRs to odoo/odoo out of some 250000 PRs for the RD
project.

And though they're likely less of an issue as they don't recurse quite
as much, the >120000 stagings during the project's history are not to
be ignored, when then number of *active* stagings at any one time is
at most the number of branches, which is half a dozen to a dozen.

For very basic numbers, as of committing this change, creating a
status in odoo/odoo (RD), optional on PRs and ignored on statuses,
on my current machine (7530U with 32GB RAM):

- without this change, 4835 queries, 37s of sql, 65s of non-SQL, RSS
  climbs to 2258128 (2.15GiB)
- with this change, 758 queries, 1.46s SQL, 2.25s non-SQL, RSS climbs
  to 187088 (182MiB)

Fixes #1067
2025-02-28 16:12:40 +01:00
..
backport [FIX] *: double forwardport when adding a PR to an existing batch 2025-02-11 14:27:53 +01:00
crons [IMP] runbot_merge: auto-trigger cron for issue closing 2024-12-16 09:11:19 +01:00
ir_cron [IMP] runbot_merge: add cron trigger button 2025-02-18 11:45:56 +01:00
project_freeze [REM] runbot_merge: freeze wizard auto-refresh 2024-12-02 16:32:53 +01:00
staging_cancel [ADD] runbot_merge: stagings canceling wizard 2022-12-08 10:46:22 +01:00
__init__.py [IMP] runbot_merge: add cron trigger button 2025-02-18 11:45:56 +01:00
batch.py [IMP] *: remove dead imports 2025-02-28 16:12:40 +01:00
commands.py [IMP] *: remove dead imports 2025-02-28 16:12:40 +01:00
events_sources.py [ADD] *: per-repository webhook secret 2024-06-06 11:07:57 +02:00
ir_actions.py [IMP] runbot_merge: add json & requests to server actions context 2023-02-20 10:13:05 +01:00
ir_ui_view.py [IMP] runbot_merge: suppress view validation warning 2024-09-27 12:53:51 +02:00
mail_thread.py [MERGE] bot from 16.0 to 17.0 2024-08-12 13:13:03 +02:00
patcher.py [IMP] *: remove dead imports 2025-02-28 16:12:40 +01:00
project.py [FIX] project creation: handling of mergebot info 2024-12-02 16:32:53 +01:00
pull_requests.py [IMP] runbot_merge: avoid updating every PR and staging on status change 2025-02-28 16:12:40 +01:00
res_partner.py [IMP] *: remove dead imports 2025-02-28 16:12:40 +01:00
stagings_create.py [IMP] *: remove dead imports 2025-02-28 16:12:40 +01:00
utils.py [IMP] *: remove dead imports 2025-02-28 16:12:40 +01:00