runbot/forwardport
Xavier Morel e20277c6ad [FIX] forwardport: storage of old garbage, repo cache sizes
Since the forwardport bot works off of PRs, when it was created
leveraging the magic refs of github (refs/pull/*/head) seemed
sensible: when updating the cache repo, the magic refs would be
updated alongside and the forward-porting would have all the commits
available.

Turns out there are a few issues with this:

- the magic refs have become a bit unreliable, so we need a fallback
  (b1c16fce8768080d30430f4e6d3788b60ce13de7)
- the forwardport bot only needs the commits on a transient basis, but
  the magic refs live forever and diverge from all other content
  (since we rarely `merge` PRs), for a large repo with lots of PRs
  this leads to a significant inflation in size of repo (both on-disk
  and objects count) e.g. odoo/odoo has about 25% more objects
  with the pull refs included (3486550 to 4395319) and takes nearly
  50% more space on disk (1.21G to 1.77)

As a result, we can just stop configuring the pull refs entirely, and
instead fetch the heads (~refs) we need as we need them. This can be a
touch more expensive at times as it requires two `fetch` calls, and
we'll have a few redundant calls as every forward port of a chain will
require a fetch of the root's head, *however* it will avoid retrieving
PR data for e.g. prs to master, as they don't get forward-ported, they
also won't get parasite updates from PRs being ignored or eventually
closed.

Eventually this could be optimised further by

- performing an update of the cache repo at the start of the cron iff
  there are batches to port
- creating a temp clone for the batch
- fetching the heads of all PRs to forward port in the temp clone in a
  single `fetch`
- performing all the ports by cloning the temp clone (and not
  `fetch`-ing into those)
- then cleaning up the temp clone after having cleaned up individual
  forward port clones

Small followup for #489
2022-12-01 10:57:32 +01:00
..
changelog [FIX] forwardport: fix pinging on forwardport PRs 2022-12-01 10:57:32 +01:00
data [IMP] forwardport: gc/maintenance of local repo caches 2022-12-01 10:57:32 +01:00
migrations/13.0.1.1 [IMP] forwardport: reliability of PR reminders 2020-05-26 15:56:36 +02:00
models [FIX] forwardport: storage of old garbage, repo cache sizes 2022-12-01 10:57:32 +01:00
tests [FIX] forwardport: fix pinging on forwardport PRs 2022-12-01 10:57:32 +01:00
__init__.py [ADD] mergebot, forwardbot: changelog 2021-10-20 15:16:48 +02:00
__manifest__.py [IMP] mergebot, forwardbot: various UI bits 2022-06-30 15:07:49 +02:00
controllers.py [ADD] mergebot, forwardbot: changelog 2021-10-20 15:16:48 +02:00