Commit Graph

243 Commits

Author SHA1 Message Date
Xavier Morel
1b7040cd26 [IMP] forwardport: logging during cherrypick process
* fix small error which generated an extra commit in case of conflict
  probably (would create a `temp` commit, then put the conflict
  information on an empty commit on top of it). Avoids committing the
  cherrypick though a probable alternative would be to commit the
  message with `amend`.
* improve the cherrypick header: instead of one log line per commit,
  put all commits within the sole header log line (with a newline),
  should make things less noisy
* put *all* log records below the correct logger (two of them were on
  the toplevel logger)
* log a purported inflateInit error as warning, should be sent to
  Sentry instead of having to wait for people to wonder why the thing
  is completely broken
2021-01-15 14:19:17 +01:00
Xavier Morel
ba1a8ee089 [IMP] runbot_merge, forwardport: update some logging
Downgrade an error and a warning to info, and upgrade two warnings to
error.

Point is to improve logic of log levels & sentry visibility.

Fixes #258
2021-01-13 16:12:35 +01:00
Xavier Morel
e175609950 [IMP] forwardport: unmodified fw automatically inherit overrides
Before this change, a CI override would have to be replicated on most
/ all forward-ports of the base PR. This was intentional to see how it
would shake out, the answer being that it's rather annoying.

Also add a `statuses_full` computed field on PRs for the aggregate
status: the existing `statuses` field is just a copy of the commit
statuses which I didn't remember I kept free of the overrides so the
commit statuses could be displayed "as-is" in the backend (the
overrides are displayed separately). And while at it fix the PR
dashboard to use that new field: that was basically the intention but
then I went on to use the "wrong" field hence #433.

Mebbe the UI part should be displayed using a computed M2M (?)
as a table or as tags instead? This m2m could indicate whether the
status is an override or an "intrinsic" status.

Also removed some dead code:

* leftover from the removed tagging feature (removed the tag
  manipulation but forgot some of the setup / computations)
* unused local variables
* an empty skipped test case

Fixes #439.

Fixes #433.
2021-01-13 16:11:14 +01:00
Xavier Morel
d751cf46a0 [IMP] forwardport: add logging to branch reinsertion
When a new branch is created between other branches, the process to
try and look for forward-port PRs to create to "fill in" currently has
no logging so it's very difficult to figure out why it decides not to
do something.

Add some logging to the process to try and better understand what
happens.

Fixes #441
2021-01-12 12:54:37 +01:00
Xavier Morel
4b5dfc2005 [FIX] forwardport: broken fp if intermediate commit requires renamelimit=0
On a cherrypick failing due to renamelimit issues, the cherrypick
would be retried after resetting the target to its *original* commit.

This only works correctly if the first commit works after a retry, if
a latter commit has to be retried then it gets re-cherry-picked onto
the target branch rather than its own parent.

Fix by remembering the previously successful cherry-picked commit and
resetting to *that*. However I can't really test it because it's
rather hard to get into a situation where the rename detection fails
using synthetic tests.

While at it, clean the logs by stripping the "performing inexact
rename detection" stuff from all stderr (both the CherrypickError from
which it was already stripped and the debug messages).

Fixes #444
2021-01-11 14:43:36 +01:00
Xavier Morel
5c19342bf6 [CHG] runbot_merge, forwardport: remove labelling
Because github materialises every labels change in the
timeline (interspersed with comments), the increasing labels churn
contributes to PRs being difficult to read and review.

This change removes the update of labels on PRs, instead the mergebot
will automatically send a comment to created PRs serving as a
notification that the PR was noticed & providing a link to the
mergebot's dashboard for that PR where users should be able to see the
PR state in detail in case they wonder what's what.

Lots of tests had to be edited to:

- remove any check on the labels of the PR
- add checks on the PR dashboard (to ensure that they're at least on
  the correct "view")
- add a helper to handle the comment now added to every PR by the 'bot
- since that helper is needed by both mergebot and forwardbot, the
  utils modules were unified and moved out of the odoo modules

Probably relevant note: no test was added for the dashboard
ACL, though since I had to explicitly unset the group on the repo used
for tests for things to work it looks to me like it at least excludes
people just fine.

Fixes #419
2020-11-20 07:41:54 +01:00
Xavier Morel
be7788be0a [FIX] forwardport: git conflict markers
2.29 changed the formatting of conflict labels (the stuff added at the
end of the closing conflict marker):
6cceea19eb

Used to be

    >>>>>>> <commit>... <text>

now is

    >>>>>>> <commit> (<text>)

Just ignore everything after the commit hash.
2020-11-20 07:41:45 +01:00
Xavier Morel
7ab7eed27e [IMP] forwardport: lower level of waiting for batch to be ready
Waiting for the current batch to validate before forward-porting it is
normal, it's useful information but should not be warned about.
2020-07-30 13:20:41 +02:00
Xavier-Do
04ad4f6b67 [FIX] forwardport: nicer skipci message 2020-07-19 14:32:21 +02:00
Xavier Morel
22e18e752b [ADD] runbot_merge: allow overriding statuses
Adds an `override` mergebot command. The ability to override is set on
an individual per-context per-repository basis, similar to but
independent from review rights. That is, a given individual may be
able to override the status X on repository A and unable to do so on
repository B.

Overrides are stored in the same format as regular statuses, but
independent from them in order to persist them across builds.

Only PR statuses can be overridden, statuses which are overridable on
PRs would simply not be required on stagings.

An alternative to implementing this feature in the mergebot would be
to add it to individual status-generating tools on a per-need
basis.

Pros of that alternative:

* display the correct status on PRs, currently the PR will be failing
  status-wise (on github) but correct as far as the mergebot is
  concerned
* remove complexity from the mergebot

Cons of that alternative:

* each status-generating tool would have to implement some sort of ACL
  system
* each status-generating tool would have to receive & parse PR
  comments
* each status-generating tool would have to maintain per-pr state in
  order to track overrides

Some sort of helper library / framework ought make that rather easy
though. It could also be linked into the central provisioning system
thing.

Closes #376
2020-07-14 13:34:05 +02:00
Xavier Morel
ce92ffc346 [ADD] forwardport: policy on PR form 2020-07-14 13:34:05 +02:00
Xavier Morel
db9e86f760 [IMP] forwardport: reliability of PR reminders
The exponential backoff offsets from the write_date of the children
PRs, however it doesn't reset, so the offsetting gets bumped up way
more than originally expected or designed if the child PRs are under
active development for some reason.

Fix this by adding a field to specifically record the date of merge of
a PR, and check that feature against the backoff offset. This should
provide more regular and reliable backoff.

Fixes #369
2020-05-26 15:56:36 +02:00
Xavier Morel
4d528465d9 [FIX] forwardport: incorrect / misleading conflict message
Given a PR batch getting forward-ported together, if one of the PRs
has a conflict the others should be considered "in conflict" as well,
and should have a note pointing in that direction and indicating that
the PR should be approved the normal way eventually. Which they do.

However, the message is confusing as it gets bolted on the normal
non-conflicting message, either noting that it's part of a chain
or (worse because it gives conflicting indication) the "terminal"
message recommending using the forwardbot to approve of the entire
chain.

I've no idea why I did it that way instead of just adding a case to
the conditional, and the commit message provides no indication. But
perform that change, it seems innocuous, hopefully there weren't good
reasons I forgot about for doing it the other way around.

Fixes #367
2020-05-20 07:43:53 +02:00
Xavier Morel
d99d1c2ad6 [ADD] forwardport: ability to skip CI when forward porting
Provides a `skipci` command to PR reviewers. This makes it so the
followup PRs (after the first one) get created immediately, without
waiting for CI to succeed on a given forward-port PR.

This can be useful if for some reason a change *must* be merged in
branch N+1 before it can be merged in branch N.

Fixes #363
2020-04-17 08:09:01 +02:00
xmo-odoo
49c1d960fa
[FIX] forwardport: race condition on PR creation (#357)
e9e08fec3c attempted to fix the issue
but obviously failed as it still occurs: when creating a PR through
the API, it's possible that the webhook gets triggered fast enough the
transaction creating the PR from the webhook commits before we get
around to creating our own PR from the API call. In which case the
forward port process aborts.

The process is re-run later on and generally succeeds fully, but we're
left with a dangling PR we created but couldn't do anything with as
its use broke.

This issue seems to be getting more frequent so it's becoming quite
important to fix it. Therefore we give our Raging Bull a Big Gun and
now he has 20 attack *cough cough* we lock the bloody table down
tight (only allow concurrent `SELECT`) until we've got the PR back and
we've done the updates we need to it and nobody can mess with it...
probably.

This is not ideal as it's going to block updates to completely
unrelated PRs but it doesn't seem like postgres really allows for
locking out creations without locking out the rest, short of using
advisory locks maybe? E.g. in the `create` override get a
`pg_advisory_xact_lock_shared`, then get a `pg_advisory_xact_lock` in
the forward-port process that way we're just blocking the concurrent
creation of PRs during forward port, but creations don't block one
another and we don't block updates.

Application-level locks wouldn't really work as the 'bot could be
deployed using multi-worker scenarios so we'd need cross-process locks
or something.

Hopefully fixes #352
2020-04-07 15:02:39 +02:00
Xavier Morel
1440aec251 [IMP] forwardport: exponential backoff on reminders
The reminder feature is a bit brutal when people go on holidays or
whatever as it keeps commenting every day.

This should comment every day for a few days, then quickly taper down.

Closes #285
2020-03-16 15:03:11 +01:00
Xavier Morel
ddf3f5013e [FIX] forwardport: fix reminder cron to avoid multiple messages
Because the reminder cron uses groupby to "merge" open PRs related to the
same source and send a single message for all of them (e.g. PR 6548
forward-ported to 6587 and 6591 should have a single reminder message per
day not one per descendant), the PRs with the same source need to be
consecutive in the search sequence.

However there was no order specified so the search would yield PRs in id
order or something, and if there happened to be an other forward-port PR
inbetween the descendants of the original would not get coalesced and would
therefore trigger a message per descendant per day (doubling or tripling the
intended spam rate).

Ordering by source_id should fix the issue as it ought make all PRs
forward-ported from the same thing contiguous, and therefore grouped
together before sending reminder messages.

An alternatively solution would be to use `groupby` instead of `search` but
it would require more modifications as we'd need to re-browse the sources
and descendants, etc...

First part of fixing #285 as this is likely why odoo/enterprise#7204 got
spammed so much: its descendants were odoo/enterprise#7367 and
odoo/enterprise#7369 and it just so happens that odoo/enterprise#7368 was
*also* a forward port PR, causing the issue explained above.
2020-03-16 15:03:11 +01:00
Xavier Morel
f60bc1d067 [IMP] forwardport: on conflict note previous FP can be r+'d
Closes #294
2020-03-16 15:03:11 +01:00
Xavier Morel
e82de3136b [IMP] runbot_merge, forwardport: unify tagging
Genericise runbot_merge's tagging (move states to the "UI" but only
store / manage actual tags), and remove forwardport.tagging as it's
now redundant.

Closes #232
2020-03-16 10:53:19 +01:00
Christophe Monniez
3e40c38e89 [FIX] forwardport: fix forward-port Odoo wiki link 2020-03-10 08:00:07 +01:00
Xavier Morel
0b73e5c640 [IMP] fwbot: warning on r+ for failed PR
approving a PR which failed CI should trigger a feedback message since
6cb58a322d (#158), the code has not been
removed and the tests still pass.

However fwbot r+ would go through its own process for r+ which would
explain why that feedback is sometimes gone / lost (cf #327 and #336).

* make fwbot r+ delegate to mergebot r+
* add dedicated logging for this operation to better analyze
  post-mortem
* automatically ping the reviewer to specifically tell them they're idiots
* move the feedback item out of the state change bit, send it even if
  it's a useless r+ (because it's already r+'d)
* add a test for forward-ports

Closes #327, closes #336
2020-03-10 07:58:09 +01:00
Xavier Morel
aa2248e379 [IMP] forwardport: close command
This is useful as the author of the original PR doesn't necessarily
have (write) access to the repository where the forward-port PR was
created. As a result, while they can r+ the PR they're unable to close
it (via github's interface).

Since the forwardport bot created the PR, it can also close it, which
seems like a useful feature.

Closes #341
2020-03-04 11:56:01 +01:00
Xavier Morel
dde59b9fe6 [IMP] runbot_merge, forwardport: better signed-off-by handling
Remove original-signed-off-by, doesn't actually seem useful given the
semantics of signed-off-by according to the kernel doc'. Plus it
didn't actually work as the intent was to keep the signoff of the
original PR in the forward-port, but that signoff is not part of the
commit we're cherrypicking (it gets added on the fly when the commit
is merged).

Therefore explicitly get the ack-chain into the PR: when merging an FP
PR, try to integrate the signoff of the original PR, that of the final
FP pr, and while at it that of the last explicit update in the commit
chain (e.g. in case there's been a conflict or something).

Fixes #284
2020-03-04 11:56:01 +01:00
Xavier Morel
6b5731f175 [FIX] forwardport: PR update through a closed PR
Fixes #328
2020-02-26 16:21:24 +01:00
Paul Morelle
1f9713cca0 [FIX] forwardport: update the wiki link
On github wiki, the link changed from
https://github.com/odoo/odoo/wiki/Mergebot
to
https://github.com/odoo/odoo/wiki/Mergebot-and-Forwardbot

Reflect these changes in the comments posted by the bot.
2020-02-12 14:06:36 +01:00
Xavier Morel
742e3219a6 [IMP] runbot_merge: make review rights repo-dependent
As the odds of having more projects or more repos with different
requirements in the same project, the need to have different sets of
reviewers for different repositories increases.

As a result, rather than be trivial boolean flags the review info
should probably depend on the user / partner and the repo. Turns out
the permission checks had already been extracted into their own
function so most of the mess comes from testing utilities which went
and configured their review rights as needed.

Incidentally it might be that the test suite could just use something
like a sequence of commoditized accounts which get configured as
needed and not even looked at unless they're used.
2020-02-11 08:07:57 +01:00
Xavier Morel
53e46001ba [IMP] runbot_merge, forwardport: notify when main crons are off
During freezes it can be useful to notify viewers that nothing is
going to forward port or merge for a while, and that this is
intentional (not something that's broken).

Fixes #307
2020-02-11 08:07:57 +01:00
Xavier Morel
9d661480fc [IMP] runbot_merge: split staging cron in two
The staging cron was already essentially split between "check if one
of the stagings is successful (and merge it)" and "check if we should
create a staging" as these were two separate loops in the cron.

But it might be useful to disable these two operations separately
e.g. we might want to stop the creation of new staging but let the
existing stagings complete.

The actual splitting is easy but it turns out a bunch of tests were
"optimised" to only run the merge cron. Most of them didn't blow up
but it seems more prudent to fix them all.

fixes odoo/runbot#310
2020-02-11 08:07:57 +01:00
Xavier Morel
fd24b791b3 [FIX] forwardport: prefix of working copy when forward porting
The missing dash makes it hard to differentiate the target's name and
the random crap TemporaryDirectory tacks on. Add a separator dash.
2020-02-07 08:29:55 +01:00
Xavier Morel
35dbfd2c12 [FIX] runbot_merge: status deduplication (maybe)
Despite the existing dedup' sometimes the "xxx failed on this
forward-port PR" would still get multiplicated due to split builds
e.g. in odoo/odoo#43935 4 such messages appear within ~5 minutes, then
one more 10mn later.

This is despite all of them having the same "build" (target_url) and
status (failure). Since the description is the only thing that's not
logged I assume that's the field which varies and makes the dedup'
fail. Therefore:

* add the description to the logging (when getting a status ping)
* exclude the description when checking if a new status should be
  taken in account or ignored: the build (and thus url) should change
  on rebuild

Hopefully fixes #281
2020-01-31 10:40:59 +01:00
Xavier Morel
3f61dc9ce4 [IMP] forwardport: warning message when co-dep PR has a conflict
* Add some more information as to why the user *should* do on the PR
  the message is printed on, the previous message left that to their
  imagination
* The PR selection was *completely* wrong as it would select the old
  PRs which really isn't what we want. And turns out there's no good
  reason to create & send the feedback in the loop creating the
  forward-port prs, that can be moved to a followup loop where we have
  created hopefully created all the forward-port PRs.

  Also technically we could do even better than currently and remap
  the prs mapped to conflict data to the new PRs and know exactly
  which of the forward-ported PRs is faulty, but that seems overkill
  for now.
2020-01-30 13:56:01 +01:00
Xavier Morel
e065b7aba7 [FIX] forwardport: reintroduce final newline in a comment
Tests check for it, easier to fix one line than change all the
relevant tests.
2020-01-30 09:41:16 +01:00
Xavier Morel
9aac1b4a3e [ADD] forwardport: special handling of adding branches to projects
If a new branch is added to a project, there's an issue with *ongoing*
forward ports (forward ports which were not merged before the branch
was forked from an existing one): the new branch gets "skipped" and
might be missing some fixes until those are noticed and backported.

This commit hooks into updating projects to try and see if the update
consists of adding a branch inside the sequence, in which case it
tries to find the FP sequences to update and queues up new
"intermediate" forward ports to insert into the existing sequences.

Note: had to increase the cron socket limit to 2mn as 1mn blew up the
big staging cron in the test (after all the forward-port PRs are
approved).

Fixes #262

[FIX]
2020-01-29 15:59:43 +01:00
Xavier Morel
43e6f5e5ec [FIX] forwardport: logging.warn -> logging.warning
logging.warn is deprecated but I've a hard time shaking the habit.
2020-01-29 15:59:43 +01:00
Xavier Morel
a0fe545d86 [IMP] forwardport: warn when linked PR failed to FW
Before this change, if multiple co-dependent PRs get forward-ported
and one of them has a conflict the notice on the others is very
limited: they're tagged as `conflict` but there is no other
information provided in the PR description or in the subsequent
message.

Add a small warning to these other PRs, for clarity.

Closes #302
2020-01-29 15:55:06 +01:00
Xavier Morel
5a8f821de0 [FIX] forwardport: recover from failure to create forward port PR
Currently if the creation of a forward-port pull request fails:

* the branches are left un-cleaned
* preceding PRs are left open
* the PR whose creation failed may or may not have actually failed,
  and may or may not still be open

We need to delete the forward port branches anyway, and IIRC
that *should* automatically close the PR. Sadly making it so github
predictably / reliably blows up when trying to create a PR via the API
is difficult so this is essentially untestable.

Closes #296
2020-01-29 15:43:57 +01:00
Xavier Morel
5fae01e53e [IMP] forwardport: feedback on commands
The forwardbot's command parsing was missing feedback when trying to
use commands without the proper ACL. This would make some situations
of comments seemingly being lost hard to diagnose.

Closes #300
2020-01-29 15:09:09 +01:00
Xavier Morel
60574d6fe2 [ADD] forwardport: support for branch filtering
Add handling of branch filtering to the forwardport module:

* don't forward port (and trigger an error) when trying to port
  PRs to different next targets
* otherwise port normally

e.g. given a project with repos A and B and branches a, b and c, with
branch b being excluded from repo B:

* a PR merged into A.a will be forward-ported to A.b and A.c
* a PR merged into B.a will be forward-ported to B.c (skipping the
  excluded B.b)
* a PR set merged into (A.a, B.a) will *not* be forward-ported, and a
  message will be posted to each PR denoting the incompatibility
2020-01-24 13:39:14 +01:00
Xavier Morel
6b3c81a177 [FIX] make pytest cross-module-runnable
The pytest suite had been partially unified between mergebot and
forwardport but because of session-scoped modules it could not run
across those.

Make the db cache lazy and able to cache multiple databases, and move
the "current required module" to function scoped, this way things
should (and seem to) work properly on runs involving mergebot & fwbot.

Next step: xdist! (need to randomise repo names for that, probably).
2020-01-24 13:39:14 +01:00
Xavier Morel
7dfa973b57 [IMP] runbot_merge, forwardport: move required statuses to repository
Allows more flexibility in project composition as different
repositories can each have their own CI passes.
2020-01-24 13:39:14 +01:00
Xavier Morel
a30a1c08e7 [FIX] runbot_merge, forwardport: missing model descriptions
Really can't be arsed to care, just want to remove the warning.
2020-01-13 08:41:54 +01:00
Xavier Morel
060709ac42 [ADD] forwardport: missing access rights 2020-01-13 08:39:47 +01:00
Xavier Morel
7d79ad8d3e [FIX] forwardport: don't keep the commit date
This is not super useful and causes issues with runbot as it uses
commit dates to decide how "old" branches are, and ignores (doesn't
create when it scans them) branches more than a month old.

So the forwardport branches will be completely ignored (not considered
let alone tested) if we happen to forward-port a PR last updated more
than a month ago. Which is somewhat inconvenient.

Closes #274
2019-12-19 15:25:23 +01:00
Xavier Morel
c7588c32fd [FIX] forwardport: triggers conflict markers detection
Conflict marker checker looks for 7 <s or >s in a row. The forwardport
contains exactly that. So replace one of the literal signs by a hex
escape.
2019-11-26 16:23:36 +01:00
Xavier Morel
a45f7260fa [IMP] forwardport: better handle authorship information
Before this:

* the forwardport bot always sets itself as the committer when it
  creates a forward-port branch
* when creating squashed / conflict commits, it becomes both author
  and committer

This is not great as it loses a fair amount of authorship information
and doesn't properly give credit where credit is due. Improve this in
the following ways:

* use the original authorship information as-is when forward-porting
  commits
* the the forward port fails, use the original authorship information
  as-is if there's a single commit to forward port
* otherwise if there's only a single author / committer across the
  branch being forwardported use that, if there are multiple give up
  and use the identity of the 'bot, since the branch will probably
  need to be re-done in full the authorship information of the
  placeholder commit should not matter overly much

Uses git's magic ENV variables as that's pretty much the only way to
properly override the COMMITTER date: conf items only provide for
author and committer *identity* (name and email), and while `commit`
allows overriding the *authorship* date (`--date`) it doesn't provide
any option for the *commit* date.

Fixes #255
2019-11-21 08:10:39 +01:00
Xavier Morel
8e67aed792 [IMP] runbot_merge: limit spamming on PR close
When closing a PR, github completely separates the events "close the
PR" and "comment on the PR" (even when using "comment and close" in
the UI, a feature which isn't even available in the API). It doesn't
aggregate the notifications either, so users following the PR for
one reason or another get 2 notifications / mails every time a PR
gets merged, which is a lot of traffic, even more so with
forward-ported PRs multiplying the amount of PRs users are involved
in.

The comment on top of the closure itself is useful though: it allows
tracking exactly where and how the PR was merged from the PR, this
information should not be lost.

While more involved than a simple comment, *deployments* seem like
a suitable solution: they allow providing links as permanent
information / metadata on the PRs, and apparently don't trigger
notifications to users.

Therefore, modify the "close" method so it doesn't do
"comment-and-close", and provide a way to close PRs with non-comment
feedback: when the feedback's message is structured (parsable as
json) assume it's intended as deployment-bound notifications.

TODO: maybe add more keys to the feedback event payload, though in my
      tests (odoo/runbot#222) none of the deployment metadata
      outside of "environment" and "target_url" is listed on the PR
      UI

Fixes #224
2019-11-21 08:10:39 +01:00
Xavier Morel
7c46a2006f [FIX] forwardport: the fix
Of course I forgot the most relevant bit
2019-10-18 12:01:47 +02:00
Xavier Morel
5f8041552b [FIX] forwardport: apparently git/refs/heads can fuzzy-match
If the ref we asked for does not exist, github apparently decides to
fall-back to prefix-matching. So if we're trying to delete
already-deleted branch A and someone called their branch A-x we're
going to get it as a result.

Thankfully they were apparently smart enough to return a list even if
there's only a single fuzzy match. So if we get a list (instead of a
dict) as response to git/refs/heads assume the branch was already
deleted as if we got a 404.
2019-10-18 11:22:13 +02:00
Xavier Morel
13d76fdfb9 [FIX] forwardport: the fix
it's a per-call parameter not a per-instance one
2019-10-18 08:11:27 +02:00
Xavier Morel
c1cef0c18b [FIX] forwardport: gh api raises by default, avoid that here 2019-10-18 08:02:04 +02:00
Xavier Morel
ea410ab6d1 [ADD] forwardport: automatic branch deleter
If a PR is *merged*, enqueue it for deletion (with a 2 weeks delay).

Mainly to avoid FW branches staying around long after they've been
merged (possibly eventually closed?), will also clean up regular
merged branches, including historical merges forgotten by their
author.

Fixes #230
2019-10-17 11:55:20 +02:00
Xavier Morel
401787b7ae [FIX] forwardport: co-dependent FPs where one PR is updated
In the case where we have a co-dependent forward port (co-dependent
PRs got forward-ported, which they should be together) where *one* of
the PRs got explicitly updated, the batch would "fall into a hole"
being handled as neither "this is part of a forward-port sequence" nor
"this is a new merge to forward-port" (the latter being the proper
one).

Modify & remove guards which checked that either no or all PRs in a
batch have parents: should be either all or not all.

Fixes #231
2019-10-15 08:54:25 +02:00
Xavier Morel
8937f379ce [FIX] forwardport: fix the fix
Re-add token field name when creating the tagging object, as it's a
required field....
2019-10-14 12:01:12 +02:00
Xavier Morel
97999318be [FIX] forwardport: don't use token field for tags update
Turns out tagging PRs requires a pretty significant level of ACLs
which we may not want to give to the forwardbot?

Anyway use the mergebot ACLs (which already include tagging) for this.
2019-10-14 10:09:48 +02:00
Xavier Morel
8f3f773eef [IMP] *: testing helpers
* add a sorted method on fake models
* fix recordset equality to ignore ids order
* when creating commits on a ref, add a param to only *update* the ref
  (forcefully): when simulating a force-push we don't want to *create*
  a ref as that might silently be done in the wrong repository entirely
* fix pytest.skip call at the module level, not sure where it came
  from and why I missed it until now
2019-10-14 10:09:48 +02:00
Xavier Morel
2eb4ece50a [FIX] forwardport: title-ing of PRs
Due to the title formatting of FP PRs, we'd get incorrectly formatted
commit messages if the PR was *merged* (rather than squashed /
fast-forwarded) due to either "merge" or "rebase-merge" integration
mode: in that case the PR message would be used as message for the
merge commit and that'd be along the lines of "Forward Port of #xxx to
<somebranch> (failed)", followed by the old PR message (e.g. see this
commit message).

* re-extract and reuse original PR title, just prefix with "[FW]"
* finally add support for tagging, and use that to tag the PRs,
  especially for the failed / conflict marker which is quite important

Closes #229
2019-10-11 13:05:36 +02:00
Xavier Morel
bad016b830 [FIX] forwardport: queue reliability changes
Previous version would break if _process_item itself committed which
was bad
2019-10-11 09:13:55 +02:00
Xavier Morel
3ce3dd9569 [IMP] forwardbot: show FP PRs in reminder message
When posting a reminder that there are open / waiting forward ports on
a source PR, also post *which* PRs those are.

While at it, move the cron code in a proper python file (so we can use
stuff from odoo.tools), and fix display_name so we can straight use
display_name as a github ref' ({owner}/{repo}#{number}). This impacts
log-grepping but it seems like an improvement nonetheless.

Closes odoo/runbot#228
2019-10-11 09:13:55 +02:00
Xavier Morel
036ae3a8ee [IMP] forwardbot: reduce length of fw branch name
* shorten the postfix, forwardbot is now a bigram!
* shorten the uniquifier: go from 5 to 3 bytes, and use urlsafe base64
  that way we only have a 4-char uniquifier instead of 8
* while at it, fix deprecated calls to logging.warn (should be
  logging.warning)

Fixes #226
2019-10-10 11:37:27 +02:00
Xavier Morel
d453943252 [IMP] *: unify gh test API between runbot and fw-bot
The fw-bot testing API should improve the perfs of mergebot tests
somewhat (less waiting around for instance).

The code has been updated to the bare minimum (context-managing repos,
change to PRs and replacing rolenames by explicit token provisions)
but extra facilities were used to avoid changing *everything*
e.g. make_commit (singular), automatic generation of PR refs, ...

The tests should eventually be updated to remove these.

Also remove the local fake / mock. Being so much faster is a huge
draw, but I don't really want to spend more time updating it,
especially when fwbot doesn't get to take advantage. A local /
lightweight fake github (as an external service over http) might
eventually be a good idea though, and more applicable (including to
third-parties).
2019-10-10 10:11:48 +02:00
Xavier Morel
557878afe9 [IMP] forwardport: processing queue reliability
The queue would get items to process one at a time, process, commit,
and go to the next. However this is an issue if one of the item fails
systematically for some reason (aka it's not just a transient
failure): the cron fails, then restarts at the exact same point, and
fails again with the same issue, leading to following items never
getting processed.

Fix by getting all the queue contents at once, processing them one by
one and "skipping" any item which fails (leaving it in place so it can
get re-processed later).

That way, even if an item causes issues, the rest of the queue gets
processed normally. The interruption was an issue following
odoo/enterprise#5670 not getting properly updated in the
backend (backend didn't get notified of the last two updates /
force-push to the PR, so it was trying to forward-port a commit which
didn't exist - and failing).
2019-10-10 08:41:33 +02:00
Xavier Morel
fafa7ef437 [IMP] *: attempt to avoid some of the FP spam
Attempt to avoid some of the comment spam by dedup-ing input (only
signaling when the status actually changes and ignoring identity
transformations) and in case of failing CI keeping the last failed
status and not signaling on the next update if it's the same failure.

Closes #225
2019-10-07 16:38:14 +02:00
Xavier Morel
e9e08fec3c [IMP] forwardport: fix creation of FP PR in some testing cases
When running tests on some machines, it's apparently possible for the
PR-creation webhook to come back before the PR creation request has,
leading to the creation of the PR from the API call duplicating that of
the webhook and blowing up.

To fix, immediately commit the transaction then check if we already have
the PR we just created in the system, and only create it explicitly if
not.
2019-10-03 10:02:37 +02:00
Xavier Morel
c5d68c20f4 [IMP] forwardport: add feedback when being wrong
User should probably be warned when they try to set the limit ("up
to") in a context where it's going to be ignored:

* on a forward port PR (should be set on the source)
* on a merged source (should be set before the PR is merged)

Closes #213
2019-10-02 17:49:54 +02:00
Xavier Morel
01ff6d13db [IMP] forwardport: move limits tests into their own file
test_simple is starting to get pretty big and we'll need some more
tests for #213
2019-10-02 17:49:54 +02:00
Xavier Morel
cd29c648e8 [IMP] forwardport: don't ping for CI failure on closed PRs
Fixes #210
2019-10-02 17:49:54 +02:00
Xavier Morel
9c3d12c964 [FIX] forwardbot: selection of ancestor in FP tail ping
In selecting the parent commits to list on the last PR, we would miss
the *first* forward-port of the sequence. Not sure why we added a
detrimental check on source_id there.

Also add a missing space between "chain" and "containing" in the case
where there's at least one forward-port PR other than the final one.

Fixes #212
2019-10-02 17:49:54 +02:00
Xavier Morel
37cf6accb7 [FIX] forwardport: FP PR getting CI'd after initial FP check
Test is probably more complex than necessary (thinking about it, the
failed staging is probably unnecessary) but that triggers the issue
and matches the original scenario.

The problem was really with new CI events being received on the last
forward-port PR of a sequence: previous PRs would have a child PR so
the check would abort, but for the last PR it would go through, fail
to find an active batch, then blow up as it tries to create a
forwardport.batch without an actual batch.

Change this to use the existence of an inactive batch not linked to a
staging as a flag that the PR has been processed and forward-ported.

Closes #218
2019-10-01 20:51:31 +02:00
Xavier Morel
78ad4b4e4b [IMP] runbot_merge, forwardport: consolidate conftests
Converge the pytest setups of runbot_merge and forwardport a bit
more (the goal is obviously to eventually share the infrastructure so
they run the same way).
2019-09-23 13:54:42 +02:00
Xavier Morel
63bef8b7ab [IMP] forwardport: notify when FP PR gets de-parented
If a PR is explicitly updated, it gets converted to a normal
PR[0]. Before this, users had no indication that this had happened and
might be wondering what they're supposed to do (or try to r+ via the
forwardbot, which doesn't work on a root PR).

[0] to an extent: the PR still has a source and might have children,
    in which case the followups will be created from the source &
    existing followups should be updated to match

Closes #206
2019-09-23 12:54:22 +02:00
Xavier Morel
8de6273498 [IMP] forwardport: unweird singleton conflicting commits
When creating the conflicting commit of a single commit PR, reuse the
original commit's meta-information so the developer / fixer can more
easily update it in-place.

Closes #204
2019-09-21 15:23:42 +02:00
Xavier Morel
446b11a28f [IMP] forwardport: link FP PRs to both root and source
In the case where an FP sequence is interrupted (e.g. there was a
conflict during one of the intermediate steps), followups get linked
to the original source but don't get linked to the "interruption" PR
which is a bit confusing.

Link FP PRs to both source and root if they're different.
2019-09-21 15:23:42 +02:00
Xavier Morel
6d4923928a [IMP] forwardport: improve disable fp UI
* always allow specifying the PR's own branch as a forward-port limit
  / target (even if deactivated or disabled)
* add an "ignore" alias to "up to <pr target>" for clarity
* add dedicated feedback when deactivating forward-port of a PR

Fixes #191
2019-09-21 15:23:42 +02:00
Xavier Morel
66d65ba550 [IMP] runbot_merge, forwardport: variable-user feedback
Having all the feedback be sent by the mergebot user (github_token) is
confusing. Add a way to specify which field of project should be used to
source the token used when sending feedback.

Fixes #190
2019-09-21 15:23:42 +02:00
Xavier Morel
e345d4d0d0 [FIX] forwardport: breakage in previous commit
leftover unnecessary change in
52699d901a broke one of the tests
2019-09-18 11:48:01 +02:00
Xavier Morel
52699d901a [IMP] forwardport: ping on CI failure
It's especially important as users / assignees don't get
pinged *during* the forwardport process.

closes #203
2019-09-18 08:32:38 +02:00
Xavier Morel
ee8f81be2a [CHG] forwardport: automatically delegate original PR author on FP PRs
This way the original author can r+ the forward ports if they succeed
(and probably requires no attention).

Closes #195
2019-09-17 14:43:21 +02:00
Xavier Morel
73f27873a3 [IMP] forwardport: less spammy reminder cron
* don't warn on every PR on the dot every week, instead wait for the
  PRs to be "sufficiently old" (at least 3 days)
* after discussion with bugfix, the reminder ping should be sent every
  day following the PR being "old enough"
* run the cron every day instead of every week
* add an override context key for test purposes

Closes #198
2019-09-16 15:28:03 +02:00
Xavier Morel
bf1d6c510d [CHG] forwardport: let PR author set the FP limit
Closes #200
2019-09-16 15:07:40 +02:00
Xavier Morel
a1a7d65ebe [IMP] forwardport: move working copy to the cache dir
Working copies were created in tempdir under the assumption that
they're, well, temporary.

However after thinking about it more there are two issues with this:

* tempdirs might not be in the same FS as the cache dir, meaning
  meaning `git clone` can't hardlink the repo objects and has to
  copy them
* tempdirs are often on RAM-backed tmpfs, which is not great when we're
  filling them with multiple GB worth of git repository...
2019-09-16 15:07:40 +02:00
Xavier Morel
f8da17994a [FIX] forwardport: not being properly notified on last FP of a seq
If the default limit of a forward-port sequence is not a valid
target (either disabled or not actually forward-ported to), the last
effective forward port in a sequence will be commented on as any
intermediate PR rather than get a proper ping and r+ instructions.

Also remove a bunch of leftover prints in the tests.

Fixes #192
2019-09-16 15:07:40 +02:00
Xavier Morel
8976f9e310 [FIX] forwardport: attempt no renamelimit & filter out messages
* a high renamelimit increases the cherrypick runtime and can
  apparently fail for reasons other than actual conflicts (cf
  odoo/odoo#36893) so first attempt to cherrypick with the default
  renamelimit, and fallback to renamelimit=0 if that fails
* in that case, filter out the spam of "progress" messages about
  inexact rename detection as it's not really useful. There seems to
  be no built-in way to suppress these.

Closes #196
2019-09-16 15:05:42 +02:00
Xavier Morel
0a64699070 [FIX] forwardport: cached repo setup
Turns out bare repositories (unlike normal ones) don't have any
refspecs by default. So adding the PR refspec would... replace the
default behaviour (apparently?) and the base branches would never get
fetched at all.

--mirror looks to be an other option as it has a default refspec but
it's a bit of an odd duck: it has pull requests in refs/pull but not
all of them? And open / closed doesn't seem to be a factor.
2019-09-12 15:03:45 +02:00
Xavier Morel
60f4db8687 [FIX] forwardport: forgot token when getting PR commits
So would break any time it needed to fetch the commits of a PR in a
private repo.
2019-09-12 14:35:56 +02:00
Xavier Morel
426bc69ea1 [FIX] forwardport: ask github what the PR's commits are
Attempts to use rev-list seem to work locally but they fail *a lot*
when live. The previous attempt to fix it in
0f4c22490c made things worse rather than
better once deployed.

Give up on that (at least for now?), and just ask github what the PR's
commits are then cherrypick exactly that.
2019-09-12 11:58:10 +02:00
Martin Trigaux
48c8e53df2 [FIX] forwardport: clarify forward port message
The ping message was not clear
- don't know if anything is needed from the author
- don't know if the last step in the chain

Ping the authors only when something needs to be done (error or last
step).

Do not ping non-reviewers as they will not have the rights to r+ or
modify the PR anyway

Closes #192
2019-09-12 09:05:49 +02:00
Xavier Morel
0f4c22490c [IMP] forwardport: refs
There's an issue of too many commits being selected for
cherry-picking. It still isn't quite clear, but the forward ports to
11.0 systematically seem to get everything from
5045b5f4c346e60c9b127403ef1fde453d49396a to the PR head, and that
commit was one of the first to land after the last merge-based forward
port.

So the commits selection seems to behave as if the commits range was
`..origin/pull/36692` rather than the `origin/10.0..origin/pull/36692`
which is passed to rev-list. This might be an issue of concurrent
access / update of the refs (though I can't reproduce it locally,
missing refs generate an error).

This change attempts to "pin" the local branch in the working copy
rather than go and get it from the repo. It's unclear that this will
change / fix anything, but it might just work.

Also stop creating shared clones, that seems like an unnecessary
risk (and might be the actual source of the issue).
2019-09-12 09:05:49 +02:00
Xavier Morel
a69ed7996a [FIX] forwardport: change method for conflicting working copy
The original method was to `git diff | git apply` in order to get a
complete overview of conflicts generated by the forward port (if
any).

However this turns out to have a huge issue in the presence of renamed
or removed files: in that case `git apply` will simply not do
anything, and fail with a completely clean working copy. Which is very
much undesirable.

-> alternative method, squash the PR to a single commit then
cherry-pick that single commit, this should provide us with proper
conflicts & their markers.

Also add tests for conflicts due to deleted files...
2019-09-12 09:05:49 +02:00
Xavier Morel
0e3d873f0f [IMP] forwardport: logging & rename during cherrypick
* add slightly better logging to the cherrypick process (init)
* cherrypick with an infinite renamelimit, seems fine according to
  linus[0] and significantly reduces "unsuccessful" forward ports

[0] http://git.661346.n2.nabble.com/Merging-limitations-after-directory-renames-interesting-test-repo-td6041103.html#a6041833
2019-09-12 09:05:49 +02:00
Xavier Morel
4ce4a3bda7 [FIX] forwardport: disable creation of draft PRs
That was only indicative and it doesn't work on private repos.
2019-09-12 09:05:49 +02:00
Xavier Morel
4fcebed563 [FIX] forwardport: PR message
* avoid losing original PR message
* add PR source to description
2019-09-10 09:54:25 +02:00
Xavier Morel
79777662df [FIX] forwardport: clone op and add logging 2019-09-10 09:28:30 +02:00
Xavier Morel
f671dcc828 [ADD] forwardbot
* Cherrypicking is handrolled because there seems to be no easy way to
  programmatically edit commit messages during the cherrypicking
  sequence: `-n` basically squashes all commits and `-e` invokes a
  subprocess. `-e` with `VISUAL=false` kinda sorta works (in that it
  interrupts the process before each commit), however there doesn't
  seem to be clean status codes so it's difficult to know if the
  cherrypick failed or if it's just waiting for a commit of this step.

  Instead, cherrypick commits individually then edit / rewrite their
  commit messages:

  * add a reference to the original commit
  * convert signed-off-by to something else as the original commit was
    signed off but not necessarily this one

* Can't assign users when creating PRs: only repository collaborators
  or people who commented on the issue / PR (which we're in the
  process of creating) can be assigned.

  PR authors are as likely to be collaborators as not, and we can have
  non-collaborator reviewers. So pinging via a regular comment seems
  less fraught as a way to notify users.
2019-09-05 10:00:07 +02:00