Commit Graph

131 Commits

Author SHA1 Message Date
xmo-odoo
4206d75256
[IMP] runbot_merge: wait for (and log) repo update / staging visibility
The race condition which prompted STAGING_SLEEP rears its ugly head
again: when pushing a base repo and its dependents, it's possible for
the update to the base repo's new head to take much longer to be visible
than the dependents (or so it seems?).

In this case, CI might pick up the correct dependent but pick an older /
incorrect revision of the base, leading to a staging failing for no good
reason.

This change uses info/refs to check for the updated staging head to be
visible at the repo level after it's been set / updated via the API. It
assumes repos are in topological order.
2019-04-29 12:42:54 +02:00
Xavier Morel
aa614c6077 [IMP] runbot_merge: more reliable blocked attribute
Use the proper / actual "is there any stageable PR" query to check if
a PR is blocked as well, that way they shoudn't be diverging all the
time even if it might make PR.blocked a bit more expensive.

fixes #111
2019-04-05 08:23:56 +02:00
Xavier Morel
e12e6db653 [IMP] runbot_merge: lock commit to update its status in hook
A status being updated on a commit is a read/modify/update, meaning
it's possible for somebody else (including a concurrent event?) to
concurrently update the commit and conflict leading to the webhook
blowing up, which is undesirable as it's a data loss (whereas if it
blows up on the other side e.g. in the cron's commit processor the
cron will just take it up next iteration).
2019-03-11 14:54:58 +01:00
Xavier Morel
f5d783eb4b [FIX] runbot_merge: error in template
8d011e03d2 contains poopy bits in the
template, fix them.
2019-03-11 14:51:15 +01:00
Xavier Morel
e0320664f9 [ADD] runbot_merge: sentry integration
Might eventually extract / generalise, but for now it's simpler to
just do it in runbot_merge's post_load, that way there's no setup
change (just a small bit of configuration), and it's only enabled on
the instances runbot_merge is installed on.

fixes #97, closes #103
2019-03-07 11:56:45 +01:00
xmo-odoo
4944d6a503
[FIX] runbot_merge: small typo in error message 2019-03-06 22:50:55 +01:00
Xavier Morel
8d011e03d2 [IMP] runbot_merge: styling of awaiting and blocked lists on dashboard 2019-03-06 09:46:57 +01:00
Xavier Morel
48e08b657b [IMP] runbot_merge: send feedback on CI failure following r+
Will comment any time a statuses update folds to a CI failure on a
reviewed pull request. Might be somewhat spammy, we'll see.

No notification if the PR is not reviewed yet.

fixes #87
2019-03-05 09:03:26 +01:00
Xavier Morel
5aa9f5a567 [IMP] runbot_merge: extract commit validation to cron
Before this, impacting a commit's statuses on the relevant PR or
staging would be performed immediatly / inline with its
consumption. This, however, is problematic if we want to implement
additional processing like #87 (and possibly though probably not #52):
webhook handlers should be kept short and fast, feeding back into
github would not be acceptable.

- flag commits as needing processing instead of processing them
  immediately, this uses a partial index as it looks like the
  recommended / proper way to index a boolean column in which one of
  the values is searched much more than the other (todo: eventually
  check if that actually does anythnig)
- add a new cron for commits processing
- alter tests so they use this new cron (mostly by migrating them to
  `run_crons` though not solely as some still need more detailed
  management to properly check intermediate steps)

Fix an issue with closing a staged PR while at it (the "merging" tag
would potentially never be removed).
2019-03-05 08:07:19 +01:00
Xavier Morel
360d0e17ca [IMP] runbot_merge: don't quote signoff
Proper RFC5322 makes for much noisier messages, and seems completely
unnecessary as examples of sign-off on the internet don't quote spaces
/ names.

closes #102
2019-03-04 13:17:10 +01:00
Xavier Morel
1f30af4345 [IMP] runbot_merge: dashboard clarity
* split out truly awaiting PRs from those waiting on an event of some
  sort
* if a staging is active but doesn't have a state yet, it should be
  considered pending not cancelled

closes #74
2019-03-04 12:11:34 +01:00
Xavier Morel
b699ea7f47 [FIX] runbot_merge: validate PRs on head update
If a PR gets sync'd to a known-valid commit, it should be marked as
valid rather than get in this weird state where it's merely open but
github knows it passes CI.

Fixes #72
2019-03-04 10:34:40 +01:00
Xavier Morel
1d2c264728 [FIX] runbot_merge: properly update squash flag on PR retarget
closes #82
2019-03-04 09:52:21 +01:00
Xavier Morel
8ab72ce8d1 [FIX] runbot_merge: gap in PR names in awaiting list
Repeatedly fixed it live, but apparently still forgot to fix it in the
source.
2019-03-01 21:34:31 +01:00
Xavier Morel
c693a7f841 [ADD] runbot_merge: button to manually cancel stagings
This is somewhat less useful with runbot's fail-fast as a runbot
failure (false positive or not) will now very quickly trigger an end
to the current staging.

Still, could be of use.

closes #89
2019-03-01 17:29:37 +01:00
Xavier Morel
eea3211f2b [IMP] runbot_merge: add logging to PR sync and reset error PRs to open
The choice to keep sync'd PRs in error means it's possible to update
the code and re-run the PR directly without it going through review &
CI again, which is a bit odd. Remove the special case and always reset
a sync'd PR to opened for clarity and simplicity.

closes #71
closes #83
2019-03-01 16:46:09 +01:00
Xavier Morel
c34e8ca083 [FIX] runbot_merge: race condition between closes #x and merging/FF
Turns out skipping locks is not very useful when there are no locks
being held because we only touch the PRs *after* the merge has been
applied.

So finally do that, lock all of a staging's PRs before we try to
fast-forward the relevant repositories, so a close command coming back
from github (from having seen the closes #xxx annotation) doesn't
screw us over.
2019-03-01 16:46:09 +01:00
Xavier Morel
0cd587fce7 [FIX] runbot_merge: don't blow the fetch loop when a PR has no label
No test because I don't understand how / why it's triggered, it's just
that some PRs don't have a label. I assumed the issue occurred when
the source branch or even repo (cross-repo PR) was deleted, but it
doesn't seem to trigger the issue (or in any case not in as short a
time as a test, maybe GH eventually does some vacuuming which causes
the issue?

Anyway we may eventually want to reclaim these PRs (allowing a lack of
label and treating them like the patch-\d labels: with no semantic
value) however the simplest thing to do for now is to just ignore the
corresponding PR.

closes #101
2019-03-01 16:42:58 +01:00
Xavier Morel
55ece42d8f [IMP] runbot_merge: delete repos being created in remote tests
In remote tests, if the deletion of a test repository fails (because
gh glitch) or the repo creation succeeded but reported a failure (for
some reason) the entire run is hosed because every test trying to
create a similarly named repository will explode.

Alter repomaker to just try to delete the repo, unless --no-delete
mode in which case just skip any further test trying to use the same
repository (not deleting the repo is the entire point of --no-delete,
as its purpose is the ability to do post-mortem debugging on
repository state).

closes #99
2019-03-01 16:42:57 +01:00
Xavier Morel
79b03a6995 [IMP] runbot_merge: retry FF on failure in case it's transient
Further improvements are possible, but that seems like a good
start (hopefully).

closes #94
2019-03-01 16:42:57 +01:00
Xavier Morel
42046cb21c [IMP] runbot_merge: logging on github requests failures
Github is subject to a fair amount of transient failures, which are
currently ill-logged: an exception is raised and the caller /
responsible might eventually log something, but it's not really
formalised and centralised, and is thus inconvenient to try and
post-mortem issues with github's support.

Change this such that *almost* all github API calls get extensively
logged (status, reason, all headers, body) on failure.

Also automatically sets debug logging for odoo in local tests, and
alter the fake response constructor thing so it doesn't set a json
mimetype when the body is not valid json.

Closes #98
2019-03-01 16:42:57 +01:00
Xavier Morel
f579b28a93 [FIX] runbot_merge: logging.warn is deprecated, use logging.warning 2019-02-28 13:50:25 +01:00
Christophe Simonis
19ffcdd4a2 [ADD] runbot_merge: sign off commits by reviewer
closes #50
closes #54
2019-02-26 13:36:46 +01:00
Xavier Morel
41cdc7e5f9 [IMP] runbot_merge: add number to PR search view
The number is probably the most common search criteria for PRs (to
track their status / issues). Having to go through custom filters to
find one is a pain in the ass.

Already done live by editing the view, but means it's getting lost
every time the module gets updated.

closes #73
2019-01-25 15:49:20 +01:00
Xavier Morel
4490c8f119 [IMP] runbot_merge: debug logging when rebasing 2019-01-25 15:45:38 +01:00
Christophe Simonis
e169934e61 [IMP]: runbot_merge: sort unready PRs alphabetically
The `_order` of pull requests are just the `number`.

Allow test `test_two_of_three_unready` to be reliable as both unready
PRs have the same number.
2019-01-25 15:45:12 +01:00
Christophe Monniez
549452f12d [IMP] runbot_merge: send feedback when merge method is changed
When a user changes the merge method via github messages, no feedback is
sent. This could lead to strange behavior, for example when a user try
to joke with the mergebot like this:

> robodoo are you goin ti merge my PR rogntudju !

This sets the merge method to "merge" and the user is not aware of it.
2018-12-13 13:28:20 +01:00
Xavier Morel
e01ad86171 [IMP] runbot_merge: make feedback more deterministic
Before this change, the order of PRs to list in an "unready" feedback
message was whatever the DB returned which could vary. This change
fixes the order by applying model order.
2018-12-12 15:29:59 +01:00
Xavier Morel
8a1b3466a7 [IMP] runbot_merge: send integratin failure comment via feedback queue
If a transient github failure makes the integration fail but also
makes the following reset fail the entire staging process would be
cancelled (and operations so far rollbacked) except for the failure
comment which would be effected, as in odoo/odoo#26380.

By pushing the comment to the feedback queue, if the reset fails the
comment is rollbacked and "unqueued".
2018-11-27 11:53:10 +01:00
xmo-odoo
e468d7116e
[FIX] runbot_merge: ignore comment edition & deletion
As well as review edition & dismissal.

Closes #53
2018-11-26 10:28:27 +01:00
xmo-odoo
6655e0ea5b
[ADD] runbot_merge: better integration controls
Closes #48
2018-11-26 10:28:13 +01:00
Xavier Morel
c814ce8f34 [FIX] runbot_merge: tagging on closed PR
Closed tagging was broken since the raw-sql alterations of the close
hook: because it's raw SQL, the write() method doesn't get invoked
anymore and as a result the tagging feedback record is not created,
and never executed.

Add a test to check for the PR's proper tagging, and fix this issue by
explicitly creating a tagging record.

Closes #49
2018-11-22 16:06:15 +01:00
Xavier Morel
46c460fd9b [IMP] runbot_merge: send comments asynchronously via feedback table
Hopefully this finally fixes the double commenting
issue (e.g. odoo/odoo#28160). This seems (according to reading the
logs and also logic) like a failure (concurrency of some sort) leading
to a transaction rollback *after* the GH API call, so the cron's DB
actions are rollbacked (then performed again on the next run) *but*
the GH API interactions are not rollbacked, and are thus performed
twice.

Since the entire purpose of the feedback table is to send comments,
send both "staging failed" and "Linked pull requests" comments via
that.
2018-10-29 09:42:26 +01:00
Xavier Morel
3c633cb70d [IMP] runbot_merge: concurrency issue with GH closing PRs being merged
Once more unto the breach, with the issue of pushing stagings (with
"closes" annotations) to the target branch making GH close the PR &
send the hook, which makes runbot_merge consider the PR closed and the
staging cancelled.

This probably still doesn't fix the issue, but it reduces the
problematic window: before this, the process first updates the
branches, then marks the PRs, then comments & closes the PRs, and
finally commits the PR update.

This means as runbot_merge is sending a comment & a status update to
each PR in a staging, GH has some time to send the "closed" webhook
behind its back, making the controller immediately cancel the current
staging, especially if the v3 endpoint is a bit slow.

By moving the commenting & closing out of the critical path (to the
feedback queue), this window should be significantly shortened.
2018-10-24 16:14:31 +02:00
Xavier Morel
e885db8a50 [FIX] runbot_merge: make local tests run in v11 as well
Runbot is supposed to run on v11, yet because the runbot_merge test
suite was built on v12 due to an API change in
Registry.enter_test_mode the (local) test suite would only run on v12.

Add a conditional pick such that the test suite can run transparently
on both v11 and v12.
2018-10-24 15:11:51 +02:00
Xavier Morel
6565a0f9a1 [FIX] runbot_merge: don't batch patch-X PRs across repos
Normally, two PRs from the same owner with the same branch
name (source) are batched together, and that's represented by batching
them by label e.g. two PRs labelled
`odoo-dev:12.0-snailmail-address-format-anp` means they probably
should be merged together somehow.

*However* this is an issue when editing via the web interface: if the
editor doesn't have write access to the repository, github
automatically createa a branch called `patch-<n>` under their
ownership, where `<n>` is a sequence (of sorts?) *within the user's
fork*.

This means it's possible (and indeed easy) to create <foo>:patch-1 on
different (non-forks) but related repositories without them actually
being co-dependent, at which point they get blocked on one
another, which can lead to them being blocked (period) due to runbot
not currently handling co-dependencies between PRs.
2018-10-23 17:39:31 +02:00
Xavier Morel
13f843a165 [IMP] runbot_merge: adds direct links to CI details to the dashboard
Currently, if a staging is ongoing or failed one has to hunt for the
staging branches on the runbot dashboard in order to find out what
happens.

This adds a dropdown to the staging box/block providing direct status
and access to all the CI information whether the CI is ongoing or done,
successful or not.
2018-10-23 17:39:30 +02:00
Xavier Morel
02dd03fca3 [IMP] runbot_merge: warning on re-reviewing a reviewed PR 2018-10-23 17:39:30 +02:00
Xavier Morel
2f4a6f82ff [IMP] runbot_merge: quickfix to mark PRs as merging
Apparently the split and move arounds caused the _tagstate to get
computed/updated earlier (/differently), and thus the tagging update
relying on it to... not work anymore.

At least restore adding a "merging" tag when a PR is staged.
2018-10-17 16:55:53 +02:00
Xavier Morel
be4c8bd491 [FIX] runbot_merge: forcefully rollback ref after a failed rebase()
rebase() can fail after merge(), during set_ref(), having already
updated the target.

Under the pre-rebase model, stage() assumed on a staging failure on a
given repo it only had to rollback stagings having succeeded. This
assumption fails in a post-rebase model as even a failed staging can
have modified the target, leading to the next staging (if multiple
batches are ready) containing the failed one.

Things can get really strange if the set_ref failure was (as it
probably is) some sort of transient failure, as the following staging
will likely succeed (under the assumption that most PRs/batches pass
staging) as PR1's content gets merged as part of PR2, and PR2 is
merged empty of content later on.
2018-10-17 14:18:49 +02:00
Xavier Morel
ea0d55a0f7 [IMP] runbot_merge: logging of set_ref
Actual flow was only partially logged, making post-mortem debugging harder than necessary
2018-10-17 12:45:54 +02:00
Xavier Morel
2521ac8439 [IMP] runbot_merge: dashboard template
Forgotten code-port of direct UI changes
2018-10-17 11:31:07 +02:00
Xavier Morel
fe8a4588d3 [FIX] runbot_merge: try to fix race condition again
Original issue (staging would get cancelled just as it was being
merged) was not really fixed but traded for a new one: serialization
errors which can lock up the mergebot for a long time, stopping
handling of all incoming signals (possibly/probably because all of
them try to write on the PR which is locked?).

Splitting the tagging cron out should already way improve things as
the status update cron should be way shorter (and thus hold its locks
for a smaller amount of time). This should also avoid the "close"
handler waiting on the extant transaction, and make the "pr update"
transaction be much shorter as each staging gets its own tnx.
2018-10-16 15:10:43 +02:00
Xavier Morel
eef9ede686 [IMP] runbot_merge: dashboard template, add awaiting PRs 2018-10-16 13:15:00 +02:00
Xavier Morel
313d405a26 [ADD] runbot_merge: various feedback messages
Send reponse comments when users mis-interact with robodoo e.g.
send comments they don't have the right to, or commands which don't make
sense in the PR's state, or tentative interactions with robodoo from
unmanaged PRs.
2018-10-16 12:40:45 +02:00
Xavier Morel
0b629a32bc [ADD] runbot_merge: warning that a ready PR is linked to a non-ready one
People get surprised/worried that their ready PR never gets picked up,
but it's because there is a non-ready (either unreviewed or failing CI)
pull request linked to it. This aims to at least warn them of the issue.
2018-10-15 16:19:29 +02:00
Xavier Morel
63b7484873 [IMP] runbot_merge: helper for creating branches in multirepo tests
Taking 3 statements to create a branch before working with it is a bit
much, a simple helper reduces that to a single function call with 4
params (from 3 function calls with 1/4/1 params).
2018-10-15 12:52:36 +02:00
Xavier Morel
41400f791b [FIX] runbot_merge: remove commit from message when closing a PR
As noted in the old comment, the provided commit was the head of the
staging branch. This confused many users as they'd click the link
expecting to see their commit and potentially got something completely
unrelated.

Since we already get backlinks from the commit through the "closes
<pr_number>" added either by the committer or by the bot itself, the
information is already available.
2018-10-12 16:56:25 +02:00
Xavier Morel
bab7e25a9a [IMP] runbot_merge: split the MegaCron into sub-methods 2018-10-12 16:24:35 +02:00
Xavier Morel
5ec2c12454 [IMP] runbot_merge: split tagging cron out of main progress cron
They're completely independent (or should be), and there's no reason
for the tagging cron to extend the "lifetime" of the main cron's
transaction (and thus extend the odds of racey behaviour).
2018-10-12 15:54:14 +02:00