Commit Graph

1032 Commits

Author SHA1 Message Date
Christophe Monniez
5ff9cc2382 [FIX] runbot: clean running dockers with done builds
In different situations, a docker container may stay alive even if the
build global_state is done. This can lead to a build failure when a
build wants to go in running state and tries to expose the same ports as
the left over build.
2019-10-21 10:52:45 +02:00
Xavier-Do
5b12f37c74 [REV] runbot: Avoid infinite loop fetching"
This reverts commit 1207daded1.

A too quick review, setting a default value is a good idea but since field is a float now,
default value should be time.time
2019-10-18 20:20:10 +02:00
Christophe Monniez
3308829e80 [IMP] runbot: uses fnmatch patterns to select modules
Actually some Odoo modules are black_listed from a set hardcoded in the
runbot code. In some cases, one needs to blacklist custom modules,
preferably in a config_step.

With this commit, the repo.modules, branch.modules,
config_step.install_modules fields are concatained in a comma separated
list of fnmatch patterns. The patterns can be prefixed with a dash to
exclude the matching module(s).

Co-authored by @Xavier-Do
2019-10-18 16:30:06 +02:00
Xavier Morel
7c46a2006f [FIX] forwardport: the fix
Of course I forgot the most relevant bit
2019-10-18 12:01:47 +02:00
Xavier Morel
5f8041552b [FIX] forwardport: apparently git/refs/heads can fuzzy-match
If the ref we asked for does not exist, github apparently decides to
fall-back to prefix-matching. So if we're trying to delete
already-deleted branch A and someone called their branch A-x we're
going to get it as a result.

Thankfully they were apparently smart enough to return a list even if
there's only a single fuzzy match. So if we get a list (instead of a
dict) as response to git/refs/heads assume the branch was already
deleted as if we got a 404.
2019-10-18 11:22:13 +02:00
Moises Lopez - https://www.vauxoo.com/
1207daded1 [FIX] runbot: Avoid infinite loop fetching
Using `repo.hook_time = None` and `repo.mode = hook` the cron run fetch command in an infinite-loop.
2019-10-18 11:18:46 +02:00
Christophe Monniez
27b1c2b5f4 [IMP] runbot: add the active_step_id on the ir_logging log
In order to have a more efficient ir_logging filtering, this commit adds
the currrent build step id on the ir_logging line.
2019-10-18 11:03:22 +02:00
Christophe Monniez
db52bff323 [IMP] runbot: number of log lines per build
When a build is running, a cron, an evil query or something else can
start to fill and bloat the runbot ir_logging table.

With this commit, a log_counter field is added on the build, starting at
100. The SQL trigger decrement this counter after a line is inserted.

When the counter drops to 0, a the last log line contains a message
stating that the limit has been reached. Further log lines are dropped
for this build step.

The counter is reset to a default of 100 before each step.
This value is configurable through an optional ir.config_parameter
runbot_maxlogs.

The runbot itself is still able to add logs lines through the build _log
method.

Thanks @Xavier-Do for the smart idea.
2019-10-18 11:03:22 +02:00
Xavier Morel
13d76fdfb9 [FIX] forwardport: the fix
it's a per-call parameter not a per-instance one
2019-10-18 08:11:27 +02:00
Xavier Morel
c1cef0c18b [FIX] forwardport: gh api raises by default, avoid that here 2019-10-18 08:02:04 +02:00
Xavier Morel
ea410ab6d1 [ADD] forwardport: automatic branch deleter
If a PR is *merged*, enqueue it for deletion (with a 2 weeks delay).

Mainly to avoid FW branches staying around long after they've been
merged (possibly eventually closed?), will also clean up regular
merged branches, including historical merges forgotten by their
author.

Fixes #230
2019-10-17 11:55:20 +02:00
Xavier-Do
ac578d430d [IMP] runbot: propagate end_time to parent_build
When a build only create sub-builds, the build_time is verry small (a few seconds),
and this information is not relevant. This commit propagates end_time to parent_build
if parent_build is done or running.
2019-10-16 14:25:43 +02:00
Xavier Morel
401787b7ae [FIX] forwardport: co-dependent FPs where one PR is updated
In the case where we have a co-dependent forward port (co-dependent
PRs got forward-ported, which they should be together) where *one* of
the PRs got explicitly updated, the batch would "fall into a hole"
being handled as neither "this is part of a forward-port sequence" nor
"this is a new merge to forward-port" (the latter being the proper
one).

Modify & remove guards which checked that either no or all PRs in a
batch have parents: should be either all or not all.

Fixes #231
2019-10-15 08:54:25 +02:00
Xavier Morel
8937f379ce [FIX] forwardport: fix the fix
Re-add token field name when creating the tagging object, as it's a
required field....
2019-10-14 12:01:12 +02:00
Xavier Morel
97999318be [FIX] forwardport: don't use token field for tags update
Turns out tagging PRs requires a pretty significant level of ACLs
which we may not want to give to the forwardbot?

Anyway use the mergebot ACLs (which already include tagging) for this.
2019-10-14 10:09:48 +02:00
Xavier Morel
8f3f773eef [IMP] *: testing helpers
* add a sorted method on fake models
* fix recordset equality to ignore ids order
* when creating commits on a ref, add a param to only *update* the ref
  (forcefully): when simulating a force-push we don't want to *create*
  a ref as that might silently be done in the wrong repository entirely
* fix pytest.skip call at the module level, not sure where it came
  from and why I missed it until now
2019-10-14 10:09:48 +02:00
Xavier Morel
cc029c2891 [IMP] runbot_merge: add some logging to controllers
The closing or reopening of PRs was not logged at all, which can be
inconvenient when trying to find out why PRs are closed (or not) in
the backend.

Also leverage PR display_name improvements from
3ce3dd9569 for more regular PR names in
logs.
2019-10-14 10:09:48 +02:00
Xavier Morel
2eb4ece50a [FIX] forwardport: title-ing of PRs
Due to the title formatting of FP PRs, we'd get incorrectly formatted
commit messages if the PR was *merged* (rather than squashed /
fast-forwarded) due to either "merge" or "rebase-merge" integration
mode: in that case the PR message would be used as message for the
merge commit and that'd be along the lines of "Forward Port of #xxx to
<somebranch> (failed)", followed by the old PR message (e.g. see this
commit message).

* re-extract and reuse original PR title, just prefix with "[FW]"
* finally add support for tagging, and use that to tag the PRs,
  especially for the failed / conflict marker which is quite important

Closes #229
2019-10-11 13:05:36 +02:00
Xavier Morel
2971d042a4 [FIX] forwardport: only send notifications to the PR we're processing 2019-10-11 09:37:03 +02:00
Xavier Morel
e5efd52674 [IMP] *: add (individual) timings to test cron runner
Useful to know if the crons are slow during testing.
2019-10-11 09:14:03 +02:00
Xavier Morel
bad016b830 [FIX] forwardport: queue reliability changes
Previous version would break if _process_item itself committed which
was bad
2019-10-11 09:13:55 +02:00
Xavier Morel
3ce3dd9569 [IMP] forwardbot: show FP PRs in reminder message
When posting a reminder that there are open / waiting forward ports on
a source PR, also post *which* PRs those are.

While at it, move the cron code in a proper python file (so we can use
stuff from odoo.tools), and fix display_name so we can straight use
display_name as a github ref' ({owner}/{repo}#{number}). This impacts
log-grepping but it seems like an improvement nonetheless.

Closes odoo/runbot#228
2019-10-11 09:13:55 +02:00
Xavier Morel
036ae3a8ee [IMP] forwardbot: reduce length of fw branch name
* shorten the postfix, forwardbot is now a bigram!
* shorten the uniquifier: go from 5 to 3 bytes, and use urlsafe base64
  that way we only have a 4-char uniquifier instead of 8
* while at it, fix deprecated calls to logging.warn (should be
  logging.warning)

Fixes #226
2019-10-10 11:37:27 +02:00
Xavier Morel
d453943252 [IMP] *: unify gh test API between runbot and fw-bot
The fw-bot testing API should improve the perfs of mergebot tests
somewhat (less waiting around for instance).

The code has been updated to the bare minimum (context-managing repos,
change to PRs and replacing rolenames by explicit token provisions)
but extra facilities were used to avoid changing *everything*
e.g. make_commit (singular), automatic generation of PR refs, ...

The tests should eventually be updated to remove these.

Also remove the local fake / mock. Being so much faster is a huge
draw, but I don't really want to spend more time updating it,
especially when fwbot doesn't get to take advantage. A local /
lightweight fake github (as an external service over http) might
eventually be a good idea though, and more applicable (including to
third-parties).
2019-10-10 10:11:48 +02:00
Xavier Morel
557878afe9 [IMP] forwardport: processing queue reliability
The queue would get items to process one at a time, process, commit,
and go to the next. However this is an issue if one of the item fails
systematically for some reason (aka it's not just a transient
failure): the cron fails, then restarts at the exact same point, and
fails again with the same issue, leading to following items never
getting processed.

Fix by getting all the queue contents at once, processing them one by
one and "skipping" any item which fails (leaving it in place so it can
get re-processed later).

That way, even if an item causes issues, the rest of the queue gets
processed normally. The interruption was an issue following
odoo/enterprise#5670 not getting properly updated in the
backend (backend didn't get notified of the last two updates /
force-push to the PR, so it was trying to forward-port a commit which
didn't exist - and failing).
2019-10-10 08:41:33 +02:00
Xavier Morel
fafa7ef437 [IMP] *: attempt to avoid some of the FP spam
Attempt to avoid some of the comment spam by dedup-ing input (only
signaling when the status actually changes and ignoring identity
transformations) and in case of failing CI keeping the last failed
status and not signaling on the next update if it's the same failure.

Closes #225
2019-10-07 16:38:14 +02:00
Xavier Morel
85035ad2c0 [IMP] runbot_merge: separate splits from other awaiting PRs
Should make the (eventual) wait and the extent of the splitting clearer.

Fixes #217
2019-10-07 09:26:27 +02:00
Xavier Morel
60c8f0f498 [FIX] runbot_merge: behaviour when no CI are required
If the required_statuses are empty, PRs should always be
validated (and just require a review) rather than never be merge-able.

Fixes #216
2019-10-04 10:00:14 +02:00
Xavier Morel
0152271fb8 [IMP] runbot_merge: layout responsivity
* only show 2 stagings on cellphones as 4 is way too much, moving to a
  vertical layout would probably be a bad idea as stagings can already
  be very tall and then we have multiple branches stacked on one
  another, unless we also make branches foldable

  the more complete list of stagings (per branch) is available on the
  branch's page anyway so providing a not-completely-broken home looks
  more useful, and at a fundamental level the current / last staging
  is really the one we care about
* remove the size bounds on stagings to avoid smushing all the cells
  together and overlapping text, sadly can't overflow scroll the
  stagings element because you can't have an overflow-x: scroll and an
  overflow-y: visible (that becomes auto)
2019-10-03 11:19:00 +02:00
Xavier Morel
e9e08fec3c [IMP] forwardport: fix creation of FP PR in some testing cases
When running tests on some machines, it's apparently possible for the
PR-creation webhook to come back before the PR creation request has,
leading to the creation of the PR from the API call duplicating that of
the webhook and blowing up.

To fix, immediately commit the transaction then check if we already have
the PR we just created in the system, and only create it explicitly if
not.
2019-10-03 10:02:37 +02:00
Xavier Morel
c5d68c20f4 [IMP] forwardport: add feedback when being wrong
User should probably be warned when they try to set the limit ("up
to") in a context where it's going to be ignored:

* on a forward port PR (should be set on the source)
* on a merged source (should be set before the PR is merged)

Closes #213
2019-10-02 17:49:54 +02:00
Xavier Morel
0ae01c6ddb [FIX] runbot_merge: dropdown in branch staging history
Had apparently been fixed "live" but not committed in code, so every
update of the module would re-break it.

Fixes #221
2019-10-02 17:49:54 +02:00
Xavier Morel
01ff6d13db [IMP] forwardport: move limits tests into their own file
test_simple is starting to get pretty big and we'll need some more
tests for #213
2019-10-02 17:49:54 +02:00
Xavier Morel
e49b112447 [FIX] runbot_merge: only update pending staging state
The staging validation routine would ignore stagings which were
cancelled or ff_failed, but it should also have ignored failed and
successful aka all terminal state.

Simplify the condition for that: just ignore a staging's validation if
the staging is not pending.

Closes #211
2019-10-02 17:49:54 +02:00
Xavier Morel
cd29c648e8 [IMP] forwardport: don't ping for CI failure on closed PRs
Fixes #210
2019-10-02 17:49:54 +02:00
Xavier Morel
9c3d12c964 [FIX] forwardbot: selection of ancestor in FP tail ping
In selecting the parent commits to list on the last PR, we would miss
the *first* forward-port of the sequence. Not sure why we added a
detrimental check on source_id there.

Also add a missing space between "chain" and "containing" in the case
where there's at least one forward-port PR other than the final one.

Fixes #212
2019-10-02 17:49:54 +02:00
Xavier Morel
37cf6accb7 [FIX] forwardport: FP PR getting CI'd after initial FP check
Test is probably more complex than necessary (thinking about it, the
failed staging is probably unnecessary) but that triggers the issue
and matches the original scenario.

The problem was really with new CI events being received on the last
forward-port PR of a sequence: previous PRs would have a child PR so
the check would abort, but for the last PR it would go through, fail
to find an active batch, then blow up as it tries to create a
forwardport.batch without an actual batch.

Change this to use the existence of an inactive batch not linked to a
staging as a flag that the PR has been processed and forward-ported.

Closes #218
2019-10-01 20:51:31 +02:00
Xavier Morel
a5794a1a24 [FIX] forwardport: better version of previous fix
Turns out we don't want to close the cursor on success, we just want to
commit, but that's not what the default context manager does.

So don't use said context manager.
2019-10-01 09:57:35 +02:00
Xavier Morel
eb9eeb670a [IMP] forwardport: avoid locking cron when a _validate blows up
If a _validate call blows up, the entire Commit._notify cron gets
stuck, which is an issue because not only does it stop creating
forward ports, it also stops "progressing" stagings.
2019-10-01 07:56:24 +02:00
Xavier Morel
7659293a2b [IMP] runbot_merge: update staging timeout on 'pending' CI
If the CI is greatly backed up (either insufficient capacity or jobs
spike) a timeout which is normally perfectly fine might be
insufficient e.g. given a 2h timeout, if a job normally takes 80mn but
the staging's job starts 40mn after the staging was actually created
we're sunk. And cancelling the staging once the job has finally gotten
started is not going to improve load on the CI, it just wastes a CI
slot.

Therefore assume a `pending` event denotes the actual start of the job
on the CI, and reset the timeout to start from that moment so
ci_timeout is the timeout of the CI job itself, not of the staging
having been created.

Closes #202
2019-09-23 15:42:18 +02:00
Xavier Morel
78ad4b4e4b [IMP] runbot_merge, forwardport: consolidate conftests
Converge the pytest setups of runbot_merge and forwardport a bit
more (the goal is obviously to eventually share the infrastructure so
they run the same way).
2019-09-23 13:54:42 +02:00
Xavier Morel
63bef8b7ab [IMP] forwardport: notify when FP PR gets de-parented
If a PR is explicitly updated, it gets converted to a normal
PR[0]. Before this, users had no indication that this had happened and
might be wondering what they're supposed to do (or try to r+ via the
forwardbot, which doesn't work on a root PR).

[0] to an extent: the PR still has a source and might have children,
    in which case the followups will be created from the source &
    existing followups should be updated to match

Closes #206
2019-09-23 12:54:22 +02:00
Xavier Morel
8de6273498 [IMP] forwardport: unweird singleton conflicting commits
When creating the conflicting commit of a single commit PR, reuse the
original commit's meta-information so the developer / fixer can more
easily update it in-place.

Closes #204
2019-09-21 15:23:42 +02:00
Xavier Morel
446b11a28f [IMP] forwardport: link FP PRs to both root and source
In the case where an FP sequence is interrupted (e.g. there was a
conflict during one of the intermediate steps), followups get linked
to the original source but don't get linked to the "interruption" PR
which is a bit confusing.

Link FP PRs to both source and root if they're different.
2019-09-21 15:23:42 +02:00
Xavier Morel
6d4923928a [IMP] forwardport: improve disable fp UI
* always allow specifying the PR's own branch as a forward-port limit
  / target (even if deactivated or disabled)
* add an "ignore" alias to "up to <pr target>" for clarity
* add dedicated feedback when deactivating forward-port of a PR

Fixes #191
2019-09-21 15:23:42 +02:00
Xavier Morel
66d65ba550 [IMP] runbot_merge, forwardport: variable-user feedback
Having all the feedback be sent by the mergebot user (github_token) is
confusing. Add a way to specify which field of project should be used to
source the token used when sending feedback.

Fixes #190
2019-09-21 15:23:42 +02:00
Christophe Monniez
4b2a93af9e [FIX] runbot: fix build_error active field changes
When a build_error active field is changed, the onchange leads to a
traceback. Anyway, the onchange was not a good idea as it only reflects
UI changes.

With this commit, the write method is overwritten to change the
child_ids active fields too. Also, the active_test context is used to
correctly compute the childs_ids and children_build_ids.

A test is also added for all that.
2019-09-20 14:58:16 +02:00
Christophe Monniez
a2a8fe31f1 [IMP] runbot: add firebase-admin to the Docker file
The new feature in odoo/enterprise#4879 needs the firebase-admin
package. As it cannot be added to the requirements.txt, the package is
added in the Dockerfile to be able to test it on the runbot instances.
2019-09-20 14:46:21 +02:00
Christophe Monniez
47ad04dc37 [FIX] runbot: remove duplicate import in tests 2019-09-18 15:32:19 +02:00
Christophe Monniez
56999ecfb4 [IMP] runbot: various improvements
- Add a keep running flag on the build to allow a build to stay in
  running state until the flag is switched off ( or the build killed)
- Do not update configs and config_steps data
- Add a first/last_seen_build and first/last_seen_date on build.error
- Children error builds now include the parent builds too
- Use a notebook on build.error form view to display builds and linked
  errors
- Update result when a build triggers a change from 'warn' to 'ko' too
- Add the sticky flag on the error logs stored sql view
2019-09-18 15:27:19 +02:00