Commit Graph

1339 Commits

Author SHA1 Message Date
Christophe Monniez
3918d266fa [IMP] runbot: add coverage xml export
When coverage is computed, a post command is used to generate the HTML
report. In order to use the coverage result locally the HTML report is
not enough.

With this commit, an XML report is also generated. It's a single xml
file, downloadable from the build result web page.

The _post_install_command method is renamed into its plural form because
it was useless to return only one command.
2020-03-13 13:40:46 +01:00
Christophe Monniez
820adae5ac [IMP] runbot: auto disable host when git fetch fail
From times to times, a git repo gets corrupted, making builds fail in
chain.

With this commit, the host on which the git fetch fails will be reserved
if not more than half the hosts are reserved.
2020-03-13 13:38:42 +01:00
Xavier-Do
6b52687ed1 [FIX] runbot: better log list
Before intensive python steps, every build exept create build should have logs.
This is not true since now many python steps are creating build, leading
to 404 links to inexisting logs file.

Checking That a step is a docker build looks like a good heuristic since since
most of the time the log file will be the one created by docker. In all case we
can access other logs files in browsing /logs.

A future improvement would be to listdir logs to create all links.
2020-03-13 11:46:50 +01:00
Xavier Morel
5bba39e153 [FIX] runbot: run with unaccent enabled 2020-03-11 13:02:48 +01:00
Christophe Monniez
1fca89e3fd [FIX] runbot: fix invalid type argument when calling _log method 2020-03-10 17:07:47 +01:00
Christophe Monniez
3e40c38e89 [FIX] forwardport: fix forward-port Odoo wiki link 2020-03-10 08:00:07 +01:00
Xavier Morel
0b73e5c640 [IMP] fwbot: warning on r+ for failed PR
approving a PR which failed CI should trigger a feedback message since
6cb58a322d (#158), the code has not been
removed and the tests still pass.

However fwbot r+ would go through its own process for r+ which would
explain why that feedback is sometimes gone / lost (cf #327 and #336).

* make fwbot r+ delegate to mergebot r+
* add dedicated logging for this operation to better analyze
  post-mortem
* automatically ping the reviewer to specifically tell them they're idiots
* move the feedback item out of the state change bit, send it even if
  it's a useless r+ (because it's already r+'d)
* add a test for forward-ports

Closes #327, closes #336
2020-03-10 07:58:09 +01:00
Christophe Monniez
3b00d2576c [IMP] runbot: allow pseudo markdown in log messages
With this commit, a kind of markdown is allowed in log messages and
build.description.
Following patterns are supported:
  **strong**
  ~~striketrough~~
  __underline__
`code`
 [link](target)
 @icon-font-awesome-class
2020-03-09 13:10:49 +01:00
Christophe Monniez
3a428d4877 [IMP] runbot: copy steps when duplicating a config
When a build.config is copied, the config steps are not copied.

The steps have to be explicitly ordered otherwise it leads to a
traceback when trying to copy a 'run' step which  have to be the last
one.
2020-03-09 11:37:54 +01:00
Christophe Monniez
31e10a059e [IMP] runbot: bump Google Chrome version to 80
The fix odoo/odoo#45347 was merged two weeks ago, it's now time to
deploy on runbot.
2020-03-09 11:34:56 +01:00
Xavier Morel
aa2248e379 [IMP] forwardport: close command
This is useful as the author of the original PR doesn't necessarily
have (write) access to the repository where the forward-port PR was
created. As a result, while they can r+ the PR they're unable to close
it (via github's interface).

Since the forwardport bot created the PR, it can also close it, which
seems like a useful feature.

Closes #341
2020-03-04 11:56:01 +01:00
Xavier Morel
a3599f34cc [IMP] runbot_merge: don't inject Related if related PR already referenced
Closes #325
2020-03-04 11:56:01 +01:00
Xavier Morel
dde59b9fe6 [IMP] runbot_merge, forwardport: better signed-off-by handling
Remove original-signed-off-by, doesn't actually seem useful given the
semantics of signed-off-by according to the kernel doc'. Plus it
didn't actually work as the intent was to keep the signoff of the
original PR in the forward-port, but that signoff is not part of the
commit we're cherrypicking (it gets added on the fly when the commit
is merged).

Therefore explicitly get the ack-chain into the PR: when merging an FP
PR, try to integrate the signoff of the original PR, that of the final
FP pr, and while at it that of the last explicit update in the commit
chain (e.g. in case there's been a conflict or something).

Fixes #284
2020-03-04 11:56:01 +01:00
Xavier-Do
2379318b0c [IMP] runbot: add subcommand support 2020-02-28 15:54:44 +01:00
Xavier Morel
745f385dd3 [FIX] runbot_merge: error in test 2020-02-26 16:21:24 +01:00
Xavier Morel
6b5731f175 [FIX] forwardport: PR update through a closed PR
Fixes #328
2020-02-26 16:21:24 +01:00
Xavier Morel
9dbd0ac623 [IMP] runbot_merge: note on behalf of which commit a staging was created
Closes #329
2020-02-26 16:21:24 +01:00
Xavier-Do
7e49b4cd2b [IMP] runbot: add a public route for monitoring 2020-02-24 14:45:23 +01:00
Xavier-Do
6b88cb7688 [IMP] runbot: custom db_names for install and run jobs.
When creating a subbuild from a custom python job/cron, some data can
be given through config_data or extra_params to change another step behaviour.

An example is to give a specific -i to an install job in order to
install different modules from the parent build. Anyway since we
are using the same install step in this case, all databases will
have the same name, and the name may not be correct in regard to
the database content.
This commit allows to give a config_param on build giving the default
db_name to use in an install step.
2020-02-24 13:52:57 +01:00
Xavier-Do
ef7029668a [FIX] runbot: take parent into account when searching last branch build
If a branch triggers an hidden build on a another one (sticky ususally),
the last build of the branch will be considered to be this one and
trigger a new build. The same problem can occur when cloning a subbuild
to slighlty change a parameter instead of making an exact rebuild
since the build type may still be normal in this case.

This commit will simply ignore subbuild in this case.
2020-02-20 15:40:34 +01:00
Christophe Monniez
d9ff43fa6b [REM] runbot: remove reverse dependency build feature
When a new commit is found, a rebuild is forced on sticky branches
builds in repositories that depends on the new commit repository.

If one of the repo is protected by groups, the main page gives access
errors for public users and finally leads to a CPU overload.

With this commit, the feature is removed as it will be re-implemented in
custom config steps.
2020-02-20 10:58:20 +01:00
Christophe Monniez
931e2bef8f [FIX] runbot: avoid dependency build from an hidden build 2020-02-19 17:28:38 +01:00
Christophe Monniez
26dce90dd8 [FIX] runbot: avoid ask kill on already dying builds 2020-02-19 17:28:38 +01:00
Christophe Monniez
3d428428eb [IMP] runbot: replace fixing_commit by test_tags errors tree 2020-02-17 16:38:22 +01:00
Christophe Monniez
baea2e73be [IMP] runbot: kill build only when needed
On non sticky branches, when a new build is found while another is
already testing, the older build is killed. This happens during when the
main runbot instance is discovering new commits and create new builds.
As a result, concurrent updates may occur while the builders access the
concerned build.

With this commit, this garbage collecting procedure occurs during the
scheduler loop that runs on runbot builder hosts.

Also, the logic changed in a way that the kill is requested only if the
host needs room to handle pending builds.
2020-02-17 16:38:22 +01:00
Christophe Monniez
149ae4a074 [IMP] runbot: improve local cleanup
When a build age reaches the gc_days parameter, its database is dropped
and its directory is removed.

With this commit, two fields are added in order to keep some builds
longer that the defined gc_days.

The gc_delay field on the build allows to add a delay (in number of
days) that is added to its gc_days to compute the gc_date.

The gc_date field is the date when the cleaning will occur.

Also, a test is added and the RunbotCase test class is improved to allow
the stop of a patcher.
2020-02-17 16:38:22 +01:00
Christophe Monniez
5d4979a5f6 [IMP] runbot: set google-chrome version precisely
With this commit, it will be possible to play with different chrome
versions without impacting all runbot instances.

Chrome issues summary:

* 71: innerText linefeed issue
* 74: random chrome crash
* 80: V8 pointer compression exceed Odoo memory limits
2020-02-17 16:37:02 +01:00
Xavier Morel
48ba61d872 [IMP] runbot_merge: display & filtering of partners list
* only provide fields which make sense for the mergebot
* provide formatting & searchability for review rights records so
  they're visible from the list directly
2020-02-12 15:35:18 +01:00
Paul Morelle
1f9713cca0 [FIX] forwardport: update the wiki link
On github wiki, the link changed from
https://github.com/odoo/odoo/wiki/Mergebot
to
https://github.com/odoo/odoo/wiki/Mergebot-and-Forwardbot

Reflect these changes in the comments posted by the bot.
2020-02-12 14:06:36 +01:00
Xavier-Do
b4a25c54e4 [IMP] runbot: better same branch build filter 2020-02-12 11:41:08 +01:00
Christophe Monniez
3e7d98a6b8 [FIX] runbot: use a custom template to create db
Since cb05f2b9d8, when creating a database, the template1 is used, this
allows to customize the template and install some needed Postgress
extensions.

Unfortunately, it's also a source of build failures. For example, if the
pg_activity util is used on a runbot host, database creation may fail
with a message like this:

source database "template1" is being accessed by other users

It's because pg_activity needs a database and uses template1.

With this commit, template1 is still used by default but can be changed
with a system parameter. That way, a custom template can be created on
runbot hosts and used when creating DB in builds.
2020-02-12 10:38:54 +01:00
Xavier-Do
c816ba2161 [IMP] runbot: add latest builds of same branch 2020-02-11 17:15:20 +01:00
Christophe Monniez
464893d248 [IMP] runbot: handle PR re-targeting
When a PR branch target is changed on Github, the change is not applied
in the runbot DB.

With this commit, the Github hook payload is taken into account to
detect such a change and the branch infos are recomputed accordingly.

Also, a button is now available on the branch form in order to manually
recompute those changes.
2020-02-11 17:05:33 +01:00
Xavier Morel
7bd136b9d2 [FIX] runbot_merge: leftover debug prints 2020-02-11 14:37:03 +01:00
Xavier Morel
c235e9f6cc [ADD] runbot_merge: substitution filter on PR labels 2020-02-11 14:20:32 +01:00
Xavier Morel
314622e326 [IMP] runbot_merge: handle on repo_ids field
So they can be resequenced by drag&drop
2020-02-11 10:05:00 +01:00
Xavier Morel
3b417b16a1 [FIX] runbot_merge: when looking for stageable PRs group by target
This is more of a sanity check as it normally should not be a factor:
labels generally contain the target name, and staging checks are
performed per-target so we're not mixing multiple targets anyway.

But let's say a third-party creates a fix-foo branch for A and a
fix-foo branch for B, we want to ensure they're not considered batched
together.
2020-02-11 09:49:39 +01:00
Xavier Morel
b96bc9a58c [FIX] runbot_merge: make github_login case insensitive
Rather than try to fix up various bits where we search & all and
wonder what index we should be using, make the column a CIText.

For mergebot the main use case would be properly handling
delegate=XXX: currently if XXX is not a case-sensitive match we're
going to create a new partner with the new github login and
give *them* delegation, and the intended target of the delegation
isn't going to work correctly.

Also try to install the citext extension if it's not in the database,
and run the database-creation process with `check=True` so if that
fails we properly bubble up the error and don't try to run tests on a
corrupted / broken DB.

Fixes #318
2020-02-11 09:17:52 +01:00
Xavier Morel
742e3219a6 [IMP] runbot_merge: make review rights repo-dependent
As the odds of having more projects or more repos with different
requirements in the same project, the need to have different sets of
reviewers for different repositories increases.

As a result, rather than be trivial boolean flags the review info
should probably depend on the user / partner and the repo. Turns out
the permission checks had already been extracted into their own
function so most of the mess comes from testing utilities which went
and configured their review rights as needed.

Incidentally it might be that the test suite could just use something
like a sequence of commoditized accounts which get configured as
needed and not even looked at unless they're used.
2020-02-11 08:07:57 +01:00
Xavier Morel
05444aaf3f [IMP] runbot_merge: downgrade PRs to priority 1 on r-
Before this change, `r-` on a pr[p=0] does essentially nothing. At
most it will unstage if the PR had been (somewhat unnecessarily) r+'d
in the past but then the PR will get re-staged immediately.

To avoid this odd behaviour, if r- is sent to a p=0 PR not only is
the PR unreviewed (if it was reviewed) it always gets unstaged, and
its priority gets reset to 1 (high priority but doesn't bypass CI and
review). Also send a comment on that subject so followers of the pr
are notified.

Fixes #313
2020-02-11 08:07:57 +01:00
Xavier Morel
d9661064d6 [FIX runbot_merge: prevent reopening a merged PR
Fixes #305
2020-02-11 08:07:57 +01:00
Xavier Morel
18736d282f [FIX] runbot_merge: incorrect deref'
in controller, pr is the dict of the PR coming from github, pr_obj is
the odoo-level PR object (if any)
2020-02-11 08:07:57 +01:00
Xavier Morel
289c3deee8 [ADD] runbot_merge: tests to check behaviour of github in case of updating closed PRs 2020-02-11 08:07:57 +01:00
Xavier Morel
53e46001ba [IMP] runbot_merge, forwardport: notify when main crons are off
During freezes it can be useful to notify viewers that nothing is
going to forward port or merge for a while, and that this is
intentional (not something that's broken).

Fixes #307
2020-02-11 08:07:57 +01:00
Xavier Morel
9d661480fc [IMP] runbot_merge: split staging cron in two
The staging cron was already essentially split between "check if one
of the stagings is successful (and merge it)" and "check if we should
create a staging" as these were two separate loops in the cron.

But it might be useful to disable these two operations separately
e.g. we might want to stop the creation of new staging but let the
existing stagings complete.

The actual splitting is easy but it turns out a bunch of tests were
"optimised" to only run the merge cron. Most of them didn't blow up
but it seems more prudent to fix them all.

fixes odoo/runbot#310
2020-02-11 08:07:57 +01:00
Christophe Monniez
46c8c6d635 [FIX] runbot: more flexible duplicate detection 2020-02-10 14:13:38 +01:00
Xavier Morel
c702fecda1 [FIX] runbot_merge: updating PRs in repos without required statuses
The PR creation had been fixed to always validate even without a
commit found (in case there was no need for a commit), but the update
of a PR in such a situation was not tested, and thus naturally did not
work because why would it work if it wasn't tested?

Also remove the conditional skip on updating a PR to a new head.
2020-02-07 16:11:12 +01:00
Xavier Morel
8f09eacfb5 [FIX] runbot_merge: handling of empty required_statuses for computed pr status 2020-02-07 15:54:14 +01:00
Xavier Morel
0831c899e8 [FIX] runbot_merge: the fix
The test was checking things would work properly with
required_statuses being an empty string, because I'd also forgotten an
empty field becomes stored as `False` in the database, so trying
things out live neither the PRs nor the staging would work as their
assumption that they could straight split the required_statuses would
always fail.

Update the test to better match expectations, and hopefully this is
the end of that saga.
2020-02-07 09:55:22 +01:00
Xavier Morel
4bdf7e5eda [FIX] runbot_merge: stagings involving repos w/o required statuses
PRs transitioning to 'ready' had been checked and tested but turns out
I had completely forgotten to test that stagings would validate
properly therefore of course they didn't.

The issue here was I'd forgotten `''.split(',')` returns `['']` rather
than `[]`, so on an empty required_statuses the staging validator
would keep looking for a status matching the context `''` and would
never find it, keeping the staging pending until timeout. So most
likely the problem could have been resolved by just adding a condition
to

    [r.strip() for r in repomap[c.sha].required_statuses.split(',')]

but I'd already done all the rest of the reorganisation by that point,
test pass and I think it's a somewhat better logic. Therefore I'll go
with that for now.

* properly handle empty required_statuses during staging validation
* remove the final postcondition, if we're missing commits which don't
  require any statuse we should not care
* expand test to include up to merging PRs
* automatically create dummy commits when creating stagings, that way
  the relevant commits are in the database (can't hurt)

PS: an other alternative would have been to filter out or skip ahead
on commits which don't require any statuses aka cmap &
required_statuse / cmap would not even have that entry
2020-02-07 08:29:55 +01:00