Commit Graph

1136 Commits

Author SHA1 Message Date
Xavier-Do
84d9425bdc [FIX] runbot: various view fixes
- fix missing batch references on builds
- fix group on action buttons
2021-04-09 14:58:17 +02:00
Christophe Monniez
57bd00672d [IMP] runbot: add a chart page for build stats
Since 360e31ade4, it's possible to add statistics values to build
results but there was no practical way to analyze them.

With this commit, there is a new button on the bundle page that leads to
a chart page that displays those values.

The default reference build is last known good build of the bundle.
Values are filtered by key and only the most significant values are
displayed. The user can then refine the chart by changing the reference
build or the key and a few other options.

Co-author: Xavier-Do <xdo@odoo.com>
2021-04-09 14:10:37 +02:00
David James
19c312d92c [IMP] runbot: add an option to exclude paths from coverage report 2021-04-07 15:47:11 +02:00
Christophe Monniez
785a7796fb [IMP] runbot: allow conditional pip requirements
Before this commit, the `requirements.txt` from a specified odoo branch
(master by default) was always installed in the Dockerfile's.

We now allow to disable this feature to test Odoo with vanilla
distributions.
2021-04-07 11:35:33 +02:00
Christophe Monniez
6bd74c07c4 [IMP] runbot: allow to install Chrome from google
When choosing to install Chrome in a Dockerfile, the chrome version is
downloaded from Odoo nightly server. This make it difficult to test
with different versions of Chrome.

With this commit, we allow to install from Google in Docker files.

By default, the install remains from Odoo Nightly server but if the key
`custom_values['chrome_source']` is set to 'google' in a Dockerfile,
the specified version will be downloaded from Google servers when the
Docker image is built.
2021-04-07 11:35:33 +02:00
Christophe Monniez
50f803ec31 [FIX] runbot: allow larger upload on runbot odoo instances 2021-04-07 11:01:35 +02:00
Andrius Laukavičius
5337ecd11f [FIX] runbot: _compute_host_id
If you try to manually create bundle, Odoo will crash, because it will
try to use `name` value that is not set yet. For that we start computing
host_id once `name` is entered.
2021-04-07 11:00:55 +02:00
Xavier-Do
0ed4728518 [IMP] runbot: use last step in wakeup when possible 2021-04-07 10:31:05 +02:00
Christophe Monniez
0f4610c8bc [FIX] runbot: remove forgotten print 2021-03-16 10:45:47 +01:00
Xavier-Do
46269ada70 [FIX] runbot: fix logged message 2021-03-15 13:12:48 +01:00
Xavier Morel
318803d7bb [IMP] runbot_merge: tag PRs with a pseudo-branch on merge to master
If a PR got merged to master (or whatever the current development
branch is), there's no easy way to know what maintenance branch it
ended up landing in, except by asking git which branches contain the
commit (which can be rather slow).

Add a special case on merge which labels the PR with a pseudo-branch
patterned after the second-to-last branch of the project:

* if the branch ends with a number, increment the number by one
  e.g. 2.0 -> 2.1, 5 -> 5.1
* otherwise, just prefix with `post-` e.g. "maint" ->
  "post-maint" (that one doesn't sound very helpful, but I guess it's
  nice for the weirdoes who call their branches "natty narwhal" and
  shit)

Fixes #450
2021-03-02 14:28:32 +01:00
Xavier Morel
60af69b69f [FIX] runbot_merge: dashboard title should link to github
Currently links to self, which is not useful (the viewer is already
there).

Fixes #452
2021-03-02 14:28:32 +01:00
Xavier Morel
4e4e4303f6 [IMP] runbot_merge: add name_search override to PRs
Should allow filtering PRs by source or parent.

Fixes #458
2021-03-02 14:28:32 +01:00
Xavier Morel
0b1e33da7c [IMP] forwardport: mitigate cat-file not finding commit on updates
Fix #457 hopefully: I didn't manage to repro / create a test for.

It looks like in some cases during the update process the PR ref lags
behind the branch itself. This means `forwardport.updates` creates a
new commit, pushes it, then on the next iteration updates the local
cache, tries to find the commit we just pushed... and that fails.

I can only assume this is because when there's enough load on the
github side the update to the `info/refs` pseudo-file can fall
behind (it's now 4MB and holding nearly 65k refs).

So cheat: take the commit we just pushed to the dev remote
and... immediately push it to the local cache under a dummy branch,
which we delete. Since we only gc "1 day ago" this should not vacuum.
2021-03-02 14:28:32 +01:00
Xavier Morel
8a924fb4b7 [FIX] forwardport: duplicates forwardport on edition
On edition of an intermediate PR in a chain, merging the PR would lead
to *it* being forward-ported, duplicating the PRs already created
from *its* source.

Add a check for PRs in the target branch with the same source,
suppresses the forward-porting of the newly merged PR.

Fixes #451 (hopefully)
2021-03-02 14:28:32 +01:00
Xavier Morel
952dafa45c [IMP] forwardport: feedback on conflict
append `git status` data to stderr, should be somewhat more
informative especially when a conflict is a DU (where the file has
been deleted on one side, so there is no conflict marker anywhere).

Fixes #461
2021-03-02 14:28:32 +01:00
Xavier Morel
a5f2d14707 [CHG] forwardport: try to create PRs in draft mode
Fall back to creating in not-draft mode if that doesn't work: draft
pull requests on private repositories requires the Team plan.

Closes #459
2021-03-02 14:28:32 +01:00
Xavier Morel
9c1585383a [IMP] forwardport: git commands logging
- When updating the local repo cache, always capture both stdout and
  stderr and log them out rather than having them in journalctl hard
  to relate to the main log.
- In the git layer, capture stderr by default and log it automatically
  on command failure.
2021-03-02 14:28:32 +01:00
Xavier Morel
2aeecb68b9 [FIX] forwardport: completely update PR data when forwarding updates
The process did properly update the state, but not the squash state.

It's somewhat unclear whether the state should be fully reset and
require reapproval though. Maybe only the validation should be reset?
The CI will eventually run and either succeed (re-validating) or
fail (devalidating, hopefully) but I'm not entirely sure this is
correct.
2021-03-02 14:28:32 +01:00
Xavier Morel
a541781ee0 [FIX] test proxy recordset union / concatenation
Apparently I wrote this when I was a dumbshit and did not think that
Python sets are implemented completely separately from dicts
and *remain unordered*.

As a result, the simplistic set union would not properly conserve
record order, which it should.
2021-03-02 14:28:32 +01:00
Xavier Morel
2b64f7feb3 [REM] duplicate method in test proxy
Probably a duplication I missed when I merged the proxies from
runbot_merge and forwardport.
2021-03-02 14:28:32 +01:00
Xavier Morel
5e05d3cd96 [REF] forwardport: move PR update tests to their own module 2021-03-02 14:28:32 +01:00
Xavier Morel
ab919c00db [FIX] conftest: ensure we always have a reason in get_ref
Apparently the "reason phrase" has been removed from HTTP/2 and github
has enabled HTTP/2, meaning responses may or may not contain a reason
phrase depending whether they're HTTP/1.1 or HTTP/2. Now I don't
understand how this triggers because requests isn't supposed to
support HTTP/2, and in a shell I do get a reason, but w/e.

Anyway the reason is used by a few tests to check more specifically
for the response they're getting, as that's used as the assertion
message by `get_ref`.

I guess I could replace the AssertionError by an HTTPError
but... w/e. Seems simpler to just fallback to the generic reason based
on the status code. It fixes both failing
tests (test_straightforward_flow and test_delete_normal) with very
little change.
2021-03-02 14:28:32 +01:00
Xavier-Do
c538bd17f8 [IMP] runbot: use build database_ids to generate connect links
aka: make it clean

This build database_ids field was generated a few months, waiting for the database to update with
this new sceme before using it.

It is still a little early to use it in cleanup methods, but this can be used to
generate the connect links dynamicaly.

To follow al specs introduced in previous commit, main fa-sign-in button should link to the -all.
It will almost always be the first one in database_ids in alphabetic order.
Then, in the dropdown, all other database are listed.

This will fix the previously broken design_theme connect link (no base nor all).

For this purpose nginx regex needs to be adapted to accept database name with underscore.
2021-03-01 15:49:23 +01:00
Xavier-Do
26ad74051f [IMP] runbot: apply al connect button spec 2021-02-26 12:13:39 +01:00
Xavier-Do
38b52411fa [FIX] runbot: use database name subdomain
Using database name as subdomain will set the web.base.url correctly when connection for the first time using this link
This will unfortunatelly break connect all and connect base links if dns and nginx are not setup.
This won't fix the web.base.url when connecting with the Connect (database selector) button either.
2021-02-22 16:03:51 +01:00
Christophe Monniez
f28485b6a8 [FIX] runbot: allow to retry to send a status
When a status cannot be sent to github, the status is created in
database but the `sent_date` field is not set.

Because of that, the status cannot be sent manually from the frontend.

With this commit, that kind of status can by tried again.
2021-02-02 13:32:03 +01:00
Christophe Monniez
4f4005896c [FIX] runbot: browse a valid record id in test_warning
Since odoo/odoo@824a651 the test_warning_from_runbot_abstract was
failing for a good reason, the `browse` parameter was a recordset
instead of an id.
2021-02-02 11:48:03 +01:00
Xavier Morel
21077c690a [FIX] runbot_merge: incorrect processing of rebased commit messages
5cf3617eef intended to create merge
messages with only the content of PR descriptions before the first
thematic break. However this processing was incorrectly applied
to all messages being processed (meaning rebased / squashed commit
messages as well).

Properly apply thematic break processing to only commit messages
created from PR descriptions.
2021-01-21 13:15:32 +01:00
Xavier Morel
08b8da08df [IMP] runbot_merge: precisely filter stagings before validating them
Before this, we would "roughly" select stagings by looking at stagings
whose heads matched a specific sha then validating them all. This
could perform extra validations on stagings once in a while but this
was assumed not to be much an issue, at least originally.

However two changes later on have contributed to this likely being the
cause of #429 (stagings never timing out):

* heads of the staging branches are uniquifier commits stored in the
  heads map, but the actual heads of the stagings are also stored
  there, some of which are no-ops (hence the uniquifiers) so assuming
  repos A and B, if a staging contains PRs touching A then the head of
  B actual will also be a head of B
* when a staging is validated, if it *contains* any pending result the
  timeout limit gets bumped back

The issue here is that if a success / failure status is lost (which
would be the most common reason for timeouts) *and* someone has forked
and is regularly rebuilding a branch-head used as-is by a staging,
each of those rebuilds will trigger a validation of the staging, which
will find that one of the statuses is still pending (because we missed
the success / failure), which will bump up the timeout limit,
continuing until the branch stops getting rebuilt.

This is probably one of the reasons why some stagings last for *way*
more than 2h, though it is far from explaining all of them: 90% of the
stagings lasting more than *3*h end up succeeding. Tho it's always
possible that this is because someone notices and resends a success
for the missing status it seems somewhat doubtful. Oh well.

Also fix the incorrect log call on `update_timeout_limit` triggering.
2021-01-20 09:22:11 +01:00
Xavier Morel
b0576d1d41 [IMP] forwardport: cherrypick logging
Enable GIT_TRACE to try and get more information pertaining to the
cherrypicking process during failure.

Related to #449
2021-01-18 07:44:17 +01:00
Xavier Morel
1b7040cd26 [IMP] forwardport: logging during cherrypick process
* fix small error which generated an extra commit in case of conflict
  probably (would create a `temp` commit, then put the conflict
  information on an empty commit on top of it). Avoids committing the
  cherrypick though a probable alternative would be to commit the
  message with `amend`.
* improve the cherrypick header: instead of one log line per commit,
  put all commits within the sole header log line (with a newline),
  should make things less noisy
* put *all* log records below the correct logger (two of them were on
  the toplevel logger)
* log a purported inflateInit error as warning, should be sent to
  Sentry instead of having to wait for people to wonder why the thing
  is completely broken
2021-01-15 14:19:17 +01:00
Xavier Morel
560c6e6b16 [IMP] mergebot: log repr of comments (might show oddities) 2021-01-14 07:42:10 +01:00
Xavier Morel
ba1a8ee089 [IMP] runbot_merge, forwardport: update some logging
Downgrade an error and a warning to info, and upgrade two warnings to
error.

Point is to improve logic of log levels & sentry visibility.

Fixes #258
2021-01-13 16:12:35 +01:00
Xavier Morel
95e35dde90 [IMP] runbot_merge: logging when bumping timeout
I'd forgotten that in order to better handle cases where the CI is
highly backed up (and / or slow for some reason), we actually update
the CI timeout to really take the last "pending" status as the "true
start" of the CI. This might explain why lots of stagings needed extra
time: as of right now, out of 28835 stagings

- 20086 had their timeout bumped by more than 15mn
- 6967 had their timeout bumped by more than 30mn
- 264 had their timeout bumped by more than 1h
- 30 had their timeout bumped by more than 2h

Add some logging every time the CI is bumped this way, so we have
better visibility into that event.

Closes #429
2021-01-13 16:12:35 +01:00
Xavier Morel
3cc87051dd [FIX] runbot_merge: avoid repeatedly warning about the same failures
The mergebot has a feature to ping users when an approved PR or
forward-port suffers from a CI failure, as those PRs might be somewhat
unattended (so the author needs to be warned explicitly).

Because the runbot can send the same failure information multiple
times, the mergebot also has a *deduplication* feature, however this
deduplication feature was too weak to handle the case where the PR has
2+ failures e.g. ci and linting as it only stores the last-seen
failure, and there would be two different failures here.

Worse, because the validation step looks at all required statuses, in
that case it would send a failure ping message for each failed
status *on each inbound status*: first it'd notify about the ci
failure and store that, then it'd see the linting failure, check
against the previous (ci), consider it a new failure, notify, and
store that. Rinse and repeat every time runbot sends a ci *or* lint
failure, leading to a lot of dumb and useless spam.

Fix by storing the entire current failure state (a map of context:
status) instead of just the last-seen status data.

Note: includes a backwards-compatibility shim where we just convert a
stored status into a full `{context: status}` map. This uses the
"current context" because we don't have the original, but if it was a
different context it's not going to match anyway (the target_url
should be different) and if it was the same context then there's a
chance we skip sending a redundant notification.

Fixes #435
2021-01-13 16:12:35 +01:00
Xavier Morel
e175609950 [IMP] forwardport: unmodified fw automatically inherit overrides
Before this change, a CI override would have to be replicated on most
/ all forward-ports of the base PR. This was intentional to see how it
would shake out, the answer being that it's rather annoying.

Also add a `statuses_full` computed field on PRs for the aggregate
status: the existing `statuses` field is just a copy of the commit
statuses which I didn't remember I kept free of the overrides so the
commit statuses could be displayed "as-is" in the backend (the
overrides are displayed separately). And while at it fix the PR
dashboard to use that new field: that was basically the intention but
then I went on to use the "wrong" field hence #433.

Mebbe the UI part should be displayed using a computed M2M (?)
as a table or as tags instead? This m2m could indicate whether the
status is an override or an "intrinsic" status.

Also removed some dead code:

* leftover from the removed tagging feature (removed the tag
  manipulation but forgot some of the setup / computations)
* unused local variables
* an empty skipped test case

Fixes #439.

Fixes #433.
2021-01-13 16:11:14 +01:00
Xavier Morel
d751cf46a0 [IMP] forwardport: add logging to branch reinsertion
When a new branch is created between other branches, the process to
try and look for forward-port PRs to create to "fill in" currently has
no logging so it's very difficult to figure out why it decides not to
do something.

Add some logging to the process to try and better understand what
happens.

Fixes #441
2021-01-12 12:54:37 +01:00
Xavier Morel
5cf3617eef [IMP] runbot_merge: split out content after separator when creating commit
Currently when creating *merges* the commit message is the
concatenation of the PR title and the PR body.

However it can be convenient to include description test which would
not be included in the commit message e.g. PR template
stuff. "Thematic breaks" seem like a good way to do this: the commit
message will only include the content preceding the first thematic
break, everything else is ignored (except headings which are not
ignored, double negations FTW).

Might be that that's a bit excessive and we should only ignore what
follows the *last* thematic break. Or that there needs to be a more
specific marker. We'll see.

Fixes #443.
2021-01-12 12:24:34 +01:00
Xavier Morel
842b1e4b96 [FIX] mergebot: handle multiples of each command
Because the commands were collected in a `dict[Command, Params]` a
command could only be present once *in the mergebot* (the forwardbot
already collected commands in a list).

* Update mergebot commands parser to collect commands in a list.
* Improve override to allow a comma-separated list of CIs to override.
* Simplify the parsing of delegate to generate one delegate command
  per target (slightly less efficient if multiple logins are provided
  but that is likely extremely rare).

Fixes #445
2021-01-12 09:54:45 +01:00
Xavier Morel
4b5dfc2005 [FIX] forwardport: broken fp if intermediate commit requires renamelimit=0
On a cherrypick failing due to renamelimit issues, the cherrypick
would be retried after resetting the target to its *original* commit.

This only works correctly if the first commit works after a retry, if
a latter commit has to be retried then it gets re-cherry-picked onto
the target branch rather than its own parent.

Fix by remembering the previously successful cherry-picked commit and
resetting to *that*. However I can't really test it because it's
rather hard to get into a situation where the rename detection fails
using synthetic tests.

While at it, clean the logs by stripping the "performing inexact
rename detection" stuff from all stderr (both the CherrypickError from
which it was already stripped and the debug messages).

Fixes #444
2021-01-11 14:43:36 +01:00
Christophe Monniez
f5034e5718 [IMP] runbot: add a button to create a new rebased batch 2020-12-09 13:32:52 +01:00
Christophe Monniez
e769a9d2af [FIX] runbot: improve the way a database name is choosen for running
When a build is reaching the run_run_odoo step, a database has to be
set. If none are found in the build params, the one from the last step
is choosen. Historically, the last one in the `Split` config was `all`
but now, the last one is `base`.

With this commit, if none are found in build params, `all` is choosen if
found in any install config steps. As a default, the one from the first
step is choosen.
2020-12-04 15:03:37 +01:00
Martin Trigaux
8c625278c3 [FIX] runbot_merge: correct Linked PRs URLs
It was not evaluated and rendered to something like
  https://mergebot.odoo.com/odoo/odoo/pull/linked.url

Courtesy of Xavier Dollé (xdo)
2020-12-03 11:01:36 +01:00
Christophe Monniez
3b6487b969 [FIX] runbot: fix _is_docker_step to incidentallly fix log menu
Since 3657a65b20 docker_run is called outside of the step and python
steps have to set the `docker_params` variable. This breaks the computed
`log_list` because the string `docker_run(` is searched in python
steps code to determine if it's a docker_step.

With this commit, the `docker_params = ` is searched instead.
2020-11-26 18:05:00 +01:00
Xavier-Do
7519406e3f [FIX] runbot: to_upgrade for complement should use base_id 2020-11-26 13:35:09 +01:00
David James
190f2ed356 [IMP] runbot: serve badges for projects which are not accessible to the public 2020-11-24 15:18:20 +01:00
Xavier-Do
4ab669e371 [IMP] runbot: add to_upgrade option
For now, the sticky flag is used to define bundle to use as target to upgrade.
The plan for future is to continue to test upgrade, even if the bundle is not sticky anymore.
This new flag will allow to remove sticky for old branche.
This will also allow to disable upgrda tests for still sticky branches when needed.

This commit also filter source bundle based on this flag
(before that all base bundle where used as source, even if not sticky)
2020-11-24 15:14:23 +01:00
Christophe Monniez
6aab868883 [IMP] runbot: add a commit form and search 2020-11-24 15:13:59 +01:00
Xavier-Do
d76a41537c [IMP] runbot: add batch_dependent triggers
Params mechanism are a way to avoid to rebuild with the exact same params.
Commit is the main identifier to know if two builds are the same, ususally
commit_link are not usefull to uniquify params, exept when the mergebase is
in use as in security check (based on diff).

We would also like to avoid to have red diff-based build in sticky/base branches when the
ci is ignored on a pr, but we want to keep the diff display on new batches
(lines and files changed).
We need a way to know if it is a is_base bundle. This was an initially forbidden behaviour
because of duplicate detection.

This commit add a batch_dependent flag. This will enable the use of the batch in the
fingerprint, making it unique per batch and disabling duplicate detection/ in this
case we can use the "is_base" information from the bundle + avoid false duplicate
detection if trigger is based on mergebase-commit diff. (pr based on another pr now merged)
2020-11-24 15:13:00 +01:00