Commit Graph

1630 Commits

Author SHA1 Message Date
Xavier Morel
52b6776204 [FIX] forwardport: add override to repository view
Otherwise the forwardport target can not be set, which is a bit of an
issue.
2021-08-24 15:39:47 +02:00
Xavier Morel
f6aff3e71c [FIX] *: prevent merging conflicts commits with loss of authorship
Proper attribution is important in general, but especially for
external contributors. Before this change, and the previous change
fixing authorship deduplication, it was rather easy for a "squashed"
conflict commit (attributed to the 'bot for lack of a really clean
option) to get merged by mistake.

This commit changes two things:

* The mergebot now refuses to stage commits without an email set, the
  github API rejects those commits anyway so any integration mode
  other than `merge` would fail, just with a very unclear error
* The forwardbot now creates commits with an empty author / committer
  email when the pull request as a whole has multiple authors /
  committers. This leverages the mergebot update.

Also clean up the staging process to provide richer error reporting
using bespoke exceptions instead of simple assertions. I'm not sure
we've ever encountered most of these errors so they're really sanity
checks but the old reporting would be... less than great.

Fixes #505
2021-08-24 15:39:47 +02:00
Xavier Morel
d249417ceb [FIX] forwardport: fix deduplication of authorship in multi-pr conflict
a45f7260fa had intended to use the
original authorship information for conflict commit even if there were
multiple commits, as long as there was only one author (/ committer)
for the entire sequence.

Sadly the deduplication was buggy as it took the *authorship date* in
account, which basically ensured commits would never be considered as
having the same authorship outside of tests (where it was possible for
commits to be created at the same second).

Related to #505
2021-08-24 15:39:47 +02:00
Xavier Morel
32829cf880 [FIX] forwardport: missing login in feedback message
Closes #503
2021-08-24 15:39:47 +02:00
Xavier Morel
747174f610 [FIX] runbot_merge: when fetching a PR, sync closed state
If a PR is closed on github and unknown by the mergebot, when fetched
it should be properly sync'd as "closed" in the backend, otherwise the
PR can get in a weird state and cause issues.

Also move the "I fetched the thing" comment before the actual creation
of the PR for workflow clarity, otherwise the reader has the
impression that the 'bot knew about the PR then fetched it anyway.

And improve savepoint management around the fetching: savepoints
should be released in all cases.

Closes #488.
2021-08-24 15:39:47 +02:00
Xavier Morel
f54c016ef9 [FIX] runbot_merge: link warning on PRs of different projects
If two PRs have the same label *in different projects entirely*, the
mergebot should not consider them to be linked, but it did as shown by
the warning message on odoo-dev/odoo#905 (two PRs created from the
same branch in different projects were seen as linked by the status
checker).

3b417b16a1 fixed the actual staging
selection, it's only the warning which did not properly segregate PRs.

Only group PRs which target the same branch (therefore are within the
same project).

Fixes #487
2021-08-24 15:39:47 +02:00
Xavier Morel
670f56b491 [ADD] forwardport: views of outstanding forwardports
Though the forwardport posts regular reminders that an fw is outdated,
it can be easy to miss for the non-subject (and apparently the
subjects often just ignore the information entirely).

Add a few relevant links there:

* on PR pages, add a link to either the source or the
  forward-ports (if applicable), as well as the merge date
* add a new page which lists all the PRs with outstanding
  forwardports, as well as the forwardports in question

Fixes #474
2021-08-24 15:39:47 +02:00
Xavier Morel
ca2742a12c [IMP] forwardport: error handling when creating PR
Don't try to parse the response as JSON in the error case(s): if the
errors are bad enough github can return complete non-parseable
garbage.

Only access the "text" property (response body, decoded, but unparsed)
in error cases, only parse in the success case.

Also avoid reusing variables for completely different values, even if
they're of the same type, especially if they can overlap.

fixes #470
2021-08-24 15:39:47 +02:00
Xavier Morel
8178b64c01 [IMP] forwardport: error message when trying to r+ via fwbot
Initial thinking was to remove the check entirely and leave it to the
mergebot, but the lack of error reporting / forwarding means while
technically correct it would probably be somewhat difficult to grok.

Instead, improve the error reporting:

* add a dedicated message when trying to r+ via fwbot on a non-fw
  PR (note: maybe the fwbot should not care? and just send it as-is
  to the mergebot in that case?)
* clarify the ACL error
* post both message as the forwardbot rather than the mergebot

Also add a missing token note for the feedback from the forwardport
limit.

fix #469
2021-08-24 15:39:47 +02:00
Xavier Morel
6b1f698c23 [IMP] forwardport: handling of updates causing conflicts on followups
If a PR is updated and has extent forward-ports, those forwardports
get updated automatically ("followup").

However there is an issue if the udpate causes a conflict in the
followup: the conflict gets silently pushed, and may fairly easily get
merged if it occurs in an area which the CI doesn't cover.

It's unclear what the policy really should be for this issue, and
there is no real way to *block* a pull request at the moment (save by
putting it in error at the mergebot level I guess?), so for now
clearly notify the user on both the modified PR and the followup,
with a comment on both.

We may want to revisit this eventually.

Fixes #467
2021-08-24 15:39:47 +02:00
Xavier Morel
f10d33ee85 [FIX] fwbot: properly prevent @up to on forward-port PRs
There was already a check, but the way the check behaved
means *detached* PRs would not be prevented from setting their
forward-port, despite that not doing anything.

Fix it by checking if the current PR has a source, not a parent.

Fixes #465
2021-08-24 15:39:47 +02:00
Xavier Morel
6a8c13b1ef [IMP] runbot_merge: show linked PRs during staging and after merging
Previously, a PR's status page would only show the linked / related
PRs when `open`.

Since the relations between PRs remains useful, also make this
information available during staging and after merging.

Fixes #463
2021-08-24 15:39:47 +02:00
Xavier Morel
c1ebe9da52 [IMP] runbot_merge: format of staging timestamps
* in the main dashboard, show the exact UTC timestamp (with a Z
  marker) on hover, not just the relative delta
* in the branch details page, show the full timestamp, zoned, in
  ISO-8601 format
2021-08-24 15:39:47 +02:00
Xavier Morel
e542dfc852 [IMP] repo creation error handling + warnings
* The repo would only be registered at the very end of the creation,
  meaning an error *during* the repo creation (e.g. while uploading the
  first blob or setting up webhooks) would leave the repository
  undeleted. Register the repository as soon as we know it was
  created, in order to correctly dispose of it afterwards.
* Migrate logging.warning call to warnings.warn on repository deletion
  failure: pytest will print warnings during its reporting, not so for
  log warnings (?)
2021-08-24 15:39:47 +02:00
Xavier-Do
b55e38ae2d [FIX] runbot: wait for chart.js to be loaded
Since the frontend_assets are loaded with `defer="defer"`,
the page sometimes fail with the message:
```
stats.js:212 Uncaught ReferenceError: Chart is not defined
    at updateChart (stats.js:212)
    at stats.js:158
    at XMLHttpRequest.xhttp.onreadystatechange (stats.js:53)
```

This commit checks that Chart is available before tring to render the graph.

Thanks to @kebeclibre for the help.
2021-08-19 15:49:41 +02:00
Xavier-Do
92f8c093bd [IMP] runbot: add childs stats to build stats 2021-08-12 11:27:29 +02:00
Xavier-Do
eb111002ed [IMP] runbot: make stagings automatically build all 2021-08-12 11:27:29 +02:00
Xavier-Do
2b8e5c5a91 [IMP] runbot: remove _log_access on build_stat
_log_access is not usefull, and increase disk space used by stats.
2021-08-04 10:46:59 +02:00
Xavier-Do
39eeb73f71 [FIX] runbot: fix keep_running
Keep running was broken for since 5.0.
This commit fixes the broken logic and adds an index
2021-08-04 10:46:59 +02:00
Christophe Monniez
b7a5dc3d8c [IMP] runbot: add an error manager group 2021-08-04 10:42:34 +02:00
Christophe Monniez
01371c5270 [REM] runbot: remove dead code 2021-08-04 10:40:13 +02:00
Andrii Skrypka
5b9c20877e [FIX] runbot: correctly get number type value from icp 2021-08-03 09:47:11 +02:00
Xavier-Do
b843fee8d9 [FIX] runbot: adapt routes and licenses 2021-07-27 16:11:49 +02:00
Xavier-Do
726c7ba423 [IMP] runbot: add static_run button to slots buttons. 2021-07-20 13:43:02 +02:00
Xavier-Do
de927d89e7 [IMP] runbot: cleanup and improve hook
When getting pull info, the alive state can be determined easily,
meaning that this field can join the "_compute_branch_infos" familly

Hook was catching some changes made on pr and was conditionnaly updating some fields
and triggering some other operations conditionaly depending on the action flag.
All of the information needed to update the pull info should always be present in the
 payload body, meaning that all fields can be updated at once in case some hook was missed,
 and additionnal operation can be triggered based on fields changes.
2021-07-20 13:24:01 +02:00
Xavier-Do
d9a09806e2 [IMP] runbot: add a static run link for documentation builds 2021-07-20 13:22:57 +02:00
Christophe Monniez
847622552f [IMP] runbot: limit memory usage of containers
In some conditions, it appears that a containerized build can eat up
all memory of the container host. This leads to disturbance of other
builds as the kernel OOM killer enters the dance.

With this commit, the docker ability to limit memory usage of a
container is used. The OOM killer will choose its victim among the
container processes.

The containers memory limit has to be set in the runbot settings. If not
set, no memory limit is used.
2021-07-15 11:15:38 +02:00
Christophe Monniez
4b16e889eb [FIX] runbot: update matplotlib to focal version
Matplotlib 3.0.2 cannot be `pip installed` on Focal.
2021-07-15 11:05:08 +02:00
Christophe Monniez
363c488c10 [IMP] runbot: add a reviewers field 2021-07-07 11:05:51 +02:00
Xavier-Do
87214a82a6 [IMP] info: log level debug to info
runbot servers are running with a log-level debug in order to have usefull
debug information, but this also causes some noise comming from odoo.

This pr changes most debug to info.
2021-07-02 15:05:42 +02:00
Xavier-Do
011bdc00e2 [IMP] add draft pr management + multiple custom fields
- Add draft pr management to avoid to trigger code owner on draft pr.
- Add check on falsy config on trigger id (avoid crash, usefull to disable trigger)
- Add extra params on custom trigger to avoid to write specific config every time.
- Trigger a new batch automaticaly when updating target/draft
2021-06-29 10:13:34 +02:00
Christophe Monniez
6e5d722882 [IMP] runbot: add a Dockerfile on project
Some projects may use a totally different Dockerfile. In order to avoid
new branches of those projects to automatically build with the generic
default Dockerfile, this commit adds the possibility to configure a
Default Dockerfile on a project.
2021-06-29 09:22:41 +02:00
Christophe Monniez
b015ec6840 [IMP] runbot: show bundles that use a Dockerfile
Before manipulating a Dockerfile, it can be useful to know which bundle
is using it.
2021-06-29 09:22:41 +02:00
Christophe Monniez
67d020a8b3 [IMP] runbot: update docker default for focal
Since Odoo 14.0, the recommended Ubuntu LTS release is Focal, the
default Docker should be updated accordingly.

A custom template with bionic have to be manually created on runbot
instances that still build Odoo < 14.0.

* A small change is made in the templates logic that builds the
  Dockerfile: a `runbot_pip` dict entry now exists in order to install the
  python libs required by the runbot. On the other hand, `additional_pip`
  should only be used to install optional python libs for Odoo.

* Upgrade Chrome version to 90.0.4430.93-1 as this one is currently in
  use on our current runbot instance. Just keep in mind that the Odoo
  screencast feature does not work anymore since Chrome 88.

* gsfont is added because of a bug [0] that affects python-reportlab in
  Focal.

* pyCrypto package is removed. It was used in an Odoo addon that
  disappeared in odoo/odoo@2738341c21

* dbfread, websocket-client are now installed as a deb package as they exists in Bionic and
  Focal

* pdfminer.six is now removed because a deb package exists in Focal but
  not in Bionic. It means that it has to be added in deb_packages_python
  in Dockerfiles for odoo > 13.0 and in additional_pip for odoo <= 13.0

[0]: https://bugs.launchpad.net/ubuntu/+source/python-reportlab/+bug/1918107
2021-06-29 09:22:41 +02:00
Xavier-Do
e65ebb570d [IMP] runbot: change ahead/behind visibility 2021-05-19 11:45:06 +02:00
Alexandre Fayolle
3c760870b5 [FIX] runbot: bug in duplicate module detection
The test in the original code will never fire because the value searched
for is not in the keys of the dictionary, but in one of the lists which
are in the values. Work around this by maintaining a reverse dictionary
module name -> commit and use this for the test.
2021-05-18 12:37:42 +02:00
Xavier-Do
0a37ff2f90 [FIX] runbot: don't log twice 2021-05-17 14:43:48 +02:00
Xavier-Do
6d4efcb470 [FIX] runbot: avoid failure for duplicate warnings 2021-05-17 14:23:18 +02:00
Christophe Monniez
b14a73b1d4 [IMP] runbot: enhance dockerfile tree view
When a dockerfile to_build field is False for any reason and a new
runbot is setup, it's easy to miss the point and builds that involves
this particular Dockerfile will fail on the new runbot.

With this commit, the Dockerfiles tree view is improved to easily spot
those kind of problems:

- to_build field is visible and the Falsy lines are in yellow (warning)
- the empty Dockerfile's are in red (danger)
- versions are now visible in the tree view too
2021-05-17 14:11:34 +02:00
Christophe Monniez
642844fdb5 [IMP] runbot: show red slots in batch tile
When a build slot is hidden in the batch tile but is responsible of the
batch failure, the failure reason may not be obvious for the user.

With this commit, an hidden slot appears if the slot build is in
failure.
2021-05-17 14:11:03 +02:00
Xavier-Do
5d3a2de698 [FIX] runbot: don't fail all branch discovery if one pull info fails.
Sometimes a pr pull info can fail.
- Most of the time it is only temporary and it will be successfull on next try.
- In some rare case the pr will always fail (github inconsistency) The pr exist in git but not on github api.
For this rare case, we store the pr in memory in order to unstuck other pr/branches update.
We consider that this error should not remain, in this case github needs to fix the inconsistency.
This is why the runbot model don't handle such a case for now.
Another solution would be to create the pr with fake pull info. This idea is not the best one
since we want to avoid to have many pr with fake pull_info in case of temporary failure of giothub services.
With this solution, the pr will be retried once every cron loop.
We dont except to have pr with this kind of persistent failure more than every few mounths/years.
2021-05-17 14:07:55 +02:00
Xavier-Do
514de022f4 [FIX] runbot: fix markdown code
When code blocks were containing markdown like text, the inside of the code
block was also formated.

This commit removes the code blocks before applying other formating and
place them back at the end.

closes #481
2021-05-10 15:11:21 +02:00
Xavier-Do
6279dfa442 [IMP] runbot: imp stat display:
- Adds a complete legend enabelling to display a custom subset of modules.
This is mainly to enable a vertical scroll on list since chart-js default
legend will be displayed on multiple column.

- Adds a "Noisy" order mode to find non-deterministic modules.

- Changes the build selection mode to a center one to easylly center
build of interrest and add a forward button.

- Small ui tweaks/fix to match new selection logic.
2021-04-13 10:35:30 +02:00
Xavier-Do
8e53dbd0db [FIX] runbot: hide testing builds from stats views. 2021-04-13 10:35:30 +02:00
Xavier-Do
84d9425bdc [FIX] runbot: various view fixes
- fix missing batch references on builds
- fix group on action buttons
2021-04-09 14:58:17 +02:00
Christophe Monniez
57bd00672d [IMP] runbot: add a chart page for build stats
Since 360e31ade4, it's possible to add statistics values to build
results but there was no practical way to analyze them.

With this commit, there is a new button on the bundle page that leads to
a chart page that displays those values.

The default reference build is last known good build of the bundle.
Values are filtered by key and only the most significant values are
displayed. The user can then refine the chart by changing the reference
build or the key and a few other options.

Co-author: Xavier-Do <xdo@odoo.com>
2021-04-09 14:10:37 +02:00
David James
19c312d92c [IMP] runbot: add an option to exclude paths from coverage report 2021-04-07 15:47:11 +02:00
Christophe Monniez
785a7796fb [IMP] runbot: allow conditional pip requirements
Before this commit, the `requirements.txt` from a specified odoo branch
(master by default) was always installed in the Dockerfile's.

We now allow to disable this feature to test Odoo with vanilla
distributions.
2021-04-07 11:35:33 +02:00
Christophe Monniez
6bd74c07c4 [IMP] runbot: allow to install Chrome from google
When choosing to install Chrome in a Dockerfile, the chrome version is
downloaded from Odoo nightly server. This make it difficult to test
with different versions of Chrome.

With this commit, we allow to install from Google in Docker files.

By default, the install remains from Odoo Nightly server but if the key
`custom_values['chrome_source']` is set to 'google' in a Dockerfile,
the specified version will be downloaded from Google servers when the
Docker image is built.
2021-04-07 11:35:33 +02:00
Christophe Monniez
50f803ec31 [FIX] runbot: allow larger upload on runbot odoo instances 2021-04-07 11:01:35 +02:00