Commit Graph

835 Commits

Author SHA1 Message Date
Xavier-Do
b252800205 [IMP] runbot: add sum on stats 2021-11-25 09:42:55 +01:00
Christophe Monniez
cfac0a11ef [FIX] runbot: prevent showing old fixed build error
When a build error is archived, a linked children with an assigned fixer
may still appear on the error frontend page.

With this commit, old children are not showing up again.
2021-09-14 16:22:16 +02:00
Xavier-Do
2604d962e9 [FIX] runbot_build_stats
Since the order was changed, the first values are actually the older ones.
This commit inverse newer_build_stats and older_build_stats
values in order to always have the new keys. Before this commit the new
keys where not displayed. A future improvement may be to combine keys
from all builds.

This commit also proposes to give a 0 value if the key did not exist in
the older build. This means that new keys will appear with a big
difference. This is maybe not a good idea and needs some testing. A
better solution would be to search for the first apparition.
2021-09-06 13:20:16 +02:00
Xavier-Do
b55e38ae2d [FIX] runbot: wait for chart.js to be loaded
Since the frontend_assets are loaded with `defer="defer"`,
the page sometimes fail with the message:
```
stats.js:212 Uncaught ReferenceError: Chart is not defined
    at updateChart (stats.js:212)
    at stats.js:158
    at XMLHttpRequest.xhttp.onreadystatechange (stats.js:53)
```

This commit checks that Chart is available before tring to render the graph.

Thanks to @kebeclibre for the help.
2021-08-19 15:49:41 +02:00
Xavier-Do
92f8c093bd [IMP] runbot: add childs stats to build stats 2021-08-12 11:27:29 +02:00
Xavier-Do
eb111002ed [IMP] runbot: make stagings automatically build all 2021-08-12 11:27:29 +02:00
Xavier-Do
2b8e5c5a91 [IMP] runbot: remove _log_access on build_stat
_log_access is not usefull, and increase disk space used by stats.
2021-08-04 10:46:59 +02:00
Xavier-Do
39eeb73f71 [FIX] runbot: fix keep_running
Keep running was broken for since 5.0.
This commit fixes the broken logic and adds an index
2021-08-04 10:46:59 +02:00
Christophe Monniez
b7a5dc3d8c [IMP] runbot: add an error manager group 2021-08-04 10:42:34 +02:00
Christophe Monniez
01371c5270 [REM] runbot: remove dead code 2021-08-04 10:40:13 +02:00
Andrii Skrypka
5b9c20877e [FIX] runbot: correctly get number type value from icp 2021-08-03 09:47:11 +02:00
Xavier-Do
b843fee8d9 [FIX] runbot: adapt routes and licenses 2021-07-27 16:11:49 +02:00
Xavier-Do
726c7ba423 [IMP] runbot: add static_run button to slots buttons. 2021-07-20 13:43:02 +02:00
Xavier-Do
de927d89e7 [IMP] runbot: cleanup and improve hook
When getting pull info, the alive state can be determined easily,
meaning that this field can join the "_compute_branch_infos" familly

Hook was catching some changes made on pr and was conditionnaly updating some fields
and triggering some other operations conditionaly depending on the action flag.
All of the information needed to update the pull info should always be present in the
 payload body, meaning that all fields can be updated at once in case some hook was missed,
 and additionnal operation can be triggered based on fields changes.
2021-07-20 13:24:01 +02:00
Xavier-Do
d9a09806e2 [IMP] runbot: add a static run link for documentation builds 2021-07-20 13:22:57 +02:00
Christophe Monniez
847622552f [IMP] runbot: limit memory usage of containers
In some conditions, it appears that a containerized build can eat up
all memory of the container host. This leads to disturbance of other
builds as the kernel OOM killer enters the dance.

With this commit, the docker ability to limit memory usage of a
container is used. The OOM killer will choose its victim among the
container processes.

The containers memory limit has to be set in the runbot settings. If not
set, no memory limit is used.
2021-07-15 11:15:38 +02:00
Christophe Monniez
363c488c10 [IMP] runbot: add a reviewers field 2021-07-07 11:05:51 +02:00
Xavier-Do
87214a82a6 [IMP] info: log level debug to info
runbot servers are running with a log-level debug in order to have usefull
debug information, but this also causes some noise comming from odoo.

This pr changes most debug to info.
2021-07-02 15:05:42 +02:00
Xavier-Do
011bdc00e2 [IMP] add draft pr management + multiple custom fields
- Add draft pr management to avoid to trigger code owner on draft pr.
- Add check on falsy config on trigger id (avoid crash, usefull to disable trigger)
- Add extra params on custom trigger to avoid to write specific config every time.
- Trigger a new batch automaticaly when updating target/draft
2021-06-29 10:13:34 +02:00
Christophe Monniez
6e5d722882 [IMP] runbot: add a Dockerfile on project
Some projects may use a totally different Dockerfile. In order to avoid
new branches of those projects to automatically build with the generic
default Dockerfile, this commit adds the possibility to configure a
Default Dockerfile on a project.
2021-06-29 09:22:41 +02:00
Christophe Monniez
b015ec6840 [IMP] runbot: show bundles that use a Dockerfile
Before manipulating a Dockerfile, it can be useful to know which bundle
is using it.
2021-06-29 09:22:41 +02:00
Christophe Monniez
67d020a8b3 [IMP] runbot: update docker default for focal
Since Odoo 14.0, the recommended Ubuntu LTS release is Focal, the
default Docker should be updated accordingly.

A custom template with bionic have to be manually created on runbot
instances that still build Odoo < 14.0.

* A small change is made in the templates logic that builds the
  Dockerfile: a `runbot_pip` dict entry now exists in order to install the
  python libs required by the runbot. On the other hand, `additional_pip`
  should only be used to install optional python libs for Odoo.

* Upgrade Chrome version to 90.0.4430.93-1 as this one is currently in
  use on our current runbot instance. Just keep in mind that the Odoo
  screencast feature does not work anymore since Chrome 88.

* gsfont is added because of a bug [0] that affects python-reportlab in
  Focal.

* pyCrypto package is removed. It was used in an Odoo addon that
  disappeared in odoo/odoo@2738341c21

* dbfread, websocket-client are now installed as a deb package as they exists in Bionic and
  Focal

* pdfminer.six is now removed because a deb package exists in Focal but
  not in Bionic. It means that it has to be added in deb_packages_python
  in Dockerfiles for odoo > 13.0 and in additional_pip for odoo <= 13.0

[0]: https://bugs.launchpad.net/ubuntu/+source/python-reportlab/+bug/1918107
2021-06-29 09:22:41 +02:00
Xavier-Do
e65ebb570d [IMP] runbot: change ahead/behind visibility 2021-05-19 11:45:06 +02:00
Alexandre Fayolle
3c760870b5 [FIX] runbot: bug in duplicate module detection
The test in the original code will never fire because the value searched
for is not in the keys of the dictionary, but in one of the lists which
are in the values. Work around this by maintaining a reverse dictionary
module name -> commit and use this for the test.
2021-05-18 12:37:42 +02:00
Xavier-Do
0a37ff2f90 [FIX] runbot: don't log twice 2021-05-17 14:43:48 +02:00
Xavier-Do
6d4efcb470 [FIX] runbot: avoid failure for duplicate warnings 2021-05-17 14:23:18 +02:00
Christophe Monniez
b14a73b1d4 [IMP] runbot: enhance dockerfile tree view
When a dockerfile to_build field is False for any reason and a new
runbot is setup, it's easy to miss the point and builds that involves
this particular Dockerfile will fail on the new runbot.

With this commit, the Dockerfiles tree view is improved to easily spot
those kind of problems:

- to_build field is visible and the Falsy lines are in yellow (warning)
- the empty Dockerfile's are in red (danger)
- versions are now visible in the tree view too
2021-05-17 14:11:34 +02:00
Christophe Monniez
642844fdb5 [IMP] runbot: show red slots in batch tile
When a build slot is hidden in the batch tile but is responsible of the
batch failure, the failure reason may not be obvious for the user.

With this commit, an hidden slot appears if the slot build is in
failure.
2021-05-17 14:11:03 +02:00
Xavier-Do
5d3a2de698 [FIX] runbot: don't fail all branch discovery if one pull info fails.
Sometimes a pr pull info can fail.
- Most of the time it is only temporary and it will be successfull on next try.
- In some rare case the pr will always fail (github inconsistency) The pr exist in git but not on github api.
For this rare case, we store the pr in memory in order to unstuck other pr/branches update.
We consider that this error should not remain, in this case github needs to fix the inconsistency.
This is why the runbot model don't handle such a case for now.
Another solution would be to create the pr with fake pull info. This idea is not the best one
since we want to avoid to have many pr with fake pull_info in case of temporary failure of giothub services.
With this solution, the pr will be retried once every cron loop.
We dont except to have pr with this kind of persistent failure more than every few mounths/years.
2021-05-17 14:07:55 +02:00
Xavier-Do
514de022f4 [FIX] runbot: fix markdown code
When code blocks were containing markdown like text, the inside of the code
block was also formated.

This commit removes the code blocks before applying other formating and
place them back at the end.

closes #481
2021-05-10 15:11:21 +02:00
Xavier-Do
6279dfa442 [IMP] runbot: imp stat display:
- Adds a complete legend enabelling to display a custom subset of modules.
This is mainly to enable a vertical scroll on list since chart-js default
legend will be displayed on multiple column.

- Adds a "Noisy" order mode to find non-deterministic modules.

- Changes the build selection mode to a center one to easylly center
build of interrest and add a forward button.

- Small ui tweaks/fix to match new selection logic.
2021-04-13 10:35:30 +02:00
Xavier-Do
8e53dbd0db [FIX] runbot: hide testing builds from stats views. 2021-04-13 10:35:30 +02:00
Xavier-Do
84d9425bdc [FIX] runbot: various view fixes
- fix missing batch references on builds
- fix group on action buttons
2021-04-09 14:58:17 +02:00
Christophe Monniez
57bd00672d [IMP] runbot: add a chart page for build stats
Since 360e31ade4, it's possible to add statistics values to build
results but there was no practical way to analyze them.

With this commit, there is a new button on the bundle page that leads to
a chart page that displays those values.

The default reference build is last known good build of the bundle.
Values are filtered by key and only the most significant values are
displayed. The user can then refine the chart by changing the reference
build or the key and a few other options.

Co-author: Xavier-Do <xdo@odoo.com>
2021-04-09 14:10:37 +02:00
David James
19c312d92c [IMP] runbot: add an option to exclude paths from coverage report 2021-04-07 15:47:11 +02:00
Christophe Monniez
785a7796fb [IMP] runbot: allow conditional pip requirements
Before this commit, the `requirements.txt` from a specified odoo branch
(master by default) was always installed in the Dockerfile's.

We now allow to disable this feature to test Odoo with vanilla
distributions.
2021-04-07 11:35:33 +02:00
Christophe Monniez
6bd74c07c4 [IMP] runbot: allow to install Chrome from google
When choosing to install Chrome in a Dockerfile, the chrome version is
downloaded from Odoo nightly server. This make it difficult to test
with different versions of Chrome.

With this commit, we allow to install from Google in Docker files.

By default, the install remains from Odoo Nightly server but if the key
`custom_values['chrome_source']` is set to 'google' in a Dockerfile,
the specified version will be downloaded from Google servers when the
Docker image is built.
2021-04-07 11:35:33 +02:00
Christophe Monniez
50f803ec31 [FIX] runbot: allow larger upload on runbot odoo instances 2021-04-07 11:01:35 +02:00
Andrius Laukavičius
5337ecd11f [FIX] runbot: _compute_host_id
If you try to manually create bundle, Odoo will crash, because it will
try to use `name` value that is not set yet. For that we start computing
host_id once `name` is entered.
2021-04-07 11:00:55 +02:00
Xavier-Do
0ed4728518 [IMP] runbot: use last step in wakeup when possible 2021-04-07 10:31:05 +02:00
Christophe Monniez
0f4610c8bc [FIX] runbot: remove forgotten print 2021-03-16 10:45:47 +01:00
Xavier-Do
46269ada70 [FIX] runbot: fix logged message 2021-03-15 13:12:48 +01:00
Xavier-Do
c538bd17f8 [IMP] runbot: use build database_ids to generate connect links
aka: make it clean

This build database_ids field was generated a few months, waiting for the database to update with
this new sceme before using it.

It is still a little early to use it in cleanup methods, but this can be used to
generate the connect links dynamicaly.

To follow al specs introduced in previous commit, main fa-sign-in button should link to the -all.
It will almost always be the first one in database_ids in alphabetic order.
Then, in the dropdown, all other database are listed.

This will fix the previously broken design_theme connect link (no base nor all).

For this purpose nginx regex needs to be adapted to accept database name with underscore.
2021-03-01 15:49:23 +01:00
Xavier-Do
26ad74051f [IMP] runbot: apply al connect button spec 2021-02-26 12:13:39 +01:00
Xavier-Do
38b52411fa [FIX] runbot: use database name subdomain
Using database name as subdomain will set the web.base.url correctly when connection for the first time using this link
This will unfortunatelly break connect all and connect base links if dns and nginx are not setup.
This won't fix the web.base.url when connecting with the Connect (database selector) button either.
2021-02-22 16:03:51 +01:00
Christophe Monniez
f28485b6a8 [FIX] runbot: allow to retry to send a status
When a status cannot be sent to github, the status is created in
database but the `sent_date` field is not set.

Because of that, the status cannot be sent manually from the frontend.

With this commit, that kind of status can by tried again.
2021-02-02 13:32:03 +01:00
Christophe Monniez
4f4005896c [FIX] runbot: browse a valid record id in test_warning
Since odoo/odoo@824a651 the test_warning_from_runbot_abstract was
failing for a good reason, the `browse` parameter was a recordset
instead of an id.
2021-02-02 11:48:03 +01:00
Christophe Monniez
f5034e5718 [IMP] runbot: add a button to create a new rebased batch 2020-12-09 13:32:52 +01:00
Christophe Monniez
e769a9d2af [FIX] runbot: improve the way a database name is choosen for running
When a build is reaching the run_run_odoo step, a database has to be
set. If none are found in the build params, the one from the last step
is choosen. Historically, the last one in the `Split` config was `all`
but now, the last one is `base`.

With this commit, if none are found in build params, `all` is choosen if
found in any install config steps. As a default, the one from the first
step is choosen.
2020-12-04 15:03:37 +01:00
Christophe Monniez
3b6487b969 [FIX] runbot: fix _is_docker_step to incidentallly fix log menu
Since 3657a65b20 docker_run is called outside of the step and python
steps have to set the `docker_params` variable. This breaks the computed
`log_list` because the string `docker_run(` is searched in python
steps code to determine if it's a docker_step.

With this commit, the `docker_params = ` is searched instead.
2020-11-26 18:05:00 +01:00
Xavier-Do
7519406e3f [FIX] runbot: to_upgrade for complement should use base_id 2020-11-26 13:35:09 +01:00
David James
190f2ed356 [IMP] runbot: serve badges for projects which are not accessible to the public 2020-11-24 15:18:20 +01:00
Xavier-Do
4ab669e371 [IMP] runbot: add to_upgrade option
For now, the sticky flag is used to define bundle to use as target to upgrade.
The plan for future is to continue to test upgrade, even if the bundle is not sticky anymore.
This new flag will allow to remove sticky for old branche.
This will also allow to disable upgrda tests for still sticky branches when needed.

This commit also filter source bundle based on this flag
(before that all base bundle where used as source, even if not sticky)
2020-11-24 15:14:23 +01:00
Christophe Monniez
6aab868883 [IMP] runbot: add a commit form and search 2020-11-24 15:13:59 +01:00
Xavier-Do
d76a41537c [IMP] runbot: add batch_dependent triggers
Params mechanism are a way to avoid to rebuild with the exact same params.
Commit is the main identifier to know if two builds are the same, ususally
commit_link are not usefull to uniquify params, exept when the mergebase is
in use as in security check (based on diff).

We would also like to avoid to have red diff-based build in sticky/base branches when the
ci is ignored on a pr, but we want to keep the diff display on new batches
(lines and files changed).
We need a way to know if it is a is_base bundle. This was an initially forbidden behaviour
because of duplicate detection.

This commit add a batch_dependent flag. This will enable the use of the batch in the
fingerprint, making it unique per batch and disabling duplicate detection/ in this
case we can use the "is_base" information from the bundle + avoid false duplicate
detection if trigger is based on mergebase-commit diff. (pr based on another pr now merged)
2020-11-24 15:13:00 +01:00
Christophe Monniez
2556145843 [FIX] runbot: do not count linked errors
On the build errors frontend page, the linked errors are displayed too.
With this commit, they are removed from the search query.
2020-11-23 12:15:21 +01:00
Christophe Monniez
733ab2dd43 [IMP] runbot: try to improve dashboard bundle line title 2020-11-18 16:49:45 +01:00
Christophe Monniez
9637976fef [IMP] runbot: bring back copy bundle name button 2020-11-18 16:49:45 +01:00
Christophe Monniez
9c7cee657d [IMP] runbot: improve build error frontend page
On the build error web page, a regular assigned error is not shown to
all users.

With this commit, a regular build error (not only random) will be shown
if the error is assigned.
2020-11-18 16:49:45 +01:00
Christophe Monniez
c6a1d202a1 [IMP] runbot: parse log redirects to the created error
When builds logs are parsed by using the contextual button, the client
stays on the same page even if a build error is created.

With this commit, the client is now redirected to the created/found
build error(s).
2020-11-18 16:49:45 +01:00
Christophe Monniez
e51a27bbbb [FIX] runbot: fix build error creation form 2020-11-18 16:49:45 +01:00
Xavier-Do
87512ae78c [IMP] runbot: improve coverage
- execute coverage results in case of failure
- fix coverage links
- use coverage directly
2020-11-18 13:36:41 +01:00
Xavier-Do
559e10754e [FIX] runbot: fix wakeup 2020-11-12 14:02:00 +01:00
Xavier-Do
3657a65b20 [IMP] runbot: call docker_run outside steps
When calling a step from a python step, it is impossible to alter some parameter, the only solution is to copy all step code.
With this change, python step are now able to override docker_run parameter of another step by modifying the dict returned by run_* steps
and assigning the result to "docker_params".

This will mainly be used to set the correct docker_image for migration pre and post test. This can also be used to alter a command,
like removing the pip install or adding extra pre/post operation on the command.
2020-11-11 10:47:26 +01:00
Christophe Monniez
4dec2c5f5c [FIX] runbot: fix warning during dockerfile test 2020-11-10 17:33:18 +01:00
Christophe Monniez
79f7fc9ed2 [FIX] runbot: fix default image_tag
Also fix Docker build process return code.
2020-11-10 11:03:30 +01:00
Christophe Monniez
981cb7e1b6 [IMP] runbot: use a Dockerfile model
Currently, runbot is using a single Dockerfile maintained in a data file
in the source code. This situation is not convenient for testing Odoo in
different environments.

With this commit, a Dockerfile Odoo model is used to allow usage of
multiple Docker containers.

This model comes with a pre-defined Dockerfile that can be used to build
the current Odoo supported versions (12.0 up to 14.0).
2020-11-10 09:52:48 +01:00
Xavier-Do
92ac1d4737 [FIX] runbot: fix field label 2020-10-20 15:26:41 +02:00
David James
237e931e57 [IMP] runbot: add extra parameters and environment variables to run stages 2020-10-19 11:29:39 +02:00
Xavier-Do
7e2d7c65a1 [FIX] runbot: fix message formating 2020-10-19 01:30:39 +02:00
Xavier-Do
43075a59e6 [IMP] runbot: allow to define errors policy for _git decode 2020-10-15 16:54:47 +02:00
Xavier-Do
1985917702 [FIX] runbot: add limit on log length 2020-10-15 15:55:33 +02:00
Xavier-Do
45034d362d [FIX] runbot: inverted condition 2020-10-15 12:09:40 +02:00
Xavier-Do
bd69c0dfa0 [FIX] runbot: manual is on trigger, not slot 2020-10-15 11:28:20 +02:00
Xavier-Do
f9272cb207 [IMP]: add linitng tools (+bunch of small fixes) 2020-10-15 10:38:36 +02:00
Xavier-Do
c3cc9013df [IMP] runbot: use last pylint version 2020-10-14 18:08:47 +02:00
David James
58eefd3b52 [FIX] runbot: serve badges for non-default projects 2020-10-12 16:39:57 +02:00
Christophe Monniez
91de4f9314 runbot: bump pdfminer.six to version 20200720
Odoo.sh team discrovered that pip also install pycryptodome alongside
with pdfminer 20181108. This installation causes clashes with pycrypto
that may be used elsewehere in Odoo code. This version was used in order
to stick with the Debian version but it appears that Debian fixed that
issue in the deb package [1] and [2]. As the runbot uses the pypi
version instead of the patched Debian package, the runbot could face the
same problems.

So as the issue is fixed in 20200720, this commit will install it in the
Dockerfile.

So we can stay sync with Odoo.sh. Thanks guys.

1.https://metadata.ftp-master.debian.org/changelogs/main/p/pdfminer/pdfminer_20181108+dfsg-3_changelog
2.https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=919008
3.https://github.com/pdfminer/pdfminer.six/releases/tag/20200720
2020-10-01 20:15:45 +02:00
Xavier-Do
67f6eb12c5 [IMP] runbot: improve build order
The old order based on status is not really needed anymore:

- Scheduled builds have a special condition and already have a low priority.
- Indirect builds don't exist anymore
- It is actually questionnable to postpone rebuild. Sometimes they are needed but stuck.

This new proposition will keep subbuild scheduling close to parent build. Some build may take 2 hours
because they are parallelized and children are stuck. Priority should be defined by top parent.
2020-09-25 15:20:26 +02:00
Christophe Monniez
6c3f14ed61 [IMP] runbot: add bundle_ids to ErrorLog SQL view 2020-09-25 11:18:46 +02:00
Xavier-Do
0e2f75c5d6 [FIX] runbot: typo 2020-09-23 14:09:23 +02:00
Xavier-Do
03b82e712a [FIX] runbot: allow to disable screencast globaly 2020-09-23 13:49:35 +02:00
Xavier-Do
b991eb5b70 [IMP] runbot: enable screencast, add access control, small fixes 2020-09-23 13:27:59 +02:00
Christophe Monniez
17f0458d06 [FIX] runbot: host must be null instead of emptry string 2020-09-23 00:29:44 +02:00
Christophe Monniez
6cf0b3e128 [IMP] runbot: add ffmpeg tool to Dockerfile
This will allow screenscasts on runbot.
2020-09-22 19:57:49 +02:00
Christophe Monniez
73426d9348 [IMP] runbot: add pdf417 library
The pdf417gen python library is needed for the l10n_cl_edi. As this module
is not mandatory, the library was not added in the requirements.txt
file.

In order to test the feature on the runbot, this commit adds the library
in the Docker container.

Related work:
    odoo/odoo#54995
    odoo/enterprise#12080
    odoo/enterprise@8290bfaf
2020-09-15 13:58:56 +02:00
Christophe Monniez
b63cc4510c [FIX] runbot: allow to change build host from interface 2020-09-14 14:47:37 +02:00
Xavier-Do
e1e965778b [FIX] runbot: keep host for child when host is assigned. 2020-09-14 11:29:21 +02:00
Xavier-Do
0a766d0a1a [FIX] runbot: simplest base example without pr 2020-09-13 13:25:32 +02:00
Xavier-Do
675e2f1688 [IMP] runbot: manual host 2020-09-11 15:12:36 +02:00
Xavier-Do
45721cdf6c [IMP] runbot: runbot 5.0
Runbot initial architechture was working for a single odoo repo, and was
adapted to build enterprise. Addition of upgrade repo and test began
to make result less intuitive revealing more weakness of the system.

Adding to the oddities of duplicate detection and branch matching,
there was some room for improvement in the runbot models.

This (small) commit introduce the runbot v5.0, designed for a closer
match of odoo's development flows, and hopefully improving devs
experience and making runbot configuration more flexible.

**Remotes:** remote intoduction helps to detect duplicate between odoo and
odoo-dev repos: a commit is now on a repo, a repo having multiple remote.
If a hash is in odoo-dev, we consider that it is the same in odoo.
Note: github seems to manage commit kind of the same way. It is possible
to send a status on a commit on odoo when the commit only exists in
odoo-dev.
This change also allows to remove some repo duplicate configuration
between a repo and his dev corresponding repo.
(modules, server files, manifests, ...)

**Trigger:** before v5.0, only one build per repo was created, making it
difficult to tweak what test to execute in what case. The example use
case was for upgrade. We want to test upgrade to master when pushing on
odoo. But we also want to test upgrade the same way when pushing on
upgrade. We introduce a build that should be ran on pushing on either
repo when each repo already have specific tests.
The trigger allows to specify a build to create with a specific config.
The trigger is executed when any repo of the trigger repo is pushed.
The trigger can define depedencies: only build enterprise when pushing
enterprise, but enterprise needs odoo. Test upgrade to master when pushing
either odoo or upgrade.
Trigger will also allows to extract some build like cla that where
executed on both enterprise and odoo, and hidden in a subbuild.

**Bundle:** Cross repo branches/pr branches matching was hidden in build
creation and can be confusing. A build can be detected as a duplicate
of a pr, but not always if naming is wrong or traget is invalid/changes.
This was mainly because of how a community ref will be found. This was
making ci on pr undeterministic if duplicate matching fails. This was
also creating two build, with one pointing to the other when duplicate
detection was working, but the visual result can be confusing.
Associtaions of remotes and bundles fix this by adding all pr and
related branches from all repo in a bundle. First of all this helps to
visualise what the runbot consider has branch matching and that should
be considered as part of the same task, giving a place where to warn
devs of some possible inconsistencies. Associate whith repo/remote, we
can consider branches in the same repo in a bundle as expected to have
the same head. Only one build is created since trigger considers repo,
not remotes.

**Batch:** A batch is a group of build, a batch on a bundle can be
compared to a build on a branch in previous version. When a branch
is pushed, the corresponding bundle creates a new batch, and wait for
new commit. Once no new update are detected in the batch for 60 seconds,
All the trigger are executed if elligible. The created build are added
to the batch in a batch_slot. It is also possible that an corresponding
build exists (duplicate) and is added to the slot instead of creating a
new build.

Co-authored-by d-fence <moc@odoo.com>
2020-09-10 13:44:38 +02:00
Xavier Morel
0b1b36d10c [FIX] runbot: mis-migrated test
73f720a55c refactored the runbot tests,
and amongst other things created a single patch point for the "mock
root" as a testcase attribute.

One of the tests was missed during that refactoring, likely because
it's skipped by default.
2020-07-08 08:15:45 +02:00
Christophe Monniez
af00bf16e7 [IMP] runbot: add ocrmypdf package to Dockerfile
The ocrmypdf suite of tools are needed by task #2238654.

Altough this package will be optional for Odoo users, it has to be
usable by dev's in order to test and/or fix the feature.
2020-06-19 15:26:31 +02:00
Christophe Monniez
9684c2d97e [FIX] runbot: avoid unallowed characters in docker names
Docker container names are derived from the dest and step name. The dest
is itself derived from the branch name.

In some rare cases, it happens that a character not allowed by Docker
appears in the container name computed by the runbot.

With this commit, a sanitize_container_name function is used to remove
unallowed characters at the container utility level.
2020-06-17 10:15:37 +02:00
Christophe Monniez
5ad74635e0 [IMP] runbot: install postgresql-client 12 in Dockerfile
The postgresql-client in the Dockerfile is the one provided by the
Debian package. When the postgresql server on the host has a higher
version than the client, some builds may fail (for example, dumping a
database with the pg_dump).

With this commit, the postgresql-client 12 from the postgresql repo is
used in the Dockerfile.
2020-05-28 16:30:38 +02:00
Xavier-Do
3bf9b263f0 [FIX] runbot: send status only once.
Runbot can send status multiple time for the same hash:
- if transaction fails in scheduler and is retried
- if multiple subbuild are failing

Leading to multiple issues:
- when github receive more than one failure status, mergebot will
be notified multiple times and send multiple mail (for forward ports mainly)
- github will answer `422 Unprocessable Entity for url...` after
1000 status.

This fix proposes to limit number of status:
- By avoiding to send status for orphan build (parent status will never change)
- By storing last send status to avoid to notify multiple time
- By sending status post commit to avoid to contact  github in case of failure.
This will also slightly reduce transaction time by removing an http request.
2020-04-24 11:39:36 +02:00
Christophe Monniez
b517df4ff0 [IMP] runbot: try to fetch multiple times before disabling
Sometimes, it happens that a `git fetch` fails with an error code 128
for example. When this happens, the runbot host is immediately disabled.

During investigations of such cases, we found that simply retrying the
fetch command works.

With this commit, the fetch command is tried 5 times with an increasing
delay before deciding to disable the runbot host.
2020-04-24 11:35:01 +02:00
Christophe Monniez
4c4b7213bb [IMP] runbot: remove pull_head_name patch filter
Since we store the target_branch_name, filtering out pull head names
that contains `patch-` is not necessary anymore.

This commit is one first step towards a clean refactoring.
2020-04-03 14:53:21 +02:00
Christophe Monniez
59d4eeaf24 [FIX] runbot: use proxy-mode for running instances
When deployed behind nginx reverse proxy and https, the Odoo proxy-mode
must be enabled.
2020-03-30 14:35:15 +02:00
Christophe Monniez
ed8d194d7e [FIX] runbot: typo in stat sql view 2020-03-23 13:54:43 +01:00
Xavier-Do
55ed520823 [FIX] runbot: avoid useless make_stats logs 2020-03-20 11:24:15 +01:00
Christophe Monniez
360e31ade4 [IMP] runbot: add a build stat model
When a build is done, various numerical informations could be extracted
from log files.  e.g.: global query count or tests query count ...

The extraction regular expression could be hard-coded in a custom step
but there is no place holder where to store the retrieved information.
In order to compare results, we need to store it.

With this commit, a new model `runbot.build.stat` is used to store
key/values pair linked to a build/config_step.  That way, extracted
values can be stored.

Also, another `runbot.build.stat.regex` is used to store regular
expressions that can be used to grep log files and extract values.

The regular expression must contain a named group like this:
`(?P<value>.+)`
The text catched by this group MUST be castable into a float.

Optionally, another named group can be used in the regular expresion
like this:
`(?P<key>.+)`
This `key` group will then be used to augment the key name in the
database.

Example:
    Consider a log line like this one:
    `odoo.addons.website_blog.tests.test_ui tested in 10.35s`

    A regular expression like this, named `test_duration`:
    `odoo.addons.(?P<key>.+) tested in (?P<value>\d+\.\d+)s`

    Should store the following key:value:
    `{
        'key': 'test_duration.website_blog.tests.test_ui',
        'value': 10.35
    }`

A `generic` boolean field is present on the build.stat.regex object,
meaning that when no regex are linked to a make_stats config step, then,
all the generic regex will be applied.

A wizard is added to help the creation the regular expressions, allowing
to test if they work against a user provided example.

A _make_stats method is added to the ConfigStep model which is called
during the _schedule of a build, just before calling the next step.
The regex search is only apllied in steps that have the `make_stats`
boolean field set to true. Also, the build branch have to be flagged
`make_stats` too or the build can have a key `make_stats` in its
config_data field.

The `make_stats` field on the branch is a compute stored field.
That way, sticky branches are automaticaly set `make_stats' true.

Finally, an SQL view is used to facilitate the stats visualisation.
2020-03-20 11:11:03 +01:00
Christophe Monniez
03d7f871be [FIX] runbot: silently ignore template database 2020-03-18 11:42:57 +01:00
Christophe Monniez
e30d2fbeb1 [FIX] runbot: show connect links when build is a duplicate
When a build is a duplicate, the connections links are not visible
because the local_state is "duplicate".
2020-03-18 11:41:42 +01:00
Martin Trigaux
60a4a9de85 [FIX] runbot: displayed collapsed error
By default, the error was displayed with arrow pointing UP, not down
2020-03-16 15:32:13 +01:00
Christophe Monniez
3918d266fa [IMP] runbot: add coverage xml export
When coverage is computed, a post command is used to generate the HTML
report. In order to use the coverage result locally the HTML report is
not enough.

With this commit, an XML report is also generated. It's a single xml
file, downloadable from the build result web page.

The _post_install_command method is renamed into its plural form because
it was useless to return only one command.
2020-03-13 13:40:46 +01:00
Christophe Monniez
820adae5ac [IMP] runbot: auto disable host when git fetch fail
From times to times, a git repo gets corrupted, making builds fail in
chain.

With this commit, the host on which the git fetch fails will be reserved
if not more than half the hosts are reserved.
2020-03-13 13:38:42 +01:00
Xavier-Do
6b52687ed1 [FIX] runbot: better log list
Before intensive python steps, every build exept create build should have logs.
This is not true since now many python steps are creating build, leading
to 404 links to inexisting logs file.

Checking That a step is a docker build looks like a good heuristic since since
most of the time the log file will be the one created by docker. In all case we
can access other logs files in browsing /logs.

A future improvement would be to listdir logs to create all links.
2020-03-13 11:46:50 +01:00
Xavier Morel
5bba39e153 [FIX] runbot: run with unaccent enabled 2020-03-11 13:02:48 +01:00
Christophe Monniez
1fca89e3fd [FIX] runbot: fix invalid type argument when calling _log method 2020-03-10 17:07:47 +01:00
Christophe Monniez
3b00d2576c [IMP] runbot: allow pseudo markdown in log messages
With this commit, a kind of markdown is allowed in log messages and
build.description.
Following patterns are supported:
  **strong**
  ~~striketrough~~
  __underline__
`code`
 [link](target)
 @icon-font-awesome-class
2020-03-09 13:10:49 +01:00
Christophe Monniez
3a428d4877 [IMP] runbot: copy steps when duplicating a config
When a build.config is copied, the config steps are not copied.

The steps have to be explicitly ordered otherwise it leads to a
traceback when trying to copy a 'run' step which  have to be the last
one.
2020-03-09 11:37:54 +01:00
Christophe Monniez
31e10a059e [IMP] runbot: bump Google Chrome version to 80
The fix odoo/odoo#45347 was merged two weeks ago, it's now time to
deploy on runbot.
2020-03-09 11:34:56 +01:00
Xavier-Do
2379318b0c [IMP] runbot: add subcommand support 2020-02-28 15:54:44 +01:00
Xavier-Do
7e49b4cd2b [IMP] runbot: add a public route for monitoring 2020-02-24 14:45:23 +01:00
Xavier-Do
6b88cb7688 [IMP] runbot: custom db_names for install and run jobs.
When creating a subbuild from a custom python job/cron, some data can
be given through config_data or extra_params to change another step behaviour.

An example is to give a specific -i to an install job in order to
install different modules from the parent build. Anyway since we
are using the same install step in this case, all databases will
have the same name, and the name may not be correct in regard to
the database content.
This commit allows to give a config_param on build giving the default
db_name to use in an install step.
2020-02-24 13:52:57 +01:00
Xavier-Do
ef7029668a [FIX] runbot: take parent into account when searching last branch build
If a branch triggers an hidden build on a another one (sticky ususally),
the last build of the branch will be considered to be this one and
trigger a new build. The same problem can occur when cloning a subbuild
to slighlty change a parameter instead of making an exact rebuild
since the build type may still be normal in this case.

This commit will simply ignore subbuild in this case.
2020-02-20 15:40:34 +01:00
Christophe Monniez
d9ff43fa6b [REM] runbot: remove reverse dependency build feature
When a new commit is found, a rebuild is forced on sticky branches
builds in repositories that depends on the new commit repository.

If one of the repo is protected by groups, the main page gives access
errors for public users and finally leads to a CPU overload.

With this commit, the feature is removed as it will be re-implemented in
custom config steps.
2020-02-20 10:58:20 +01:00
Christophe Monniez
931e2bef8f [FIX] runbot: avoid dependency build from an hidden build 2020-02-19 17:28:38 +01:00
Christophe Monniez
26dce90dd8 [FIX] runbot: avoid ask kill on already dying builds 2020-02-19 17:28:38 +01:00
Christophe Monniez
3d428428eb [IMP] runbot: replace fixing_commit by test_tags errors tree 2020-02-17 16:38:22 +01:00
Christophe Monniez
baea2e73be [IMP] runbot: kill build only when needed
On non sticky branches, when a new build is found while another is
already testing, the older build is killed. This happens during when the
main runbot instance is discovering new commits and create new builds.
As a result, concurrent updates may occur while the builders access the
concerned build.

With this commit, this garbage collecting procedure occurs during the
scheduler loop that runs on runbot builder hosts.

Also, the logic changed in a way that the kill is requested only if the
host needs room to handle pending builds.
2020-02-17 16:38:22 +01:00
Christophe Monniez
149ae4a074 [IMP] runbot: improve local cleanup
When a build age reaches the gc_days parameter, its database is dropped
and its directory is removed.

With this commit, two fields are added in order to keep some builds
longer that the defined gc_days.

The gc_delay field on the build allows to add a delay (in number of
days) that is added to its gc_days to compute the gc_date.

The gc_date field is the date when the cleaning will occur.

Also, a test is added and the RunbotCase test class is improved to allow
the stop of a patcher.
2020-02-17 16:38:22 +01:00
Christophe Monniez
5d4979a5f6 [IMP] runbot: set google-chrome version precisely
With this commit, it will be possible to play with different chrome
versions without impacting all runbot instances.

Chrome issues summary:

* 71: innerText linefeed issue
* 74: random chrome crash
* 80: V8 pointer compression exceed Odoo memory limits
2020-02-17 16:37:02 +01:00
Xavier-Do
b4a25c54e4 [IMP] runbot: better same branch build filter 2020-02-12 11:41:08 +01:00
Christophe Monniez
3e7d98a6b8 [FIX] runbot: use a custom template to create db
Since cb05f2b9d8, when creating a database, the template1 is used, this
allows to customize the template and install some needed Postgress
extensions.

Unfortunately, it's also a source of build failures. For example, if the
pg_activity util is used on a runbot host, database creation may fail
with a message like this:

source database "template1" is being accessed by other users

It's because pg_activity needs a database and uses template1.

With this commit, template1 is still used by default but can be changed
with a system parameter. That way, a custom template can be created on
runbot hosts and used when creating DB in builds.
2020-02-12 10:38:54 +01:00
Xavier-Do
c816ba2161 [IMP] runbot: add latest builds of same branch 2020-02-11 17:15:20 +01:00
Christophe Monniez
464893d248 [IMP] runbot: handle PR re-targeting
When a PR branch target is changed on Github, the change is not applied
in the runbot DB.

With this commit, the Github hook payload is taken into account to
detect such a change and the branch infos are recomputed accordingly.

Also, a button is now available on the branch form in order to manually
recompute those changes.
2020-02-11 17:05:33 +01:00
Christophe Monniez
46c8c6d635 [FIX] runbot: more flexible duplicate detection 2020-02-10 14:13:38 +01:00
Xavier-Do
6909b7e583 [IMP] runbot: allow build copy 2020-02-05 15:59:57 +01:00
Xavier-Do
c561f212e2 [IMP] runbot: improve build form view 2020-02-05 15:59:57 +01:00
Xavier-Do
a7ecf6a631 [FIX] runbot: fix build creation from view form
Before this commit, creating branch/build from view form may lead to errors
because of missing fields/ inexisting id.
2020-02-05 15:59:57 +01:00
Xavier-Do
3345d18050 [IMP] runbot: split prending count and scheduled count 2020-02-04 14:35:07 +01:00
Christophe Monniez
e267533513 [IMP] runbot: add pdfminer to Docker image
As pdfminer is going to be used in Odoo attachement_indexation module and
that there is no python3-pdfminer package in Ubuntu Bionic, it was
decided to have an optional dependency.

In order to run the attachement_indexation tests, the pip package is
added to the runbot Docker image.
2020-01-30 14:52:29 +01:00
Xavier-Do
e65859c161 [FIX] runbot: catch ValueError instead of RunbotException in safe_eval 2020-01-24 13:09:36 +01:00
Christophe Monniez
edda6c0265 [IMP] runbot: build docker image once per loop turn
At this moment, the Docker image is built at the beginning of each
runbot build. This blocks the _scheduler while the image is built.

With this commit, the image is built before calling the _scheduler and
is not linked to a runbot build.

Also, the necessary dirs are created in the static path before starting
the loop.
2020-01-22 13:50:01 +01:00
Christophe Monniez
61101eaf46 [FIX] runbot: fix separtor class in build template 2020-01-22 13:32:27 +01:00
Christophe Monniez
e4e2634de1 [FIX] runbot: keep tests directory
Since 857821e4 a screenshot can be saved in the build directory under
the "tests" subdirectory.
Unfortunately, this directory is cleaned in case of local_cleanup.

With this commit, the 'tests' subdirectory is kept in place.
2020-01-22 13:32:27 +01:00
Xavier Morel
5d1772a9a5 [FIX] runbot: redirect to the relevant build on actions
Currently killing, waking or rebuilding a build then redirects to the
repository root which is worse than useless: you've lost the build you
come from, and for rebuilds it's no help getting to the new build.

Since it's very easy to go from a build to its repository, redirect to
the "most useful" build instead:

* if rebuilding the current build (from its page), open the page of
  the new build which was just created
* otherwise reload the current page whatever it is
2020-01-21 09:21:02 +01:00
Xavier-Do
4ae6907e3a [FIX] runbot: keep description on exact rebuild 2020-01-20 14:44:16 +01:00
Christophe Monniez
ca4768230d [FIX] runbot: avoid crash when creating migration symlink 2020-01-17 17:14:12 +01:00
Christophe Monniez
bf9ec0b7db [FIX] runbot: add file command to build Odoo 8.0 and 9.0 2020-01-17 17:10:25 +01:00
Xavier-Do
eede4dcd77 [IMP] runbot: filter hidden on monitoring page 2020-01-17 16:36:30 +01:00
Xavier-Do
3f997cff25 [IMP]: runbot: improve create performances. 2020-01-17 14:41:18 +01:00
Xavier-Do
9556723617 [FIX] runbot: fix exact rebuild without config_data 2020-01-17 14:41:18 +01:00
Xavier-Do
7058868c0f [IMP] runbot: select for update in ask_kill
When a build has a lot of children, _ask_kill can fail due to concurrent_update.
This solution locks all build before reading/updating builds.
2020-01-17 14:41:18 +01:00
Xavier-Do
534bb728ce [IMP] runbot: add parent_path 2020-01-17 14:41:18 +01:00
Xavier-Do
a2256fffb8 [FIX] runbot: don't detect duplicate on rebuild. 2020-01-17 14:41:18 +01:00
Christophe Monniez
cb05f2b9d8 [FIX] runbot: use template1 when creating databases 2020-01-17 13:53:52 +01:00
Xavier-Do
2274f7ab8b [FIX] runbot: handle externally killed docker.
If a docker is killed from outside, the start file will still be there but
not the end file. (when restarting docker service for instance)
This commits add a docker_state specific to non running docker
with start a file, that should be handled like unknow state:
This state is acceptable for a while, but build should be killed
if this state remains for to long.
2020-01-16 14:47:40 +01:00