Commit Graph

775 Commits

Author SHA1 Message Date
Christophe Monniez
7adfa6c7eb [FIX] runbot: add an index on local state
A slow query was detected on the runbot, causing a latency when loading
any page. After some investigations (thnks jle), we found that an index
on local_state could improve speed.
2019-06-19 15:37:31 +02:00
Christophe Monniez
c13128a229 [FIX] runbot: try to update github status multiple times
When updating github statuses, it happens that we face a "Bad gateway"
from github. In that case, the error is logged in the runbot logs and
that's it. As a consequence, when the runbot_merge is waiting status for
the staging branch and this kind of error occurs, the runbot_merge
timeouts and the users vainly search the reason.

With this commit, the runbot tries to update the status at least twice.
If it fails, an INFO message is logged on the build itself.
2019-06-19 10:20:38 +02:00
Xavier-Do
2fb0f2c79a [FIX] runbot: kill children even if state is done 2019-06-18 13:33:45 +02:00
Christophe Monniez
ae1eed46f5 [IMP] runbot: add optional warning message on frontend 2019-06-18 12:43:25 +02:00
Xavier-Do
080d5a4e1b [IMP] runbot: remove testing count from repo view 2019-06-18 12:42:07 +02:00
Xavier-Do
6d36455ef1 [FIX] runbot: ensure that job and build end are always set. 2019-06-18 12:42:07 +02:00
Xavier-Do
af7a8b7d8b [IMP] runbot: kill subbuild when killing build 2019-06-18 12:42:07 +02:00
Xavier-Do
a246d60c71 [IMP] runbot: always create new build at rebuild 2019-06-18 12:42:07 +02:00
Xavier-Do
17c57d499e [IMP] runbot: exact rebuild
allow to rebuild using exact config, dependencies, ...
2019-06-18 12:42:07 +02:00
Xavier-Do
a3ff9d102d [IMP] runbot: small runbot_branch view improvements 2019-06-18 12:42:07 +02:00
Xavier-Do
7e9826ba4c [IMP] runbot: add build creation log 2019-06-18 12:42:07 +02:00
Xavier-Do
f691ab5a39 [IMP] add hidden in view form 2019-06-18 12:42:07 +02:00
Xavier-Do
55c4e49ada [FIX] runbot: fix time ralated failing test
Some test where failing since _find_new_commits will skip
commit older than max_age
2019-06-18 12:42:07 +02:00
Xavier-Do
bdc6506ede [IMP] runbot: always keep one empty slot when assigning sheduled
Nightly build have a low priority but once they have a slot, they keep it.
When pushing a branch or asking robodoo to nicelly merge a branch for the
third time at 22:00, there may be no slot left since all the nightly build are created.

This commit will only assign scheduled build if there is no other build to create and
will always keep a free slot for other builds.
2019-06-18 12:42:07 +02:00
Xavier-Do
86ffeac589 [FIX] runbot: test_enable should be True by default 2019-06-18 12:42:07 +02:00
Christophe Monniez
2715585e31 [IMP] runbot: add python3-markdown in Dockerfile
Needed for running migration scripts.
2019-06-18 11:50:48 +02:00
Christophe Monniez
b98ff23cd9 [IMP] runbot: add write_file and make_dirs methods
When using a "python job", it's sometimes useful to write a file or
create a directory.
Instead of giving a wide open access to the os module in the
"_run_python" context, this commit adds a write_file and make_dirs
methods on the build which is usable in the _run_python eval context.
2019-06-18 11:50:48 +02:00
Christophe Monniez
800691a14c [IMP] runbot: add a mode parameter to readfile
When using the read_method in a "python job", it's sometimes needed to
read a file in binary mode.

With this commit, the mode can be specified when calling the method.
2019-06-18 11:50:48 +02:00
Christophe Monniez
799f3aed22 [IMP] runbot: add more objects in run python
Since the use of the "python jobs", we spotted various needs that were
not fulfilled. In order to add flexibility to "python jobs", this commit
adds some useful objetcs in the _run_python eval context.

Also, the glob.glob function is given instead of the whole glob module
to avoid giving access to the os module via glob.os.
2019-06-18 11:50:48 +02:00
Xavier-Do
c5588b1861 [IMP] runbot: display config name 2019-06-18 11:31:14 +02:00
Christophe Monniez
a9cd5d7a60 [IMP] runbot: allow to configure workers per host
When a runbot instance is scheduling builds, the numbers of builds
depends of a global ir.config_parameter. Even if one of the runbot
instance is running on a more powerful systsem, its number of workers is
limited by this global parameter.

With this commit, this parameter still exists but can be overriden by
specific ir.config_parameter.

For example, if the host 'runbot24.odoo.com' has more cpu power, the
number of workers for this host can be specified in the
ir.config_parameter named 'runbot24.odoo.com.workers'.
2019-06-17 12:44:14 +02:00
Christophe Simonis
483fc5319a [IMP] runbot_merge: show failed PR on dashboard
Closes #138
2019-06-11 11:31:24 +02:00
Christophe Monniez
b1add16fbd [FIX] runbot: add missing apt update in Dockerfile
When installing software with apt-get in a Dockerfile, it should be
preceded with an apt-get update in the same RUN. Otherwise, the step may
fail if the needed package has been updated.
2019-06-07 10:15:24 +02:00
Christophe Monniez
82dbf43083 [IMP] runbot: add es-check tool to the Docker image
In order to use the new test odoo/odoo#33724 the es-check tool is needed
in the Docker runbot image.
2019-06-04 07:37:51 +02:00
Xavier-Do
f6c220d3b6 [IMP] runbot_cla: add cla_check to All only config
Mergebot is using All only config, but need cla check.
This commit will avoid to remove this step when updating runbot.
2019-05-23 13:04:56 +02:00
Christophe Monniez
c49e4d21ad [IMP] runbot: allow a parent to ignore some child builds
In a create config, a parent result is computed based on children
results

In some situations, it could be handy to ignore the result of some
sub-builds.

Example: the nightly tests are just the children of one nightly build
with a create config. The external tests are failing randomly and as a
consequence, the nightly result is always red. On the other hand,
keeping the test running, just to have logs is a good idea.

With this commit, a config_step of type create can be marked as
orphan_result, that way, the result is not taken into account in the
parent build result.
2019-05-23 12:32:35 +02:00
Christophe Monniez
c24df5fac2 [FIX] runbot: choose quickconnect build with same config
When the quickconnect button is used, the last running build is
searched in the last 10 builds. If no running build is found, the last
one is rebuilt, even if it's a nightly build.

With this commit, the quickconnect build is choosen only among the ones
with the same config.
2019-05-23 10:52:13 +02:00
Xavier-Do
0384f2c98b [FIX] runbot: correct indentation 2019-05-22 14:08:45 +02:00
Christophe Monniez
b31735406d [FIX] runbot: adapt build write method to api multi
And add tests to avoid regression.
2019-05-22 10:58:30 +02:00
Xavier-Do
534368b675 [FIX] runbot: minimize transactionnal errors
With recursive states computation, schedule is
most likely to have transactionnal errors.
This is particularly a problem when external
operations are done during the transaction,
like running a docker.

Adding some commits will help to reduce
transactionnal errors, and ensure that the db
is consistent with docker states.
2019-05-22 10:15:53 +02:00
Christophe Monniez
45516f9d33 [FIX] runbot: properly call _log when time exceeds 2019-05-22 09:25:56 +02:00
Christophe Monniez
5761fd1694 [FIX] runbot: properly omit manifest files in coverage 2019-05-21 23:21:45 +02:00
Xavier-Do
519f50790d [FIX] runbot: make python jobs usable and improve logs 2019-05-21 16:03:47 +02:00
Christophe Monniez
d3e3f75224 [FIX] runbot: only logged in users are allowed to kill
As the public user needs to be in runbot user group to display the
frontend, the public user is allowed to kill or rebuild a build.

With this commit, only the logged in users have access to the Rebuil/Kill
menu entry.
2019-05-21 09:42:46 +02:00
Christophe Monniez
d55a71d675 [FIX] runbot: oversight of a repo arg not needed anymore 2019-05-20 11:23:27 +02:00
Xavier-Do
427639b77a [FIX] runbot: various fixes (data improvement, python step, logs, ...) 2019-05-20 10:52:13 +02:00
Xavier-Do
8a7e0b20aa [FIX] runbot: fix config access rights 2019-05-18 15:13:21 +02:00
Xavier-Do
98913f6d39 [FIX] runbot: fix cla step 2019-05-18 14:20:05 +02:00
Xavier-Do
fce51d6dbe [FIX] runbot: correctly store get_ref_time 2019-05-18 11:16:08 +02:00
Christophe Monniez
16e1eaa30b [IMP] runbot: decrease for-each-ref calls
When searchings for new refs to create builds, the for-each-ref git
commit is run and each ref is searched in the database which is a
somewhat heavy operation.

With this commit, the timestamp of the last database update with the
refs is stored in a field on the repo. This timestamp is checked each
time a for-each-ref is needed, running the operation only when
necessary.
2019-05-18 10:42:31 +02:00
Christophe Monniez
3aabfd4bd3 [IMP] runbot: increase version and add migration scripts
Migration scripts from runbot 3.1 to runbot 4.0.
2019-05-18 10:42:31 +02:00
Xavier-Do
8ef6bcfde7 [IMP] runbot: replace jobs by build configs
This commit aims to replace static jobs by fully configurable build config.

Each build has a config (custom or inherited from repo or branch).
Each config has a list of steps.
For now, a step can test/run odoo or create a new child build. A python job is
also available.

The mimic the previous behaviour of runbot, a default config is available with
three steps, an install of base, an install+test of all modules, and a last step
for run.

Multibuilds are replace by a config containing cretaion steps.
The created builds are not displayed in main views, but are available
on parent build log page. The result of a parent takes the result of
all children into account.

This new mechanics will help to create some custom behaviours for specifics
use cases, and latter help to parallelise work.
2019-05-18 10:42:31 +02:00
Christophe Monniez
bb35b1cc9d [FIX] runbot: remove typo that prevent kill when timeout 2019-05-14 11:54:59 +02:00
Christophe Monniez
cff0133e1a [FIX] runbot: convert runbot_timeout to int
The cpu limit used in job_20 uses the runbot_timeout config_parameter
since b539112a7e. When measuring coverage, this parameter is multiplied
and leads to an error because the type of ir.config_parameter.get_param
method is str.

With this commit, the this value is converted into integer before usage
in job_20.
2019-05-12 19:29:07 +02:00
Christophe Monniez
12acfda83a [FIX] runbot: add dbuser parameter to the odoo cmd
When running Odoo in the Docker container, the username used to connect
to the database is the username defined in the docker container
(actually odoo).

A problem may arise if the user of the runbot process is not the same.
An authentication error is then raised by postgres because of the
username mismatch.

With this commit, the '-r' parameter of Odoo is added to the command
with the username used by the runbot process.

While at it, unused imports are removed.
2019-05-10 11:42:18 +02:00
Christophe Monniez
b539112a7e [FIX] runbot: make the cpu limit the same as the runbot timeout
When a build exceeds the cpu limit, it is simply killed by the kernel.
As a safeguard the "Initiating shutdown." sentence should be searched
in the log file, and the build marked as "ko" if not found.

Unfortunateley, there is no period (.) at the end of the sentence in the
Odoo logs (see: https://github.com/odoo/odoo/blob/12.0/odoo/service/server.py#L444)
Thus, this condition is never fulfilled.
On top of that, this was masked by the first part of the condition,
checking that the 'test/common.py' has no "post_install" string.
The "test" directory does not exists in Odoo ( but "tests" exists) , so
the condition was always falsy.

Finally, a build can be marked as "ok" when he is killed and no errors
are found until the kill.

With this commit:
    * The legacy grep for post_install is removed as it now exists in
      all Odoo supported versions.
    * The period typo is fixed.
    * A log is inserted when the final sentence is not found.
    * The cpu_limit is set as the same as the runbot_timeout parameter
      for better consitency.
    * The time exceeded log message is now logged in the build instead
      of the runbot log.

Co-authored-by: @Xavier-Do
2019-05-10 10:56:55 +02:00
Christophe Simonis
f3ae3cac5a [IMP] runbot: sort sticky branches chronologically
On a par with `/glances` page.
2019-05-10 10:08:31 +02:00
Xavier Morel
76ea2a4f5d [FIX] runbot_merge: ensure co-authored-by are at the bottom
Sometimes people add co-authored-by lines in the middle of their
message, where github ignores them.

Since we previously added properly handling existing (correct) C-A-B
lines in the case where we're adding fixes and signed-off-by, we might
as well fix-up existing but mispalced co-authored-by lines.

Fixes #107
2019-05-07 13:22:13 +02:00
Xavier Morel
c9f9b3050b [FIX] runbot_merge: possibly missing commit object in try_splitting
Would trigger a TypeError when trying to json.loads(False).
2019-05-07 12:53:55 +02:00
Xavier Morel
eb91e2371b [FIX] runbot_merge: check CI status when reopening a PR
Previously, creating a PR would validate the head (in case it had
already passed CI) but reopening it would not, which is inconvenient
as the CI would not automatically run on a reopened PR.

Update both the state and the head of the PR on reopen to force a
revalidation, that way if the head has already passed CI the PR will
be reopened validated and there won't be an unclear need to perform an
explicit CI run.

Fixes #119
2019-05-07 12:32:02 +02:00