Commit Graph

1033 Commits

Author SHA1 Message Date
Xavier-Do
47396ddb46 [IMP] runbot: only drop database and clean localfiles after 30 days. 2019-06-25 16:43:47 +02:00
Xavier-Do
5c72655e18 [FIX] runbot: invalidate cache at each commit.
In some rare case db could become inconsistent if some data
in cache were used between two _schedule loop turn.
2019-06-25 16:15:24 +02:00
Xavier-Do
94a680f0c8 [FIX] runbot: get_ref_time as float
get_ref_time was cast from float to datetime and thus,
milliseconds where lost. Storing it as float make code
easier to read and avoid this rounding that was breaking
 this feature.
2019-06-24 15:59:21 +02:00
Xavier-Do
b64c7adce2 [FIX] runbot: make nb_testing count correct when children are duplicates 2019-06-24 15:59:21 +02:00
Christophe Monniez
e51412d558 [IMP] runbot: only consider refs newer than max_age
When getting new refs, a lot of them are really old and the
find_new_commits is called for each one and thus browsing branches.

With this commit, refs older than configured max_age are ignored.

Co-authored-by: Xavier Dollé (xdo@odoo.com)
2019-06-24 14:33:17 +02:00
Xavier-Do
54f9b9b546 [IMP] runbot: use counts for global states 2019-06-24 14:29:52 +02:00
Xavier-Do
03da48a07a [IMP] runbot: compute nb states without accessing childrens
Accessing childrens can create rollback, especially for builds with
a lot of them, since other runbot will concurently access and update
states. This commit tries to improve this by incrementing and
decrementing counters instead of counting all of them each time.
2019-06-24 14:29:52 +02:00
Xavier-Do
239340e1bc [IMP] runbot: add test for children 2019-06-24 14:29:52 +02:00
Christophe Monniez
7adfa6c7eb [FIX] runbot: add an index on local state
A slow query was detected on the runbot, causing a latency when loading
any page. After some investigations (thnks jle), we found that an index
on local_state could improve speed.
2019-06-19 15:37:31 +02:00
Christophe Monniez
c13128a229 [FIX] runbot: try to update github status multiple times
When updating github statuses, it happens that we face a "Bad gateway"
from github. In that case, the error is logged in the runbot logs and
that's it. As a consequence, when the runbot_merge is waiting status for
the staging branch and this kind of error occurs, the runbot_merge
timeouts and the users vainly search the reason.

With this commit, the runbot tries to update the status at least twice.
If it fails, an INFO message is logged on the build itself.
2019-06-19 10:20:38 +02:00
Xavier-Do
2fb0f2c79a [FIX] runbot: kill children even if state is done 2019-06-18 13:33:45 +02:00
Christophe Monniez
ae1eed46f5 [IMP] runbot: add optional warning message on frontend 2019-06-18 12:43:25 +02:00
Xavier-Do
080d5a4e1b [IMP] runbot: remove testing count from repo view 2019-06-18 12:42:07 +02:00
Xavier-Do
6d36455ef1 [FIX] runbot: ensure that job and build end are always set. 2019-06-18 12:42:07 +02:00
Xavier-Do
af7a8b7d8b [IMP] runbot: kill subbuild when killing build 2019-06-18 12:42:07 +02:00
Xavier-Do
a246d60c71 [IMP] runbot: always create new build at rebuild 2019-06-18 12:42:07 +02:00
Xavier-Do
17c57d499e [IMP] runbot: exact rebuild
allow to rebuild using exact config, dependencies, ...
2019-06-18 12:42:07 +02:00
Xavier-Do
a3ff9d102d [IMP] runbot: small runbot_branch view improvements 2019-06-18 12:42:07 +02:00
Xavier-Do
7e9826ba4c [IMP] runbot: add build creation log 2019-06-18 12:42:07 +02:00
Xavier-Do
f691ab5a39 [IMP] add hidden in view form 2019-06-18 12:42:07 +02:00
Xavier-Do
55c4e49ada [FIX] runbot: fix time ralated failing test
Some test where failing since _find_new_commits will skip
commit older than max_age
2019-06-18 12:42:07 +02:00
Xavier-Do
bdc6506ede [IMP] runbot: always keep one empty slot when assigning sheduled
Nightly build have a low priority but once they have a slot, they keep it.
When pushing a branch or asking robodoo to nicelly merge a branch for the
third time at 22:00, there may be no slot left since all the nightly build are created.

This commit will only assign scheduled build if there is no other build to create and
will always keep a free slot for other builds.
2019-06-18 12:42:07 +02:00
Xavier-Do
86ffeac589 [FIX] runbot: test_enable should be True by default 2019-06-18 12:42:07 +02:00
Christophe Monniez
2715585e31 [IMP] runbot: add python3-markdown in Dockerfile
Needed for running migration scripts.
2019-06-18 11:50:48 +02:00
Christophe Monniez
b98ff23cd9 [IMP] runbot: add write_file and make_dirs methods
When using a "python job", it's sometimes useful to write a file or
create a directory.
Instead of giving a wide open access to the os module in the
"_run_python" context, this commit adds a write_file and make_dirs
methods on the build which is usable in the _run_python eval context.
2019-06-18 11:50:48 +02:00
Christophe Monniez
800691a14c [IMP] runbot: add a mode parameter to readfile
When using the read_method in a "python job", it's sometimes needed to
read a file in binary mode.

With this commit, the mode can be specified when calling the method.
2019-06-18 11:50:48 +02:00
Christophe Monniez
799f3aed22 [IMP] runbot: add more objects in run python
Since the use of the "python jobs", we spotted various needs that were
not fulfilled. In order to add flexibility to "python jobs", this commit
adds some useful objetcs in the _run_python eval context.

Also, the glob.glob function is given instead of the whole glob module
to avoid giving access to the os module via glob.os.
2019-06-18 11:50:48 +02:00
Xavier-Do
c5588b1861 [IMP] runbot: display config name 2019-06-18 11:31:14 +02:00
Christophe Monniez
a9cd5d7a60 [IMP] runbot: allow to configure workers per host
When a runbot instance is scheduling builds, the numbers of builds
depends of a global ir.config_parameter. Even if one of the runbot
instance is running on a more powerful systsem, its number of workers is
limited by this global parameter.

With this commit, this parameter still exists but can be overriden by
specific ir.config_parameter.

For example, if the host 'runbot24.odoo.com' has more cpu power, the
number of workers for this host can be specified in the
ir.config_parameter named 'runbot24.odoo.com.workers'.
2019-06-17 12:44:14 +02:00
Christophe Simonis
483fc5319a [IMP] runbot_merge: show failed PR on dashboard
Closes #138
2019-06-11 11:31:24 +02:00
Christophe Monniez
b1add16fbd [FIX] runbot: add missing apt update in Dockerfile
When installing software with apt-get in a Dockerfile, it should be
preceded with an apt-get update in the same RUN. Otherwise, the step may
fail if the needed package has been updated.
2019-06-07 10:15:24 +02:00
Christophe Monniez
82dbf43083 [IMP] runbot: add es-check tool to the Docker image
In order to use the new test odoo/odoo#33724 the es-check tool is needed
in the Docker runbot image.
2019-06-04 07:37:51 +02:00
Xavier-Do
f6c220d3b6 [IMP] runbot_cla: add cla_check to All only config
Mergebot is using All only config, but need cla check.
This commit will avoid to remove this step when updating runbot.
2019-05-23 13:04:56 +02:00
Christophe Monniez
c49e4d21ad [IMP] runbot: allow a parent to ignore some child builds
In a create config, a parent result is computed based on children
results

In some situations, it could be handy to ignore the result of some
sub-builds.

Example: the nightly tests are just the children of one nightly build
with a create config. The external tests are failing randomly and as a
consequence, the nightly result is always red. On the other hand,
keeping the test running, just to have logs is a good idea.

With this commit, a config_step of type create can be marked as
orphan_result, that way, the result is not taken into account in the
parent build result.
2019-05-23 12:32:35 +02:00
Christophe Monniez
c24df5fac2 [FIX] runbot: choose quickconnect build with same config
When the quickconnect button is used, the last running build is
searched in the last 10 builds. If no running build is found, the last
one is rebuilt, even if it's a nightly build.

With this commit, the quickconnect build is choosen only among the ones
with the same config.
2019-05-23 10:52:13 +02:00
Xavier-Do
0384f2c98b [FIX] runbot: correct indentation 2019-05-22 14:08:45 +02:00
Christophe Monniez
b31735406d [FIX] runbot: adapt build write method to api multi
And add tests to avoid regression.
2019-05-22 10:58:30 +02:00
Xavier-Do
534368b675 [FIX] runbot: minimize transactionnal errors
With recursive states computation, schedule is
most likely to have transactionnal errors.
This is particularly a problem when external
operations are done during the transaction,
like running a docker.

Adding some commits will help to reduce
transactionnal errors, and ensure that the db
is consistent with docker states.
2019-05-22 10:15:53 +02:00
Christophe Monniez
45516f9d33 [FIX] runbot: properly call _log when time exceeds 2019-05-22 09:25:56 +02:00
Christophe Monniez
5761fd1694 [FIX] runbot: properly omit manifest files in coverage 2019-05-21 23:21:45 +02:00
Xavier-Do
519f50790d [FIX] runbot: make python jobs usable and improve logs 2019-05-21 16:03:47 +02:00
Christophe Monniez
d3e3f75224 [FIX] runbot: only logged in users are allowed to kill
As the public user needs to be in runbot user group to display the
frontend, the public user is allowed to kill or rebuild a build.

With this commit, only the logged in users have access to the Rebuil/Kill
menu entry.
2019-05-21 09:42:46 +02:00
Christophe Monniez
d55a71d675 [FIX] runbot: oversight of a repo arg not needed anymore 2019-05-20 11:23:27 +02:00
Xavier-Do
427639b77a [FIX] runbot: various fixes (data improvement, python step, logs, ...) 2019-05-20 10:52:13 +02:00
Xavier-Do
8a7e0b20aa [FIX] runbot: fix config access rights 2019-05-18 15:13:21 +02:00
Xavier-Do
98913f6d39 [FIX] runbot: fix cla step 2019-05-18 14:20:05 +02:00
Xavier-Do
fce51d6dbe [FIX] runbot: correctly store get_ref_time 2019-05-18 11:16:08 +02:00
Christophe Monniez
16e1eaa30b [IMP] runbot: decrease for-each-ref calls
When searchings for new refs to create builds, the for-each-ref git
commit is run and each ref is searched in the database which is a
somewhat heavy operation.

With this commit, the timestamp of the last database update with the
refs is stored in a field on the repo. This timestamp is checked each
time a for-each-ref is needed, running the operation only when
necessary.
2019-05-18 10:42:31 +02:00
Christophe Monniez
3aabfd4bd3 [IMP] runbot: increase version and add migration scripts
Migration scripts from runbot 3.1 to runbot 4.0.
2019-05-18 10:42:31 +02:00
Xavier-Do
8ef6bcfde7 [IMP] runbot: replace jobs by build configs
This commit aims to replace static jobs by fully configurable build config.

Each build has a config (custom or inherited from repo or branch).
Each config has a list of steps.
For now, a step can test/run odoo or create a new child build. A python job is
also available.

The mimic the previous behaviour of runbot, a default config is available with
three steps, an install of base, an install+test of all modules, and a last step
for run.

Multibuilds are replace by a config containing cretaion steps.
The created builds are not displayed in main views, but are available
on parent build log page. The result of a parent takes the result of
all children into account.

This new mechanics will help to create some custom behaviours for specifics
use cases, and latter help to parallelise work.
2019-05-18 10:42:31 +02:00