Commit Graph

1005 Commits

Author SHA1 Message Date
Xavier-Do
99fcd9638a [IMP] runbot: take care or extras
If a param is in extra, dont add it automaticaly.
This will allow to remove custom 'test from extras' step.
2019-07-17 15:51:18 +02:00
Xavier-Do
87c794a9aa [IMP] runbot: add wake up option on builds 2019-07-17 11:34:45 +02:00
Christophe Monniez
dbe44e2a76 [FIX] runbot: prevent inter Docker communication
When an Odoo instance is run in a Docker container, it listen on all
interfaces by default and a bridge interface is used to communicate with
the outside world.

This bridge interface is necessary to allow the instance to send logging
messages into the runbot postgresql database.

Even though Docker isolate the container from the oustide world, it's
still possible to reach the Odoo instace from the runbot host and worse,
from other Docker containers using the same bridge interface.

To confirm the  Murphy's law, it finally happened with this commit
odoo/enterprise@0ba0ef99de that scanned the ip range of the interface,
disturbing other builds.

One solution would be to create a bridge interface for each instance to
isolate each Docker but that would imply a big change to garbage collect
forgotten bridges ...

An 'icc' (Inter Container Communication) option exists for the Docker
daemon which defaults to True. Setting it to False was tested and works
but it appears that this option can be messed-up by the firewall on the
runbot host.

Finally, if applied, this commit will prevent the Odoo instance to
listen on the bridge interface by only listening to 127.0.0.1 during
tests. A local_only parameter is a added on the _cmd method which is
true by default and means that the Odoo instance will only listen on
127.0.0.1. This parameter is set to False for the running step to allow
the running to be contacted through the Docker exposed ports.
2019-07-17 10:52:14 +02:00
Xavier-Do
81fefee137 [FIX] runbot: fix coverage since shared sources
The requirements path and python version where defined from
server in cmd. Since in coverage we add a 'python' before server,
it is difficult to define which element of the cmd is the server.

A solution here is simply to define requirements install and
python version when building cmd since we have access to all
build/source informations. We also add python part in every
cases, and coverage params are now a _cmd python_params.

The _cmd method now returns a Command object instead of a
list, which behave has a list for the cmd part but also contains
a pres and posts list.

pres are requirement install, preparation, ...
cmd is the original cmd list, element can be append or added, this
will allow to keep existing python job without to much changes.
posts are post cmd commands, like coverage result making.

This commit also fix issue with create_job dependencies.
2019-07-16 12:06:09 +02:00
Xavier-Do
785001acac [FIX] runbot: fix _coverage_params 2019-07-12 16:37:39 +02:00
Xavier-Do
c31c9642be [IMP] runbot: add read_source on commit 2019-07-12 16:31:17 +02:00
Xavier-Do
21c3686543 [FIX] runbot: _log not log 2019-07-12 15:18:40 +02:00
Xavier-Do
f7a4fb7ac3 [IMP] runbot: share sources between builds
Multibuild can create generate a lots of checkout, especially for small
and fast jobs, which can overload runbot discs since we are trying not
to clean build immediatly. (To ease bug fix and allow wake up)

This commit proposes to store source on a single place, so that
docker can add them as ro volume in the build directory.
The checkout is also moved to the installs jobs, so that
builds containing only create builds steps won't checkout
the sources.

This change implies to use --addons-path correctly, since odoo
and enterprise addons wont be merged in the same repo anymore.
This will allow to test addons a dev will do, with a closer
command line.

This implies to change the code structure a litle, some changes
where made to remove no-so-usefull fields on build, and some
hard-coded logic (manifest_names and server_names) are now
stored on repo instead.

This changes implies that a build CANNOT write in his sources.
It shouldn't be the case, but it means that runbot cannot be
tested on runbot untill datas are written elsewhere than in static.

Other possibilities are possible, like bind mounting the sources
in the build directory instead of adding ro volumes in docker.
Unfortunately, this needs to give access to mount as sudo for
runbot user and changes docjker config to allow mounts
in volumes which is not the case by default. A plus of this
solution would be to be able to make an overlay mount.
2019-07-12 14:50:59 +02:00
Christophe Monniez
0830557cd6 [FIX] runbot: avoid "Modules loaded not found in logs"
In some conditions, Docker can take a little time to start a container.
In that case, if the runbot checks that the container is running before
it starts, runbot consider the job as finished. It the tries to grep the
logs and, as expected, it does not find the "Modules loaded".

With this commit, we consider young builds (less than 15 sec) as
running, giving more time to Docker for starting it.
2019-07-09 13:15:14 +02:00
Christophe Monniez
19b36e4ee6 [FIX] runbot: set force_rebuild when creating multi builds
When creating multi builds config steps, the force_build option is often
forgotten. In that case, the multi builds are detected as duplicate of
the first one.

With this commit, when asking for more that one multi build, the
force_build is chnaged to to True.
2019-07-05 18:03:24 +02:00
Xavier-Do
1b2940705d [FIX] runbot: fix ask_kill for children_ids 2019-07-04 12:17:22 +02:00
Xavier-Do
c4837641cc [FIX] runbot: fix result when make_result crash 2019-07-04 11:48:33 +02:00
Xavier-Do
2de144d299 [IMP] runbot: select all db strating by dest when cleaning 2019-07-04 10:25:25 +02:00
Xavier-Do
fe6788e349 [IMP] runbot: lock an host
In order to be able to test some custom code on one specific worker,
this commit will allow to lock a runbot so that he will only run
assigned build.
2019-07-03 16:23:48 +02:00
Xavier-Do
6bc666b5c3 [IMP] runbot: avoid loosing hooks
In the same spirit of getting garanties that community is updated when enterprise is,
force the hook time of dependencies in case it was lost.
2019-07-03 16:23:48 +02:00
Xavier-Do
f51a021dfe [IMP] runbot: update repo in reversed order
When updating repo, order is  odoo/odoo, odoo-dev/odoo, odoo/enterprise, odoo-dev/enterprise
The problem with this is that if a community hash and an enterprise hash arrives exactly
between odoo/odoo and odoo/enterprise update, enterprise could have a newest version
than community when creating builds.

By inversing this order, we have less chances to have this cornercase (as unlikelly as
it could be)
2019-07-03 16:23:48 +02:00
Xavier-Do
83a0568403 [FIX] runbot: write duplicate_id instead of duplicate when killing build 2019-07-03 16:22:08 +02:00
Xavier-Do
1ffa0468c1 [IMP] runbot: keep cron on transaction rollback 2019-07-03 12:58:58 +02:00
Xavier-Do
c5e0da174a [FIX] runbot: fix exact rebuild 2019-07-03 10:56:57 +02:00
Xavier-Do
53267d64bb [IMP] runbot: improve error logs 2019-07-03 10:50:47 +02:00
Christophe Simonis
5e611f54cb [IMP] runbot_merge: allow sorting & deactivating branches
Closes #144, closes #145
2019-06-28 11:31:06 +02:00
Xavier-Do
84ad4cefe0 [IMP] runbot: only display default config in glances 2019-06-27 12:43:24 +02:00
Xavier-Do
47396ddb46 [IMP] runbot: only drop database and clean localfiles after 30 days. 2019-06-25 16:43:47 +02:00
Xavier-Do
5c72655e18 [FIX] runbot: invalidate cache at each commit.
In some rare case db could become inconsistent if some data
in cache were used between two _schedule loop turn.
2019-06-25 16:15:24 +02:00
Xavier-Do
94a680f0c8 [FIX] runbot: get_ref_time as float
get_ref_time was cast from float to datetime and thus,
milliseconds where lost. Storing it as float make code
easier to read and avoid this rounding that was breaking
 this feature.
2019-06-24 15:59:21 +02:00
Xavier-Do
b64c7adce2 [FIX] runbot: make nb_testing count correct when children are duplicates 2019-06-24 15:59:21 +02:00
Christophe Monniez
e51412d558 [IMP] runbot: only consider refs newer than max_age
When getting new refs, a lot of them are really old and the
find_new_commits is called for each one and thus browsing branches.

With this commit, refs older than configured max_age are ignored.

Co-authored-by: Xavier Dollé (xdo@odoo.com)
2019-06-24 14:33:17 +02:00
Xavier-Do
54f9b9b546 [IMP] runbot: use counts for global states 2019-06-24 14:29:52 +02:00
Xavier-Do
03da48a07a [IMP] runbot: compute nb states without accessing childrens
Accessing childrens can create rollback, especially for builds with
a lot of them, since other runbot will concurently access and update
states. This commit tries to improve this by incrementing and
decrementing counters instead of counting all of them each time.
2019-06-24 14:29:52 +02:00
Xavier-Do
239340e1bc [IMP] runbot: add test for children 2019-06-24 14:29:52 +02:00
Christophe Monniez
7adfa6c7eb [FIX] runbot: add an index on local state
A slow query was detected on the runbot, causing a latency when loading
any page. After some investigations (thnks jle), we found that an index
on local_state could improve speed.
2019-06-19 15:37:31 +02:00
Christophe Monniez
c13128a229 [FIX] runbot: try to update github status multiple times
When updating github statuses, it happens that we face a "Bad gateway"
from github. In that case, the error is logged in the runbot logs and
that's it. As a consequence, when the runbot_merge is waiting status for
the staging branch and this kind of error occurs, the runbot_merge
timeouts and the users vainly search the reason.

With this commit, the runbot tries to update the status at least twice.
If it fails, an INFO message is logged on the build itself.
2019-06-19 10:20:38 +02:00
Xavier-Do
2fb0f2c79a [FIX] runbot: kill children even if state is done 2019-06-18 13:33:45 +02:00
Christophe Monniez
ae1eed46f5 [IMP] runbot: add optional warning message on frontend 2019-06-18 12:43:25 +02:00
Xavier-Do
080d5a4e1b [IMP] runbot: remove testing count from repo view 2019-06-18 12:42:07 +02:00
Xavier-Do
6d36455ef1 [FIX] runbot: ensure that job and build end are always set. 2019-06-18 12:42:07 +02:00
Xavier-Do
af7a8b7d8b [IMP] runbot: kill subbuild when killing build 2019-06-18 12:42:07 +02:00
Xavier-Do
a246d60c71 [IMP] runbot: always create new build at rebuild 2019-06-18 12:42:07 +02:00
Xavier-Do
17c57d499e [IMP] runbot: exact rebuild
allow to rebuild using exact config, dependencies, ...
2019-06-18 12:42:07 +02:00
Xavier-Do
a3ff9d102d [IMP] runbot: small runbot_branch view improvements 2019-06-18 12:42:07 +02:00
Xavier-Do
7e9826ba4c [IMP] runbot: add build creation log 2019-06-18 12:42:07 +02:00
Xavier-Do
f691ab5a39 [IMP] add hidden in view form 2019-06-18 12:42:07 +02:00
Xavier-Do
55c4e49ada [FIX] runbot: fix time ralated failing test
Some test where failing since _find_new_commits will skip
commit older than max_age
2019-06-18 12:42:07 +02:00
Xavier-Do
bdc6506ede [IMP] runbot: always keep one empty slot when assigning sheduled
Nightly build have a low priority but once they have a slot, they keep it.
When pushing a branch or asking robodoo to nicelly merge a branch for the
third time at 22:00, there may be no slot left since all the nightly build are created.

This commit will only assign scheduled build if there is no other build to create and
will always keep a free slot for other builds.
2019-06-18 12:42:07 +02:00
Xavier-Do
86ffeac589 [FIX] runbot: test_enable should be True by default 2019-06-18 12:42:07 +02:00
Christophe Monniez
2715585e31 [IMP] runbot: add python3-markdown in Dockerfile
Needed for running migration scripts.
2019-06-18 11:50:48 +02:00
Christophe Monniez
b98ff23cd9 [IMP] runbot: add write_file and make_dirs methods
When using a "python job", it's sometimes useful to write a file or
create a directory.
Instead of giving a wide open access to the os module in the
"_run_python" context, this commit adds a write_file and make_dirs
methods on the build which is usable in the _run_python eval context.
2019-06-18 11:50:48 +02:00
Christophe Monniez
800691a14c [IMP] runbot: add a mode parameter to readfile
When using the read_method in a "python job", it's sometimes needed to
read a file in binary mode.

With this commit, the mode can be specified when calling the method.
2019-06-18 11:50:48 +02:00
Christophe Monniez
799f3aed22 [IMP] runbot: add more objects in run python
Since the use of the "python jobs", we spotted various needs that were
not fulfilled. In order to add flexibility to "python jobs", this commit
adds some useful objetcs in the _run_python eval context.

Also, the glob.glob function is given instead of the whole glob module
to avoid giving access to the os module via glob.os.
2019-06-18 11:50:48 +02:00
Xavier-Do
c5588b1861 [IMP] runbot: display config name 2019-06-18 11:31:14 +02:00