When getting new refs, a lot of them are really old and the
find_new_commits is called for each one and thus browsing branches.
With this commit, refs older than configured max_age are ignored.
Co-authored-by: Xavier Dollé (xdo@odoo.com)
Accessing childrens can create rollback, especially for builds with
a lot of them, since other runbot will concurently access and update
states. This commit tries to improve this by incrementing and
decrementing counters instead of counting all of them each time.
A slow query was detected on the runbot, causing a latency when loading
any page. After some investigations (thnks jle), we found that an index
on local_state could improve speed.
When updating github statuses, it happens that we face a "Bad gateway"
from github. In that case, the error is logged in the runbot logs and
that's it. As a consequence, when the runbot_merge is waiting status for
the staging branch and this kind of error occurs, the runbot_merge
timeouts and the users vainly search the reason.
With this commit, the runbot tries to update the status at least twice.
If it fails, an INFO message is logged on the build itself.
Nightly build have a low priority but once they have a slot, they keep it.
When pushing a branch or asking robodoo to nicelly merge a branch for the
third time at 22:00, there may be no slot left since all the nightly build are created.
This commit will only assign scheduled build if there is no other build to create and
will always keep a free slot for other builds.
When using a "python job", it's sometimes useful to write a file or
create a directory.
Instead of giving a wide open access to the os module in the
"_run_python" context, this commit adds a write_file and make_dirs
methods on the build which is usable in the _run_python eval context.
When using the read_method in a "python job", it's sometimes needed to
read a file in binary mode.
With this commit, the mode can be specified when calling the method.
Since the use of the "python jobs", we spotted various needs that were
not fulfilled. In order to add flexibility to "python jobs", this commit
adds some useful objetcs in the _run_python eval context.
Also, the glob.glob function is given instead of the whole glob module
to avoid giving access to the os module via glob.os.
When a runbot instance is scheduling builds, the numbers of builds
depends of a global ir.config_parameter. Even if one of the runbot
instance is running on a more powerful systsem, its number of workers is
limited by this global parameter.
With this commit, this parameter still exists but can be overriden by
specific ir.config_parameter.
For example, if the host 'runbot24.odoo.com' has more cpu power, the
number of workers for this host can be specified in the
ir.config_parameter named 'runbot24.odoo.com.workers'.
When installing software with apt-get in a Dockerfile, it should be
preceded with an apt-get update in the same RUN. Otherwise, the step may
fail if the needed package has been updated.
In a create config, a parent result is computed based on children
results
In some situations, it could be handy to ignore the result of some
sub-builds.
Example: the nightly tests are just the children of one nightly build
with a create config. The external tests are failing randomly and as a
consequence, the nightly result is always red. On the other hand,
keeping the test running, just to have logs is a good idea.
With this commit, a config_step of type create can be marked as
orphan_result, that way, the result is not taken into account in the
parent build result.
When the quickconnect button is used, the last running build is
searched in the last 10 builds. If no running build is found, the last
one is rebuilt, even if it's a nightly build.
With this commit, the quickconnect build is choosen only among the ones
with the same config.
With recursive states computation, schedule is
most likely to have transactionnal errors.
This is particularly a problem when external
operations are done during the transaction,
like running a docker.
Adding some commits will help to reduce
transactionnal errors, and ensure that the db
is consistent with docker states.
As the public user needs to be in runbot user group to display the
frontend, the public user is allowed to kill or rebuild a build.
With this commit, only the logged in users have access to the Rebuil/Kill
menu entry.
When searchings for new refs to create builds, the for-each-ref git
commit is run and each ref is searched in the database which is a
somewhat heavy operation.
With this commit, the timestamp of the last database update with the
refs is stored in a field on the repo. This timestamp is checked each
time a for-each-ref is needed, running the operation only when
necessary.
This commit aims to replace static jobs by fully configurable build config.
Each build has a config (custom or inherited from repo or branch).
Each config has a list of steps.
For now, a step can test/run odoo or create a new child build. A python job is
also available.
The mimic the previous behaviour of runbot, a default config is available with
three steps, an install of base, an install+test of all modules, and a last step
for run.
Multibuilds are replace by a config containing cretaion steps.
The created builds are not displayed in main views, but are available
on parent build log page. The result of a parent takes the result of
all children into account.
This new mechanics will help to create some custom behaviours for specifics
use cases, and latter help to parallelise work.
The cpu limit used in job_20 uses the runbot_timeout config_parameter
since b539112a7e. When measuring coverage, this parameter is multiplied
and leads to an error because the type of ir.config_parameter.get_param
method is str.
With this commit, the this value is converted into integer before usage
in job_20.
When running Odoo in the Docker container, the username used to connect
to the database is the username defined in the docker container
(actually odoo).
A problem may arise if the user of the runbot process is not the same.
An authentication error is then raised by postgres because of the
username mismatch.
With this commit, the '-r' parameter of Odoo is added to the command
with the username used by the runbot process.
While at it, unused imports are removed.
When a build exceeds the cpu limit, it is simply killed by the kernel.
As a safeguard the "Initiating shutdown." sentence should be searched
in the log file, and the build marked as "ko" if not found.
Unfortunateley, there is no period (.) at the end of the sentence in the
Odoo logs (see: https://github.com/odoo/odoo/blob/12.0/odoo/service/server.py#L444)
Thus, this condition is never fulfilled.
On top of that, this was masked by the first part of the condition,
checking that the 'test/common.py' has no "post_install" string.
The "test" directory does not exists in Odoo ( but "tests" exists) , so
the condition was always falsy.
Finally, a build can be marked as "ok" when he is killed and no errors
are found until the kill.
With this commit:
* The legacy grep for post_install is removed as it now exists in
all Odoo supported versions.
* The period typo is fixed.
* A log is inserted when the final sentence is not found.
* The cpu_limit is set as the same as the runbot_timeout parameter
for better consitency.
* The time exceeded log message is now logged in the build instead
of the runbot log.
Co-authored-by: @Xavier-Do
