The initial pull_info_failures feature was introduced before the separate
process for the leader. Since this is here for a rare case, this
wasn't tested since then.
Testing the new infrastructure without token, there was many
pull_info_failure, leading to an error on cleanup.
When rendering templates for git configuration and nginx configuration,
the templates are rendered as Markup, while a byte-like object was
expected for the saving.
With this commit, the Markup is casted into str and the files are saved
as text files.
Some non tested code was making running build available without
an nginx config (using port and local address instead of using a proper
dns/nginx config).
Removing this part to reduce complexity.
Source cleanup will check in multiple place for potentially used
commits. This maked the cleanup logic complex, plus limit the usable
commits in python steps. The current use case is to export the mergebase
commits.
The poposed solution is to save the exported commit and clean this list
when the build is done. All commit in this list cannot be cleaned.
The main motivation is to be able to see existing config_data on params
and edit configuration on triggers.
A simple version would be to use the `FieldChar` with a simple
`JSON.stringify` formater, but having some indent on field make it
easier to read and edit.
The multiline need make it closet to the `FieldText` than the `FieldChar`
but the reset makes the browser freeze.
The actual way to add a custom multibuild on a dev bundle will be:
- use the multibuild wizard to create 2 configs and 2 steps
- add this config on the bundle
- customize this config to make it fatser by installing/restoring...
The usefull parameters are the number of build, the test tags and
modules to install.
Another usefull operation is to restore a dump instead of installing,
especially for post install test only breaking on full databases.
This commit proposes to replace the multi build wizard with a custom
trigger wizard. The main idea is to have generic config, parametrized by
config_data. The wizard will only help to generate the corresponding
config_data.
When a build is garbage collected and the local folders are cleaned, the
log files are kept to allow build errors investigations.
With this commit, a full local cleanup will be done 365 days after the
garbage collect. This means that the build dir will be completely
removed. The default number of days can be tweaked with the
`runbot.full_gc_days` ir.config_parameter.
Also, the _local_cleanup method can be called with a `full` boolean
parameter to force a full cleanup. e.g.: when called in a config step.
While at it, the res_config_settings view is a bit reorganized.
When a git archive fails, the partially exported source tree is left in
place. If another builds tries to use the same commit, the tree is not
exported anymore as the directory exists. This leads to non
deterministic behaviors.
Right now single version repo like upgrade are managed using
a regex to limit name prefix, this avoid grouping branches when
mergebot wont be able to merge them together but the ci can be painfull
since the branch needs to be renamed (closing existing pr) or a manual
operation to move the branch into a new bundle must be performed.
This commit proposes to replace the forbidden_regex mechanism with
an explicit single_version mechanism.
In this case the reference name will be automatically prefixes with
the version. The prefix contains `---` to indicate that some
magic was applied.
Runbot layout modifies the website/portal base layout to remove navbar,
footer, overides some custom styles. A lot of assets are loaded but not
used. The only real usefull elements are base assets (bootstrap, ...)
and the login button.
Migrating to the next version of odoo is usually painfull because some
xpath may break, extra element added, or some style change may break the
page, needing to add more and more xpath, css rules, ... for very little
benefits.
This cleanup creates a custom base layout for runbot independant from
base odoo templates.
Also add a breadcrumb, navigation arrow, and improve batch links
- searching on number will search for both pr and branche name
- hooks are now using payload to define repo when not given in url
- fixes .git cleaning in repo
(remove rstrip since it can fail for repo starting with g, i, t)
- recompute base on prepare if base was not found
- remove local_result form write values if there is a single record
(instead of raising, makes python step easier to write).
- avoid stucked build/loop after removing a step from a config.
- avoid to send ci for linked base_commit
- add a fallback mechanism for base if no master branch is found
- add option on project to avoid to keep sticky running, usefull
when using a lots of projects
WARNING: this is a change of default behaviour, need to update
existing projects.
- always discover new commits for branch matching base paterns.
This is especially usefull to discover old versions on project with
low merge frequency.
- always create a batch, event if there is now trigger. This helps to
notice that commits are discovered
- add line-through on death branches/pr
- manual trigger are now displayed on main page
As for the builder, this give the ability to run the discovery of new
commits and all related logic in a separate process.
This will mainly be usefull to restart frontend without waiting for cron
or restart "leader" without stoping the frontend. This will also be
usefull for optimisation purpose.
As a custom codeowner system was successfully implemented in a python
step on our runbot instance, it's now time to have a real model for
that.
This commit adds a skeleton Codeowner model in order to be used for a
basic usage.
This should be improved in the future after some battle testing.
With the increasing usage of runbot to test various things and to take
care of random bugs in tests, the need of a team dashboard arose.
This commit adds a `runbot.team` model. Internal users can be
linked to the team. Module wildcards can be used to automatically assign
build errors to a team at 'build.error` creation.
Also, an upgrade exception can be assigned to a team in order to display
it on a dashboard.
A dashboard model is used to create custom dashboards on the team
frontend page. By default, a dashboard is meant to display a list of
failed builds. The failed builds are selected by specifying a project, a
trigger category (e.g. nightly), a config and a domain (which select
failed builds by default).
The dashboard can be customized by specifying a custom view.
Each created team has a frontend page that displays all the team
dashboards and the errors assigned to the team.
A few other improvement also come with this commit:
* The cleaned error is now in a tab on the build error form
* Known errors are displayed as "known" on the build log page
* The build form shows the config used for the build
When getting pull info, the alive state can be determined easily,
meaning that this field can join the "_compute_branch_infos" familly
Hook was catching some changes made on pr and was conditionnaly updating some fields
and triggering some other operations conditionaly depending on the action flag.
All of the information needed to update the pull info should always be present in the
payload body, meaning that all fields can be updated at once in case some hook was missed,
and additionnal operation can be triggered based on fields changes.
In some conditions, it appears that a containerized build can eat up
all memory of the container host. This leads to disturbance of other
builds as the kernel OOM killer enters the dance.
With this commit, the docker ability to limit memory usage of a
container is used. The OOM killer will choose its victim among the
container processes.
The containers memory limit has to be set in the runbot settings. If not
set, no memory limit is used.
runbot servers are running with a log-level debug in order to have usefull
debug information, but this also causes some noise comming from odoo.
This pr changes most debug to info.
- Add draft pr management to avoid to trigger code owner on draft pr.
- Add check on falsy config on trigger id (avoid crash, usefull to disable trigger)
- Add extra params on custom trigger to avoid to write specific config every time.
- Trigger a new batch automaticaly when updating target/draft
Some projects may use a totally different Dockerfile. In order to avoid
new branches of those projects to automatically build with the generic
default Dockerfile, this commit adds the possibility to configure a
Default Dockerfile on a project.
The test in the original code will never fire because the value searched
for is not in the keys of the dictionary, but in one of the lists which
are in the values. Work around this by maintaining a reverse dictionary
module name -> commit and use this for the test.
Sometimes a pr pull info can fail.
- Most of the time it is only temporary and it will be successfull on next try.
- In some rare case the pr will always fail (github inconsistency) The pr exist in git but not on github api.
For this rare case, we store the pr in memory in order to unstuck other pr/branches update.
We consider that this error should not remain, in this case github needs to fix the inconsistency.
This is why the runbot model don't handle such a case for now.
Another solution would be to create the pr with fake pull info. This idea is not the best one
since we want to avoid to have many pr with fake pull_info in case of temporary failure of giothub services.
With this solution, the pr will be retried once every cron loop.
We dont except to have pr with this kind of persistent failure more than every few mounths/years.
Since 360e31ade4, it's possible to add statistics values to build
results but there was no practical way to analyze them.
With this commit, there is a new button on the bundle page that leads to
a chart page that displays those values.
The default reference build is last known good build of the bundle.
Values are filtered by key and only the most significant values are
displayed. The user can then refine the chart by changing the reference
build or the key and a few other options.
Co-author: Xavier-Do <xdo@odoo.com>
If you try to manually create bundle, Odoo will crash, because it will
try to use `name` value that is not set yet. For that we start computing
host_id once `name` is entered.
When a build is reaching the run_run_odoo step, a database has to be
set. If none are found in the build params, the one from the last step
is choosen. Historically, the last one in the `Split` config was `all`
but now, the last one is `base`.
With this commit, if none are found in build params, `all` is choosen if
found in any install config steps. As a default, the one from the first
step is choosen.
Since 3657a65b20 docker_run is called outside of the step and python
steps have to set the `docker_params` variable. This breaks the computed
`log_list` because the string `docker_run(` is searched in python
steps code to determine if it's a docker_step.
With this commit, the `docker_params = ` is searched instead.
For now, the sticky flag is used to define bundle to use as target to upgrade.
The plan for future is to continue to test upgrade, even if the bundle is not sticky anymore.
This new flag will allow to remove sticky for old branche.
This will also allow to disable upgrda tests for still sticky branches when needed.
This commit also filter source bundle based on this flag
(before that all base bundle where used as source, even if not sticky)
Params mechanism are a way to avoid to rebuild with the exact same params.
Commit is the main identifier to know if two builds are the same, ususally
commit_link are not usefull to uniquify params, exept when the mergebase is
in use as in security check (based on diff).
We would also like to avoid to have red diff-based build in sticky/base branches when the
ci is ignored on a pr, but we want to keep the diff display on new batches
(lines and files changed).
We need a way to know if it is a is_base bundle. This was an initially forbidden behaviour
because of duplicate detection.
This commit add a batch_dependent flag. This will enable the use of the batch in the
fingerprint, making it unique per batch and disabling duplicate detection/ in this
case we can use the "is_base" information from the bundle + avoid false duplicate
detection if trigger is based on mergebase-commit diff. (pr based on another pr now merged)
On the build error web page, a regular assigned error is not shown to
all users.
With this commit, a regular build error (not only random) will be shown
if the error is assigned.
When builds logs are parsed by using the contextual button, the client
stays on the same page even if a build error is created.
With this commit, the client is now redirected to the created/found
build error(s).
When calling a step from a python step, it is impossible to alter some parameter, the only solution is to copy all step code.
With this change, python step are now able to override docker_run parameter of another step by modifying the dict returned by run_* steps
and assigning the result to "docker_params".
This will mainly be used to set the correct docker_image for migration pre and post test. This can also be used to alter a command,
like removing the pip install or adding extra pre/post operation on the command.
Currently, runbot is using a single Dockerfile maintained in a data file
in the source code. This situation is not convenient for testing Odoo in
different environments.
With this commit, a Dockerfile Odoo model is used to allow usage of
multiple Docker containers.
This model comes with a pre-defined Dockerfile that can be used to build
the current Odoo supported versions (12.0 up to 14.0).
The old order based on status is not really needed anymore:
- Scheduled builds have a special condition and already have a low priority.
- Indirect builds don't exist anymore
- It is actually questionnable to postpone rebuild. Sometimes they are needed but stuck.
This new proposition will keep subbuild scheduling close to parent build. Some build may take 2 hours
because they are parallelized and children are stuck. Priority should be defined by top parent.
Runbot initial architechture was working for a single odoo repo, and was
adapted to build enterprise. Addition of upgrade repo and test began
to make result less intuitive revealing more weakness of the system.
Adding to the oddities of duplicate detection and branch matching,
there was some room for improvement in the runbot models.
This (small) commit introduce the runbot v5.0, designed for a closer
match of odoo's development flows, and hopefully improving devs
experience and making runbot configuration more flexible.
**Remotes:** remote intoduction helps to detect duplicate between odoo and
odoo-dev repos: a commit is now on a repo, a repo having multiple remote.
If a hash is in odoo-dev, we consider that it is the same in odoo.
Note: github seems to manage commit kind of the same way. It is possible
to send a status on a commit on odoo when the commit only exists in
odoo-dev.
This change also allows to remove some repo duplicate configuration
between a repo and his dev corresponding repo.
(modules, server files, manifests, ...)
**Trigger:** before v5.0, only one build per repo was created, making it
difficult to tweak what test to execute in what case. The example use
case was for upgrade. We want to test upgrade to master when pushing on
odoo. But we also want to test upgrade the same way when pushing on
upgrade. We introduce a build that should be ran on pushing on either
repo when each repo already have specific tests.
The trigger allows to specify a build to create with a specific config.
The trigger is executed when any repo of the trigger repo is pushed.
The trigger can define depedencies: only build enterprise when pushing
enterprise, but enterprise needs odoo. Test upgrade to master when pushing
either odoo or upgrade.
Trigger will also allows to extract some build like cla that where
executed on both enterprise and odoo, and hidden in a subbuild.
**Bundle:** Cross repo branches/pr branches matching was hidden in build
creation and can be confusing. A build can be detected as a duplicate
of a pr, but not always if naming is wrong or traget is invalid/changes.
This was mainly because of how a community ref will be found. This was
making ci on pr undeterministic if duplicate matching fails. This was
also creating two build, with one pointing to the other when duplicate
detection was working, but the visual result can be confusing.
Associtaions of remotes and bundles fix this by adding all pr and
related branches from all repo in a bundle. First of all this helps to
visualise what the runbot consider has branch matching and that should
be considered as part of the same task, giving a place where to warn
devs of some possible inconsistencies. Associate whith repo/remote, we
can consider branches in the same repo in a bundle as expected to have
the same head. Only one build is created since trigger considers repo,
not remotes.
**Batch:** A batch is a group of build, a batch on a bundle can be
compared to a build on a branch in previous version. When a branch
is pushed, the corresponding bundle creates a new batch, and wait for
new commit. Once no new update are detected in the batch for 60 seconds,
All the trigger are executed if elligible. The created build are added
to the batch in a batch_slot. It is also possible that an corresponding
build exists (duplicate) and is added to the slot instead of creating a
new build.
Co-authored-by d-fence <moc@odoo.com>
Runbot can send status multiple time for the same hash:
- if transaction fails in scheduler and is retried
- if multiple subbuild are failing
Leading to multiple issues:
- when github receive more than one failure status, mergebot will
be notified multiple times and send multiple mail (for forward ports mainly)
- github will answer `422 Unprocessable Entity for url...` after
1000 status.
This fix proposes to limit number of status:
- By avoiding to send status for orphan build (parent status will never change)
- By storing last send status to avoid to notify multiple time
- By sending status post commit to avoid to contact github in case of failure.
This will also slightly reduce transaction time by removing an http request.
Sometimes, it happens that a `git fetch` fails with an error code 128
for example. When this happens, the runbot host is immediately disabled.
During investigations of such cases, we found that simply retrying the
fetch command works.
With this commit, the fetch command is tried 5 times with an increasing
delay before deciding to disable the runbot host.
Since we store the target_branch_name, filtering out pull head names
that contains `patch-` is not necessary anymore.
This commit is one first step towards a clean refactoring.
When a build is done, various numerical informations could be extracted
from log files. e.g.: global query count or tests query count ...
The extraction regular expression could be hard-coded in a custom step
but there is no place holder where to store the retrieved information.
In order to compare results, we need to store it.
With this commit, a new model `runbot.build.stat` is used to store
key/values pair linked to a build/config_step. That way, extracted
values can be stored.
Also, another `runbot.build.stat.regex` is used to store regular
expressions that can be used to grep log files and extract values.
The regular expression must contain a named group like this:
`(?P<value>.+)`
The text catched by this group MUST be castable into a float.
Optionally, another named group can be used in the regular expresion
like this:
`(?P<key>.+)`
This `key` group will then be used to augment the key name in the
database.
Example:
Consider a log line like this one:
`odoo.addons.website_blog.tests.test_ui tested in 10.35s`
A regular expression like this, named `test_duration`:
`odoo.addons.(?P<key>.+) tested in (?P<value>\d+\.\d+)s`
Should store the following key:value:
`{
'key': 'test_duration.website_blog.tests.test_ui',
'value': 10.35
}`
A `generic` boolean field is present on the build.stat.regex object,
meaning that when no regex are linked to a make_stats config step, then,
all the generic regex will be applied.
A wizard is added to help the creation the regular expressions, allowing
to test if they work against a user provided example.
A _make_stats method is added to the ConfigStep model which is called
during the _schedule of a build, just before calling the next step.
The regex search is only apllied in steps that have the `make_stats`
boolean field set to true. Also, the build branch have to be flagged
`make_stats` too or the build can have a key `make_stats` in its
config_data field.
The `make_stats` field on the branch is a compute stored field.
That way, sticky branches are automaticaly set `make_stats' true.
Finally, an SQL view is used to facilitate the stats visualisation.
When coverage is computed, a post command is used to generate the HTML
report. In order to use the coverage result locally the HTML report is
not enough.
With this commit, an XML report is also generated. It's a single xml
file, downloadable from the build result web page.
The _post_install_command method is renamed into its plural form because
it was useless to return only one command.
From times to times, a git repo gets corrupted, making builds fail in
chain.
With this commit, the host on which the git fetch fails will be reserved
if not more than half the hosts are reserved.
Before intensive python steps, every build exept create build should have logs.
This is not true since now many python steps are creating build, leading
to 404 links to inexisting logs file.
Checking That a step is a docker build looks like a good heuristic since since
most of the time the log file will be the one created by docker. In all case we
can access other logs files in browsing /logs.
A future improvement would be to listdir logs to create all links.
With this commit, a kind of markdown is allowed in log messages and
build.description.
Following patterns are supported:
**strong**
~~striketrough~~
__underline__
`code`
[link](target)
@icon-font-awesome-class
When a build.config is copied, the config steps are not copied.
The steps have to be explicitly ordered otherwise it leads to a
traceback when trying to copy a 'run' step which have to be the last
one.
When creating a subbuild from a custom python job/cron, some data can
be given through config_data or extra_params to change another step behaviour.
An example is to give a specific -i to an install job in order to
install different modules from the parent build. Anyway since we
are using the same install step in this case, all databases will
have the same name, and the name may not be correct in regard to
the database content.
This commit allows to give a config_param on build giving the default
db_name to use in an install step.
If a branch triggers an hidden build on a another one (sticky ususally),
the last build of the branch will be considered to be this one and
trigger a new build. The same problem can occur when cloning a subbuild
to slighlty change a parameter instead of making an exact rebuild
since the build type may still be normal in this case.
This commit will simply ignore subbuild in this case.
When a new commit is found, a rebuild is forced on sticky branches
builds in repositories that depends on the new commit repository.
If one of the repo is protected by groups, the main page gives access
errors for public users and finally leads to a CPU overload.
With this commit, the feature is removed as it will be re-implemented in
custom config steps.
On non sticky branches, when a new build is found while another is
already testing, the older build is killed. This happens during when the
main runbot instance is discovering new commits and create new builds.
As a result, concurrent updates may occur while the builders access the
concerned build.
With this commit, this garbage collecting procedure occurs during the
scheduler loop that runs on runbot builder hosts.
Also, the logic changed in a way that the kill is requested only if the
host needs room to handle pending builds.
When a build age reaches the gc_days parameter, its database is dropped
and its directory is removed.
With this commit, two fields are added in order to keep some builds
longer that the defined gc_days.
The gc_delay field on the build allows to add a delay (in number of
days) that is added to its gc_days to compute the gc_date.
The gc_date field is the date when the cleaning will occur.
Also, a test is added and the RunbotCase test class is improved to allow
the stop of a patcher.
Since cb05f2b9d8, when creating a database, the template1 is used, this
allows to customize the template and install some needed Postgress
extensions.
Unfortunately, it's also a source of build failures. For example, if the
pg_activity util is used on a runbot host, database creation may fail
with a message like this:
source database "template1" is being accessed by other users
It's because pg_activity needs a database and uses template1.
With this commit, template1 is still used by default but can be changed
with a system parameter. That way, a custom template can be created on
runbot hosts and used when creating DB in builds.
When a PR branch target is changed on Github, the change is not applied
in the runbot DB.
With this commit, the Github hook payload is taken into account to
detect such a change and the branch infos are recomputed accordingly.
Also, a button is now available on the branch form in order to manually
recompute those changes.
At this moment, the Docker image is built at the beginning of each
runbot build. This blocks the _scheduler while the image is built.
With this commit, the image is built before calling the _scheduler and
is not linked to a runbot build.
Also, the necessary dirs are created in the static path before starting
the loop.
Since 857821e4 a screenshot can be saved in the build directory under
the "tests" subdirectory.
Unfortunately, this directory is cleaned in case of local_cleanup.
With this commit, the 'tests' subdirectory is kept in place.
If a docker is killed from outside, the start file will still be there but
not the end file. (when restarting docker service for instance)
This commits add a docker_state specific to non running docker
with start a file, that should be handled like unknow state:
This state is acceptable for a while, but build should be killed
if this state remains for to long.
When two steps in the same build needs to exchange informations, some
hacks have to be used. E.g. using the extra_params fields to store comma
separated values.
With this commit, a config_data field is added alongside with a
JsonDictField that automatically transform the data into json.
When changing the sticky value in a branch form, it triggers the
computes of previous version and intermediate versions.
In the onchange situation, the branch.id is a NEW id and it fails with a
traceback.
With this commit, a verification is made to ensure that the id is there.
ORM does not support non_searchable.non_stored dependency.
thus, the closest_sticky.previous_version dependency will log an error
when previous_version is written.
this dependency is usefull to make the compute recursive, avoiding to have
both record and record.closest_sticky in self, in that order, making the record.previous_version
empty in all cases.
Writing self on sticky will mitigate the problem. but it is still posible to
have computation errors if defined_sticky is not sticky. (which is not a normal use case)
This reverts commit 54f9b9b546.
The main reason is linked to inconsistency in state compute because
of error in nb_ computations.
This was to avoid concurrent update, witch is not a such a big problem
now since workers are no longer using crons. (retry on failure is faster).
When trying to remove test_tags on a build_error, the validation fails
because it tries to iterate on False. Also, the ValidationError
exception was not properly imported.
With this commit, the validation is fixed and a test is added.
When a repo is set to "no_build", there is no way to force a branch to
build from the backend.
With this commit, a button is added on a branch form to ask rebuild of the
branch, even when the repo is set to "no_build".
Migration tests comming on runbot, it will be usefull to have quick
way to obtain branches related to current build.
This commit adds a field for the colsest sticky branch, previous version
and intermediates stickies.
Example when last sticky is saas-13.2:
branch_name: master-test-tri
closest_sticky: master
previous_version: 13.0
intermediate_stickies: saas-13.1, saas-13.2
This commit add the possibility to add custom checks to python steps,
as well as ignoring triggered result if log of level error/warning
is not considered as a problem.
As the --upgrade-paths options does not work as expected in Odoo, a
symbolic link has to be created in odoo/addons/base/maintenance pointing
to the migration scripts.
The runbot uses Docker read-only volumes to access the sources that are
shared between builds, preventing the creation of such a link.
With this commit, a symbolic link is created right after the export of a
commit only when the repo is a "server" repo.
This link is broken outside of the Docker volumes but uses the mount
points of the sources inside the container.
Two ir.config_parameter's are used to enable and configure this feature:
* runbot_migration_ln: the relative path where the link should be placed
(typically odoo/addons/base/maintenance)
* runbot_migration_repo_id: the id of the migration scripts repo, used
to determine the name of the mount point inside the Docker container
A change is also made in the "reverse dependcy build" to avoid the
creation of a build in the migration repo for each push in its
dependency. Simply set the no_build field on this migration repo.
