- clean thread username
- allow to write on params for debug (was mainly usefull to forbid it
at the beginning)
- imrpove some guidelines about method and actions naming/ ordering
- move some code for a cleaner organisation.
- remove some useless request.env.user (not useful anymore)
This commit will replace the symlink used for upgrade by the
upgrade-path.
The symlink was used before because old version does not support upgrade
path, but the decision was taken to now limit the testing to version
suporting upgrade paths in order to be able to support utils in another
repository latter.
One of the most common custom trigger is to restore a build before
starting some test, either to create a multibuild or make the execution
and debug of some test faster.
It is somethimes tedious to use because we need to give an url of a
build to restore. This build must correspond to the right commits,
must still exixt, ... this means that the dump url must be adapted
everytime a branch is rebased.
The way the dump_url is defined is by going on the last batch, following
the link to the `base_reference_batch_id`, finding a slot corresponding
to the right repo set, (ex: Custom enterprise -> enterprise), and
copying the dump_url in this build.
The base_reference_batch_id is eay to automated but we have to find the
right trigger, this is now a parameter of the custom trigger wizard.
There are actually 2 strategy now to define how to download the dump:
- `url`, using `restore_ dump_url`
- `auto`, using `restore_trigger_id` and `restore_database_suffix`
To ease the setup, a `restore_trigger_id` is added on a trigger, so that
when selecting a trigger, lets say `Custom enterprise`, the defined
`trigger.restore_trigger_id` is automatically chosen for the
`custom_trigger.restore_trigger_id` and the `restore_mode` is setted to
auto.
Two actions are also added to the header of a bundle, a shorcut to
setup a multi build (restore in children) or a restore and test build
(restore in parent).
Since removeprefix was not available in ubuntu 20.04, a easier alternative
using rebase was used.
The initial assumption was that the prefix would look like `odoo/addons/`
and won't be in the filename.
When the repo is enterprise, the prefix is `enterprise/` meaning that
module name ending with `enterprise` will be truncated
`repo._get_module('enterprise/mail_enterprise/static/src/widgets/form_renderer/form_renderer.js')`
will output
`mail_static`
The auto disable host is mainly usefull when there are a lot of host for
well configured repositories.
If for any reason a repo is corrupted on one host, this host will be
disabled until a manual intervention cleans the repo.
For other cases, where thjere are many repositories with not so many
host, it is most likely that a fetch will fail because of an invalid
repository configuration. Disabling the host in this case is not a good
idea.
With this commit, a settings allows to enable or disable this feature.
Right now, a branch with a numerical name will be added to the database,
but it can conflict with pr since the name of a pr is a number.
This means that a unique (name, repo_id) constraints can be broken.
We could use the 'is_pr' in the unicity constraints to avoid this issue
but searching on branch name will give confusing result if some of them
can be numerical.
Moreover, a runbot branch name should start with the version name
meaning that a numerical branch name was a bad idea from the begining.
This idea was postpone for a while since this was most a mergebot
responsability but having the github login of the user will help
for some team feature requests.
The main one is to only ping a team if the pr was not opened by a member
of the team. We want to let the team manager manage that as much as
possible so the team manager group will be able to write the user
github login (as well as the user himself) and add a list of non user
github_login to consider if not all user have a account on runbot.
This commit also improves the views for team managers.
Regarding the number of refs in odoo repo (arround 18 million at this
time), the parsing of the date was significant when filtering old refs.
Using unix time allows a direct comparaison without parsing the date,
and improved performance, going from ~7 seconds to ~1.3 seconds.
The status are currently sent by leader when build are created and by
workers on post_commit.
If the leader fails during the preparing of a batch (while creating
builds) the transaction is rollbacked and the status are send again.
The number of status to send makes it quite slow, making the transaction
longuer, and the retry even more expensive. This leads to preparing time
being quite important, sometimes ten minutes after many retries.
This commit proposes to send status in another dedicated transaction.
Since status are sent in batch, we can also try to use a unique session,
and uniquify commit+context status.
This allows to remove the postcommit logic
A further improvement would be to wait before sending status in order to
skip the pending status if the build is verry fast.
An option is also added on the remote to skip the status: if the remote
is a fork, sending the satus on the main remote should be enough.
When rendering templates for git configuration and nginx configuration,
the templates are rendered as Markup, while a byte-like object was
expected for the saving.
With this commit, the Markup is casted into str and the files are saved
as text files.
Right now single version repo like upgrade are managed using
a regex to limit name prefix, this avoid grouping branches when
mergebot wont be able to merge them together but the ci can be painfull
since the branch needs to be renamed (closing existing pr) or a manual
operation to move the branch into a new bundle must be performed.
This commit proposes to replace the forbidden_regex mechanism with
an explicit single_version mechanism.
In this case the reference name will be automatically prefixes with
the version. The prefix contains `---` to indicate that some
magic was applied.
- searching on number will search for both pr and branche name
- hooks are now using payload to define repo when not given in url
- fixes .git cleaning in repo
(remove rstrip since it can fail for repo starting with g, i, t)
- recompute base on prepare if base was not found
- remove local_result form write values if there is a single record
(instead of raising, makes python step easier to write).
- avoid stucked build/loop after removing a step from a config.
- avoid to send ci for linked base_commit
- add a fallback mechanism for base if no master branch is found
- add option on project to avoid to keep sticky running, usefull
when using a lots of projects
WARNING: this is a change of default behaviour, need to update
existing projects.
- always discover new commits for branch matching base paterns.
This is especially usefull to discover old versions on project with
low merge frequency.
- always create a batch, event if there is now trigger. This helps to
notice that commits are discovered
- add line-through on death branches/pr
- manual trigger are now displayed on main page
When getting pull info, the alive state can be determined easily,
meaning that this field can join the "_compute_branch_infos" familly
Hook was catching some changes made on pr and was conditionnaly updating some fields
and triggering some other operations conditionaly depending on the action flag.
All of the information needed to update the pull info should always be present in the
payload body, meaning that all fields can be updated at once in case some hook was missed,
and additionnal operation can be triggered based on fields changes.
Sometimes a pr pull info can fail.
- Most of the time it is only temporary and it will be successfull on next try.
- In some rare case the pr will always fail (github inconsistency) The pr exist in git but not on github api.
For this rare case, we store the pr in memory in order to unstuck other pr/branches update.
We consider that this error should not remain, in this case github needs to fix the inconsistency.
This is why the runbot model don't handle such a case for now.
Another solution would be to create the pr with fake pull info. This idea is not the best one
since we want to avoid to have many pr with fake pull_info in case of temporary failure of giothub services.
With this solution, the pr will be retried once every cron loop.
We dont except to have pr with this kind of persistent failure more than every few mounths/years.
Since 360e31ade4, it's possible to add statistics values to build
results but there was no practical way to analyze them.
With this commit, there is a new button on the bundle page that leads to
a chart page that displays those values.
The default reference build is last known good build of the bundle.
Values are filtered by key and only the most significant values are
displayed. The user can then refine the chart by changing the reference
build or the key and a few other options.
Co-author: Xavier-Do <xdo@odoo.com>
Params mechanism are a way to avoid to rebuild with the exact same params.
Commit is the main identifier to know if two builds are the same, ususally
commit_link are not usefull to uniquify params, exept when the mergebase is
in use as in security check (based on diff).
We would also like to avoid to have red diff-based build in sticky/base branches when the
ci is ignored on a pr, but we want to keep the diff display on new batches
(lines and files changed).
We need a way to know if it is a is_base bundle. This was an initially forbidden behaviour
because of duplicate detection.
This commit add a batch_dependent flag. This will enable the use of the batch in the
fingerprint, making it unique per batch and disabling duplicate detection/ in this
case we can use the "is_base" information from the bundle + avoid false duplicate
detection if trigger is based on mergebase-commit diff. (pr based on another pr now merged)
Runbot initial architechture was working for a single odoo repo, and was
adapted to build enterprise. Addition of upgrade repo and test began
to make result less intuitive revealing more weakness of the system.
Adding to the oddities of duplicate detection and branch matching,
there was some room for improvement in the runbot models.
This (small) commit introduce the runbot v5.0, designed for a closer
match of odoo's development flows, and hopefully improving devs
experience and making runbot configuration more flexible.
**Remotes:** remote intoduction helps to detect duplicate between odoo and
odoo-dev repos: a commit is now on a repo, a repo having multiple remote.
If a hash is in odoo-dev, we consider that it is the same in odoo.
Note: github seems to manage commit kind of the same way. It is possible
to send a status on a commit on odoo when the commit only exists in
odoo-dev.
This change also allows to remove some repo duplicate configuration
between a repo and his dev corresponding repo.
(modules, server files, manifests, ...)
**Trigger:** before v5.0, only one build per repo was created, making it
difficult to tweak what test to execute in what case. The example use
case was for upgrade. We want to test upgrade to master when pushing on
odoo. But we also want to test upgrade the same way when pushing on
upgrade. We introduce a build that should be ran on pushing on either
repo when each repo already have specific tests.
The trigger allows to specify a build to create with a specific config.
The trigger is executed when any repo of the trigger repo is pushed.
The trigger can define depedencies: only build enterprise when pushing
enterprise, but enterprise needs odoo. Test upgrade to master when pushing
either odoo or upgrade.
Trigger will also allows to extract some build like cla that where
executed on both enterprise and odoo, and hidden in a subbuild.
**Bundle:** Cross repo branches/pr branches matching was hidden in build
creation and can be confusing. A build can be detected as a duplicate
of a pr, but not always if naming is wrong or traget is invalid/changes.
This was mainly because of how a community ref will be found. This was
making ci on pr undeterministic if duplicate matching fails. This was
also creating two build, with one pointing to the other when duplicate
detection was working, but the visual result can be confusing.
Associtaions of remotes and bundles fix this by adding all pr and
related branches from all repo in a bundle. First of all this helps to
visualise what the runbot consider has branch matching and that should
be considered as part of the same task, giving a place where to warn
devs of some possible inconsistencies. Associate whith repo/remote, we
can consider branches in the same repo in a bundle as expected to have
the same head. Only one build is created since trigger considers repo,
not remotes.
**Batch:** A batch is a group of build, a batch on a bundle can be
compared to a build on a branch in previous version. When a branch
is pushed, the corresponding bundle creates a new batch, and wait for
new commit. Once no new update are detected in the batch for 60 seconds,
All the trigger are executed if elligible. The created build are added
to the batch in a batch_slot. It is also possible that an corresponding
build exists (duplicate) and is added to the slot instead of creating a
new build.
Co-authored-by d-fence <moc@odoo.com>
Sometimes, it happens that a `git fetch` fails with an error code 128
for example. When this happens, the runbot host is immediately disabled.
During investigations of such cases, we found that simply retrying the
fetch command works.
With this commit, the fetch command is tried 5 times with an increasing
delay before deciding to disable the runbot host.
From times to times, a git repo gets corrupted, making builds fail in
chain.
With this commit, the host on which the git fetch fails will be reserved
if not more than half the hosts are reserved.
If a branch triggers an hidden build on a another one (sticky ususally),
the last build of the branch will be considered to be this one and
trigger a new build. The same problem can occur when cloning a subbuild
to slighlty change a parameter instead of making an exact rebuild
since the build type may still be normal in this case.
This commit will simply ignore subbuild in this case.
When a new commit is found, a rebuild is forced on sticky branches
builds in repositories that depends on the new commit repository.
If one of the repo is protected by groups, the main page gives access
errors for public users and finally leads to a CPU overload.
With this commit, the feature is removed as it will be re-implemented in
custom config steps.
On non sticky branches, when a new build is found while another is
already testing, the older build is killed. This happens during when the
main runbot instance is discovering new commits and create new builds.
As a result, concurrent updates may occur while the builders access the
concerned build.
With this commit, this garbage collecting procedure occurs during the
scheduler loop that runs on runbot builder hosts.
Also, the logic changed in a way that the kill is requested only if the
host needs room to handle pending builds.
At this moment, the Docker image is built at the beginning of each
runbot build. This blocks the _scheduler while the image is built.
With this commit, the image is built before calling the _scheduler and
is not linked to a runbot build.
Also, the necessary dirs are created in the static path before starting
the loop.
When a repo is set to "no_build", there is no way to force a branch to
build from the backend.
With this commit, a button is added on a branch form to ask rebuild of the
branch, even when the repo is set to "no_build".
As the --upgrade-paths options does not work as expected in Odoo, a
symbolic link has to be created in odoo/addons/base/maintenance pointing
to the migration scripts.
The runbot uses Docker read-only volumes to access the sources that are
shared between builds, preventing the creation of such a link.
With this commit, a symbolic link is created right after the export of a
commit only when the repo is a "server" repo.
This link is broken outside of the Docker volumes but uses the mount
points of the sources inside the container.
Two ir.config_parameter's are used to enable and configure this feature:
* runbot_migration_ln: the relative path where the link should be placed
(typically odoo/addons/base/maintenance)
* runbot_migration_repo_id: the id of the migration scripts repo, used
to determine the name of the mount point inside the Docker container
A change is also made in the "reverse dependcy build" to avoid the
creation of a build in the migration repo for each push in its
dependency. Simply set the no_build field on this migration repo.
