When adding a new project, if no branch matches a base name,
the created bundles won't have a version and it will fail.
A simple fix will be to add a master bundle for all projects.
The build error view was unstructured and contains to much information.
This commit organize fields in groups and also validate some
modification on records in order to avoid build error manager to
disable test-tags by mistake.
An error cannot be deactivated if it appeared less than 24 hours ago to
avoid disabling a non forxardported pr that will fail the next nightly
generating another build error.
Test tags can only be added or removed by administrators.
Also adds a menu for easier User managerment
Also fixed the dname search and display.
Right now single version repo like upgrade are managed using
a regex to limit name prefix, this avoid grouping branches when
mergebot wont be able to merge them together but the ci can be painfull
since the branch needs to be renamed (closing existing pr) or a manual
operation to move the branch into a new bundle must be performed.
This commit proposes to replace the forbidden_regex mechanism with
an explicit single_version mechanism.
In this case the reference name will be automatically prefixes with
the version. The prefix contains `---` to indicate that some
magic was applied.
When getting pull info, the alive state can be determined easily,
meaning that this field can join the "_compute_branch_infos" familly
Hook was catching some changes made on pr and was conditionnaly updating some fields
and triggering some other operations conditionaly depending on the action flag.
All of the information needed to update the pull info should always be present in the
payload body, meaning that all fields can be updated at once in case some hook was missed,
and additionnal operation can be triggered based on fields changes.
- Add draft pr management to avoid to trigger code owner on draft pr.
- Add check on falsy config on trigger id (avoid crash, usefull to disable trigger)
- Add extra params on custom trigger to avoid to write specific config every time.
- Trigger a new batch automaticaly when updating target/draft
Runbot initial architechture was working for a single odoo repo, and was
adapted to build enterprise. Addition of upgrade repo and test began
to make result less intuitive revealing more weakness of the system.
Adding to the oddities of duplicate detection and branch matching,
there was some room for improvement in the runbot models.
This (small) commit introduce the runbot v5.0, designed for a closer
match of odoo's development flows, and hopefully improving devs
experience and making runbot configuration more flexible.
**Remotes:** remote intoduction helps to detect duplicate between odoo and
odoo-dev repos: a commit is now on a repo, a repo having multiple remote.
If a hash is in odoo-dev, we consider that it is the same in odoo.
Note: github seems to manage commit kind of the same way. It is possible
to send a status on a commit on odoo when the commit only exists in
odoo-dev.
This change also allows to remove some repo duplicate configuration
between a repo and his dev corresponding repo.
(modules, server files, manifests, ...)
**Trigger:** before v5.0, only one build per repo was created, making it
difficult to tweak what test to execute in what case. The example use
case was for upgrade. We want to test upgrade to master when pushing on
odoo. But we also want to test upgrade the same way when pushing on
upgrade. We introduce a build that should be ran on pushing on either
repo when each repo already have specific tests.
The trigger allows to specify a build to create with a specific config.
The trigger is executed when any repo of the trigger repo is pushed.
The trigger can define depedencies: only build enterprise when pushing
enterprise, but enterprise needs odoo. Test upgrade to master when pushing
either odoo or upgrade.
Trigger will also allows to extract some build like cla that where
executed on both enterprise and odoo, and hidden in a subbuild.
**Bundle:** Cross repo branches/pr branches matching was hidden in build
creation and can be confusing. A build can be detected as a duplicate
of a pr, but not always if naming is wrong or traget is invalid/changes.
This was mainly because of how a community ref will be found. This was
making ci on pr undeterministic if duplicate matching fails. This was
also creating two build, with one pointing to the other when duplicate
detection was working, but the visual result can be confusing.
Associtaions of remotes and bundles fix this by adding all pr and
related branches from all repo in a bundle. First of all this helps to
visualise what the runbot consider has branch matching and that should
be considered as part of the same task, giving a place where to warn
devs of some possible inconsistencies. Associate whith repo/remote, we
can consider branches in the same repo in a bundle as expected to have
the same head. Only one build is created since trigger considers repo,
not remotes.
**Batch:** A batch is a group of build, a batch on a bundle can be
compared to a build on a branch in previous version. When a branch
is pushed, the corresponding bundle creates a new batch, and wait for
new commit. Once no new update are detected in the batch for 60 seconds,
All the trigger are executed if elligible. The created build are added
to the batch in a batch_slot. It is also possible that an corresponding
build exists (duplicate) and is added to the slot instead of creating a
new build.
Co-authored-by d-fence <moc@odoo.com>
Since we store the target_branch_name, filtering out pull head names
that contains `patch-` is not necessary anymore.
This commit is one first step towards a clean refactoring.
When a build is done, various numerical informations could be extracted
from log files. e.g.: global query count or tests query count ...
The extraction regular expression could be hard-coded in a custom step
but there is no place holder where to store the retrieved information.
In order to compare results, we need to store it.
With this commit, a new model `runbot.build.stat` is used to store
key/values pair linked to a build/config_step. That way, extracted
values can be stored.
Also, another `runbot.build.stat.regex` is used to store regular
expressions that can be used to grep log files and extract values.
The regular expression must contain a named group like this:
`(?P<value>.+)`
The text catched by this group MUST be castable into a float.
Optionally, another named group can be used in the regular expresion
like this:
`(?P<key>.+)`
This `key` group will then be used to augment the key name in the
database.
Example:
Consider a log line like this one:
`odoo.addons.website_blog.tests.test_ui tested in 10.35s`
A regular expression like this, named `test_duration`:
`odoo.addons.(?P<key>.+) tested in (?P<value>\d+\.\d+)s`
Should store the following key:value:
`{
'key': 'test_duration.website_blog.tests.test_ui',
'value': 10.35
}`
A `generic` boolean field is present on the build.stat.regex object,
meaning that when no regex are linked to a make_stats config step, then,
all the generic regex will be applied.
A wizard is added to help the creation the regular expressions, allowing
to test if they work against a user provided example.
A _make_stats method is added to the ConfigStep model which is called
during the _schedule of a build, just before calling the next step.
The regex search is only apllied in steps that have the `make_stats`
boolean field set to true. Also, the build branch have to be flagged
`make_stats` too or the build can have a key `make_stats` in its
config_data field.
The `make_stats` field on the branch is a compute stored field.
That way, sticky branches are automaticaly set `make_stats' true.
Finally, an SQL view is used to facilitate the stats visualisation.
When a PR branch target is changed on Github, the change is not applied
in the runbot DB.
With this commit, the Github hook payload is taken into account to
detect such a change and the branch infos are recomputed accordingly.
Also, a button is now available on the branch form in order to manually
recompute those changes.
When changing the sticky value in a branch form, it triggers the
computes of previous version and intermediate versions.
In the onchange situation, the branch.id is a NEW id and it fails with a
traceback.
With this commit, a verification is made to ensure that the id is there.
ORM does not support non_searchable.non_stored dependency.
thus, the closest_sticky.previous_version dependency will log an error
when previous_version is written.
this dependency is usefull to make the compute recursive, avoiding to have
both record and record.closest_sticky in self, in that order, making the record.previous_version
empty in all cases.
Writing self on sticky will mitigate the problem. but it is still posible to
have computation errors if defined_sticky is not sticky. (which is not a normal use case)
When a repo is set to "no_build", there is no way to force a branch to
build from the backend.
With this commit, a button is added on a branch form to ask rebuild of the
branch, even when the repo is set to "no_build".
Migration tests comming on runbot, it will be usefull to have quick
way to obtain branches related to current build.
This commit adds a field for the colsest sticky branch, previous version
and intermediates stickies.
Example when last sticky is saas-13.2:
branch_name: master-test-tri
closest_sticky: master
previous_version: 13.0
intermediate_stickies: saas-13.1, saas-13.2
When the quickconnect button is used, the last running build is
searched in the last 10 builds. If no running build is found, the last
one is rebuilt, even if it's a nightly build.
With this commit, the quickconnect build is choosen only among the ones
with the same config.
This commit aims to replace static jobs by fully configurable build config.
Each build has a config (custom or inherited from repo or branch).
Each config has a list of steps.
For now, a step can test/run odoo or create a new child build. A python job is
also available.
The mimic the previous behaviour of runbot, a default config is available with
three steps, an install of base, an install+test of all modules, and a last step
for run.
Multibuilds are replace by a config containing cretaion steps.
The created builds are not displayed in main views, but are available
on parent build log page. The result of a parent takes the result of
all children into account.
This new mechanics will help to create some custom behaviours for specifics
use cases, and latter help to parallelise work.
Before this commit, dependencies (i.e. community commit to use when testing enterprise)
were computed at checkout, when the build was going from pending to testing state and
were not stored.
Since the duplicate detection was done at create, the get_closest_branch_name was called
in a loop for each posible duplicate candidate, then a last time at checkout. The main idea of this
pr is to store the build dependecies on build at create, making the duplicate detection
faster (especially when the build name is matching many indirect builds).
The side effect of this change is that the build dependencies won't be affected if a new
commit is pushed between the build creation and the checkout. The build is fully
determined at creation. get_closest_branch is only called once per build
The duplicate detection will also be more precise since we are matching on the commits groups
that were used to run the build, and not only the branch name.
Some work has also been done to rework the closest branch detection in order to manage new corner
cases. Hopefully, everything should work as before (or in a better way).
In a soon future, it will also be possible to use this information to make an "exact rebuild"
or to find corresponding community build.
Pr: #117
Since the runbot_merge module, some branches does not need to be built.
For example the tmp.* branches.
Some other branches does need to be tested but it could be useless to
keep them running. For example the staging branches.
Finally, some builds are generated by server actions during the night.
Those builds does not need to be kept running despite the branch configuration.
For example, the master branch can be configured to create builds with
testing and running but nightly multiple builds can be generated with
testing only.
For that purpose, this commit adds a job_type selection field on the
branch. That way, a branch can be configured by selecting the type of
jobs wanted.
A same kind of job_type was also added on the build that uses the
branch's value if nothing is specified at build creation.
A decorator is used on the job_ methods to specify their job types.
For example, a job method decorated by 'testing' will run if the
branch/build job_type is 'testing' or 'all'.
In the case of PR, the name contains 'refs/pull/3175', the branch_name
contains '3175' and because of the previous fix e095170f8c, the
pull_head_name sontains something like 'blah:12.0-something'.
In that case, the _get_closest_branch_name reaches the fallback.
With this commit, the target_branch_name is stored in a new field and is
used as the fallback.
When searching for a matching PR in a target repo, the match was made on
the 'base ref' which is the base branch that the PR targets.
This means that the second case of the _get_closest_branch method was
never reached.
Example:
An enterprise PR is created that targets Odoo branch 12.0 but with a
matching community PR with a branch with the same name.
The corresponding PR is never found by the runbot because the first
rule match: '12.0' is the base ref of the PR.
This could have been fixed by using the branch pull_head_name field to
build the domain but that leads to a problem with the second rule:
If a PR branch is named 'patch-1', the domain will match each PR with
the same pull_head_name.
With this commit, the pull_head_name field will store the pull head
label. It's not a problem with older PR as a newly created PR in
enterprise will not accidentally match with older ones.
Another issue appeared with github branch naming like 'patch-'.
If someone creates a PR with a branch auto-named 'patch-1',
targeting the community repo and later creates an unrelated PR,
targetting another repo depending of the community, they will be
matched.
To avoid that, the pull head names that ends with 'patch-n' are not
stored. Those pull requests will be built against the target branch
head.
The actual behavior, before this commit, is a blocking point for the
runbot_merge which is based on PR's only. It means that when a community
PR is wrongly matched with with an enterprise PR runbot_merge will not
merge the PR's.
This commit permits to prioritize a branch when scheduling builds.
It's main purpose is for the runbot_merge module. It avoids to have
staging branches as sticky and pollutes the main repo view.
Also, that branch can benefit of the autokill feature when a newer build
is found, freeing ressources for other builds.
Closes#43
When the coverage is activated on a branch, the coverage result is not
stored.
With this commit, the coverage result will be stored on a build.
The last result will be shown on the frontend for sticky branches.
Also, an extra_parameter field is added on the build model.
The previous code of runbot and runbot_cla was made for Odoo API version
8.0. This commit makes it work with Odoo API 11.0 and Python 3.
Also, the present refactoring splits the code into multiple files to
make it easier to read (I hope).
The main change due to Python 3 is the job locking mechanism:
Since PEP-446 file descriptors are non-inheritable by default.
A new method (os.set_inheritable) was introduced to explicitely make
fd inheritable. Also, the close_fds parameter of the subprocess.Popen
method is now True by default.
Finally, PEP-3151 changed the exception raised by fcntl.flock from IOError to OSError
(and IOError became an alias of OSError).
As a consequence of all that, the runbot locking mechanism to check if a
job is finished was not working in python3.
