When disabling tests on runbot by using the test_tags on a build error,
the tests are disabled on every tested version. This can be annoying
when a test only fails from one version up to another.
With this commit, a min and max version can be specified on a
build_error when the test_tags field is used.
If the min and max are not used, the test will be disabled cross
versions.
A common error on runbot is to generate link containing a __init__.py
file
[/some/path/to/__init__.py](/some/path/to/__init__.py)
This would be rendered as
<a href="/some/path/to/<ins>init<ins>.py">/some/path/to/<ins>init<ins>.py</a>
Breaking the link, and the display of the name
By default markdown will not render links avoiding this issue, but it
will remain for the content of the a, needing to manage some kind of
escaping.
The way to escape markdown is to add a \ before any special character
This must be done upront before formating, adding the method
markdown_escape
Our implementation of markdown is not meant to meet the exact
specification of markdown but better suit our needs. One of the
requirements is to be able to use it to format message easily but adding
dynamic countent comming from the outside. One of the error than can
occur is also
'Some code `%s`' % code can also cause problem if code contains `
This issue could be solved using indented code block, but this would
need to complexify the generated string, have a dedicated method to
escape the code blocs, ...
Since we have the controll on the input, we can easily sanitize all
ynamic content to avoid such issues. For code block we introduce a way
to escape backtick (\`). It is non standard but will be easier to use.
Combine with that, the build._log method now allows to add args with the
values to format the string (similar to logging) but will escape
params by default. (cr.execute spirit)
name = '__init__.py'
url = 'path/to/__init__.py'
code = '# comment `for` something'
build._log('f', 'Some message [%s](%s) \n `%s`', name, url, code)
name, url and code will be escaped
The stats are only kept when the build finishes, this makes sence to
avoid collecting stats of a manually killed build (or because of newer
build found), but the stats mays be interresting when the build timeouts.
Adding a manual collection of stats in this case.
The current runbot infrastructure has 100+ machine that will each build
all docker images.
This is unpractical for multiple reasons:
- impotant load on all machine when the build could be done once
- possible differences for the same image depending on the moment the
base was pulled on the host. An extreme example was the version of
python (3.11 or 3.12) when building the noble docker image before it was
stabilized.
- increase the chance to have a docker build failure in case of network
problem. (random)
A centralized registry will help with that, and allow future devlopment
to generate more docker images. All docker images will be exactly the
same, the pull time is faster than build time, only one build per docker
image, ...
When a docker fails to build, the output is logged in the chatter
leading to a lot of noise and a not so readable output. Moreover, the
output tries to format markdown and don't render line break correctly.
This commit proposes to introduce a model to store this output, as well
as some other info like the image identifier, build time, ...
This will help to compare images versions between hosts and should be
useful later to have multiple version of the same image with variant
once the docker registry is introduced.
The batch prepare fill missing won't work if the repo is single version,
in a foreign bundle.
This commit simply adds the branches depending on their version instead
of the bundle version.
When cleaning build errors before fingerprinting, it's only possible to
replace the matching regex with something else but not an empty string.
Since the python 3.11 that may adds lines in error message in order to
visually improve them, the fingerprint of those errors does not match
anymore between different versions.
With this commit, when the replacement string is two consecutive simple
quotes, the matching element is replaced by an empty sting, allowing to
remove unwanted characters.
The order of requirements may have an impact on final outcome.
In documentation builds, we have two requirements with conflicting needs
Lets make the doc requirements install after the odoo ones
(sequence dependant)
There is an issue in unidiff 0.7.3 fixed in 0.7.4
a3faffc54e
This version is not able to parse a diff with removed files.
Since the unidiff packaged version in noble is 0.7.3 patching it looks
like the easiest solution
The MAKE_CELL opcode appeared in python 3.11 and is needed in some
python steps when using closures and generators.
Like:
`(all(s > e for e in [1,2]) for s in [0,1])`
The "Error or traceback found in logs" message is sometimes confusing
since we don't know what is the cause of the issue.
The first idea is to split the two concept, error and traceback to have
a better idea of the cause of the issue
The second one will also log the content of the line in the error
message. For traceback, tries to get the complete traceback, getting all
indentend lines and one last non idented one.
While working on it, cleaning slighty to partially get rid of
returning a dict, artefact from odoo < 13.0
Somme trigger may have an important depth and nightly result can be long
to check.
A custom view was already done for upgrade nightly, but this is hidden
and the same logic could be applied to the distro builds, ...
This commit adds a custom view on the trigger and related controller to
display a custom view for a trigger.
The current logic to define if a build has logs or not is based on
is_docker_step. This related logic is not always valid, since it is
based on step type and step content, but there are many way to have a
result that could be a docker start, one of them being to call another
step. The main issue is that we don't always now if this other step is a
docker step or not before runtime. Moreover the logic becamore more and
more complex adding more conditions like commands, _run_, docker_params,
...
Moreover, the log list is completed before the build start, meaning that
if the build is killed before completion, some logs may be missing but
will be listed. This is also an issue when trying to access logs before
the correspondong step ran.
This commit changes the logic by adding the log file to the list (and
start log) when starting a docker, wich should give a more consistent
result.
The bootstrap version was freezed during a previous migration to avoid
loosing to much time adapting the style again to fit the previous
look and feel.
Anyway, the latest version of bootsrap offers more flexibility about
themes, and it could be a good oportunity to modernise a little the
runbot interface and answer to long lasting requests.
The main part of the adaptation is to tweak colors to match the
previous style, and adapt some class in xml views.
Some css rules are also tweaked to keep the same looks without the need
to rewrite xml views too much, this could be done in a future commit.
The current stratey to check if a build can be killed is to ensure that
no slot unskipped slots points to the same build. Since the check is
only done after a prepare, it should ensure that the new slot have the
build linked before checking if the previous batch can kill it's build.
Since the slot are now filled latter on (after the minimal check) it is
possible that a build is considered killable an killed before being
linked again.
To fix this, we just need to consier params instead of builds to define
if a build is killable or not. If the params of a builds are linked
elsewere, don't kill them.
Previous commit introduced commit tree hash, but only when calling
get_commit_infos. This fixes the get_ref and find_new_commit to have
the same infos.
Right now the commit is considered as a part of the build uniquifier.
If it makes sence for a check on the commit metadata, it is not the case
for execution tests where only the content matters.
This will allow to mark a trigger as non depending on commit but tree
hash, and avoid rebuild when only fixing a commit message.
In order to keep it coherent, the cpu_limit computation can be done in
one place instead of defining it in all _run* methods.
In python steps, it can still be overridden when returning
docker_params.
When build eats all the CPU's resources, it may interfere with other
builds and cause collateral damages.
With this commit, a new settings parameter `Containers CPUs` is added in
order to limit the usage of available CPU's on runbot instances.
If left to 0, no limts are applied.
Otherwise, the cpu_quota docker parameter is computed as Containers
CPU's * (logical cpu's count / nb parallel builds) * cpu period which defaults to 100000.
e.g.:
- on a host with 16 logical CPU's
- with 8 parallel builds allowed
- with Containers CPUs set to 1.5
- with the default cpu_period
cpu_quota will be:
(16/8) * 1.5 * 100000 = 300000
This system parameter can be overridden by the `container_cpus` field on
steps.
When using a repo as a dependency for another trigger, the default
module filter for a repo is not always ideal
As an example, when using odoo as a dependency for another repo,
we may only want to install the module from the new repo.
This iss done right now by creating a custom config but this lead to
duplicates config and steps only to customize the module to install.
This commit proposes a new model to store the filters.
Note that this may be used later as module blacklist on repo too.
Git gc can last a few minutes, it's not a big deal since it's executed
once a day but the transaction is kept idele during this time wich is
not useful. This commit should help to avoid this.
Because of a bad dependency on the compute, the first seen date and last
seen date are not always updated.
e.g.: a new build is scanned and a build is added to a linked error, the
parent error seen dates are not updated.
A test is added to reproduce the case.
When build errors are merged together, the builds of the merged errors
should be moved to the only error that will be kept.
It 's not the case because the merge method is assigning a compute field
and moreover it was hidden in the tests because the compute was not
triggered.
With this commit, the build_error_link is updated to point to the new
error. The test is modified to properly check the case and also to add a
case when the link already exists.
The access rights are updated to allow admin to unlink the
build_error_link records. Otherwise the action could fail when the link
already exists.
When a build that contains the same error that appears two times is
parsed, it crashes because of the unique constraint on build error link.
With this commit, only one link with the same error is created.
Two tests are added for the two cases:
- a new error appearing two times in a same build
- an existing error appearing two times in a same build
Since c6f9d1f0c a new model was added to link build errors and builds.
The _search_version and _search_trigger_ids were not adapted to work
with this new model.
In the build error view, a list of build is displayed with a confusing
create date. The create date in the list is the creation date of the
build, leading to a confusion with the creation of the build log
creation.
With this commit, the real log creation is used in this view.
To achieve that, the many2many relation is extended with a
log_date which is filled when a build log entry is parsed.
When filtering the build error tree view based on the versions equality,
the results may not be what you expect.
e.g.: searching for `versions is equal to 16.0` gives the errors that
appeared in `16.0` (hopefully) but also those which appeared in other
versions too.
With this commit, this search will give the errors that appeared in the
specified version only. When the user wants to list errors that appeared
in `16.0` and other versions too, he has to use the `contains 16.0`
criteria.
When defining cleaning regex, the replacement character is always the
percent sign as it's hard coded in various methods.
With this commit, a replacement string can be defined by cleaning regex
and fallback to the percent sign by default.
With this commit, when build errors are re-cleaned, they are also merged
if the fingerprints when fingerprints are matching.
Also, this fixes the ir_logging compute that associate a build error
so that the active build error is preferred over an inactive one.
Using dev branch from foreign project to fill missing commits looks like
a bad idea, mainly because the lifecycle of the branches is not the same
In some case, it can be useful to allow that to test a branch with a
future change in the base project that will be needed to make the branch
work. As an example introducing a small api change in odoo to make an
override easier, or introducing a module that may be needed to use
the feature.
This commit changes that by allowing to configure on the project or
bundle if we allow to use foreign bundle as reference *before* checking
the base bundle.
A check was add to avoid to wakeup a child if there is a parent database
Most of the time, it was a mistake.
In some case it can be legit, if the parent only creates subbuid without
installing any database.
This commit fixes that by allowing to wake up child if the parent has no
database.
In some circumstances, a commit is created from scratch with only a hash
for sole information. This is not very convenient and can lead to issues
that may be difficult to investigate.
With this commit, when creating this kind of commit object, we try to
get commit informations from git repo.
When exporting a commit, the commit date is used in the `tar` command to
set the date of the exported folder. On the other hand it happens that a
commit is not found in the database and should be quickly created on the
fly. e.g.: with the `_get` method. In this case, if the commit needs to
be exported later, the method fails and may break a runbot build.
It happened with a custom python step.
- clean thread username
- allow to write on params for debug (was mainly usefull to forbid it
at the beginning)
- imrpove some guidelines about method and actions naming/ ordering
- move some code for a cleaner organisation.
- remove some useless request.env.user (not useful anymore)
The runbot settings view is a bit messy and the 16.0 upgrade added mess
on the existing one.
This commit is an attempt to make it a bit clearer and cleaner.
This commit will replace the symlink used for upgrade by the
upgrade-path.
The symlink was used before because old version does not support upgrade
path, but the decision was taken to now limit the testing to version
suporting upgrade paths in order to be able to support utils in another
repository latter.
This reverts commit 9e7441e098.
This doesn't work as expected because of db filter.
Will eb changed latter, reverting for now
Token field is kept, could still be used later.
There are two wizards for the runbot build errors:
- One to close an error with a reason
- One to update the team/user or PR
With this commit, the two wizards are merged into one wizard that helps
to update errors in bulk.
Also, a button is added in the list view that allow to save a mouse
click.
The `NEW` button is removed from the tree view as it should not be of
any use.
Pausing a host can be usefull in some case, mainly when testing new code
The loop will have no effect avoiding to break some build wainting for
testing.
Profile will help to identify potential performance flows during the
loop.
One of the most common custom trigger is to restore a build before
starting some test, either to create a multibuild or make the execution
and debug of some test faster.
It is somethimes tedious to use because we need to give an url of a
build to restore. This build must correspond to the right commits,
must still exixt, ... this means that the dump url must be adapted
everytime a branch is rebased.
The way the dump_url is defined is by going on the last batch, following
the link to the `base_reference_batch_id`, finding a slot corresponding
to the right repo set, (ex: Custom enterprise -> enterprise), and
copying the dump_url in this build.
The base_reference_batch_id is eay to automated but we have to find the
right trigger, this is now a parameter of the custom trigger wizard.
There are actually 2 strategy now to define how to download the dump:
- `url`, using `restore_ dump_url`
- `auto`, using `restore_trigger_id` and `restore_database_suffix`
To ease the setup, a `restore_trigger_id` is added on a trigger, so that
when selecting a trigger, lets say `Custom enterprise`, the defined
`trigger.restore_trigger_id` is automatically chosen for the
`custom_trigger.restore_trigger_id` and the `restore_mode` is setted to
auto.
Two actions are also added to the header of a bundle, a shorcut to
setup a multi build (restore in children) or a restore and test build
(restore in parent).
