Commit Graph

1259 Commits

Author SHA1 Message Date
Xavier-Do
faef017a25 [FIX] runbot: local cleanup 2024-08-29 08:44:57 +02:00
Christophe Monniez
d722160247 [IMP] drunbot: docker registry
The current runbot infrastructure has 100+ machine that will each build
all docker images.

This is unpractical for multiple reasons:
- impotant load on all machine when the build could be done once
- possible differences for the same image depending on the moment the
base was pulled on the host. An extreme example was the version of
python (3.11 or 3.12) when building the noble docker image before it was
stabilized.
- increase the chance to have a docker build failure in case of network
problem. (random)

A centralized registry will help with that, and allow future devlopment
to generate more docker images. All docker images will be exactly the
same, the pull time is faster than build time, only one build per docker
image, ...
2024-08-29 08:44:57 +02:00
Christophe Monniez
4a2c9d7491 [FIX] runbot: remove useless fields in test 2024-08-28 10:23:29 +02:00
Christophe Monniez
8ac21fb60b [IMP] runbot: allow to archive dockerfile 2024-08-28 10:23:08 +02:00
Xavier-Do
2e2eb4e428 [FIX] runbot: move error message to the end
When building a docker image the error is part of the stream, and
at the end.

The current behaviour will append the error message at the begining of
the "result log" breaking the temporality of the output. Adding it at
the end should be more intuitive to read. This will also help to get a
more  usefull error sumary in some cases.
2024-08-12 09:40:48 +02:00
Xavier-Do
eaadd7886d [FIX] runbot: fix host id 2024-08-08 15:39:13 +02:00
Xavier-Do
917e39dd65 [IMP] runbot: adapt for noble
The migration to the new dockerfile also includes a change to the noble
docker image

This is done in a separate commit to ease the comparaison.

Notes:

The postgresql-client was initially pinned to version 16 and removed
from debian control, but this version proposes add the repositories and
let apt chose the latest version. This is experimental but was tested
on ubuntu:jammy, with 16 version installed as expected.

The requirements are installed as user instead of root to avoid the need
to use the ignore_installed flag (and solve a bunch of issues related
to this flag)
2024-08-08 15:11:34 +02:00
Xavier-Do
d051e68448 [IMP] runbot: official docker file route
This route was already existing on runbot using a website page.
Making it official
2024-08-08 15:11:34 +02:00
Xavier-Do
9c6124d645 [IMP] runbot: faster docker update after change 2024-08-08 15:11:34 +02:00
Xavier-Do
d2872f8e16 [IMP] runbot: rework dockerfile generation 2024-08-08 15:11:34 +02:00
Xavier-Do
9a86941fcc [IMP] runbot: introduce a base test for docker_default content 2024-08-08 15:11:34 +02:00
Xavier-Do
e5ad7c6242 [IMP] runbot: store docker build output in a dedicated model
When a docker fails to build, the output is logged in the chatter
leading to a lot of noise and a not so readable output. Moreover, the
output tries to format markdown and don't render line break correctly.

This commit proposes to introduce a model to store this output, as well
as some other info like the image identifier, build time, ...

This will help to compare images versions between hosts and should be
useful later to have multiple version of the same image with variant
once the docker registry is introduced.
2024-08-08 15:11:34 +02:00
Xavier-Do
700c018b33 [FIX] runbot: single version fill missing
The batch prepare fill missing won't work if the repo is single version,
in a foreign bundle.

This commit simply adds the branches depending on their version instead
of the bundle version.
2024-08-06 09:33:22 +02:00
Christophe Monniez
6284baa1dc [IMP] runbot: allow removal when cleaning build errors
When cleaning build errors before fingerprinting, it's only possible to
replace the matching regex with something else but not an empty string.

Since the python 3.11 that may adds lines in error message in order to
visually improve them, the fingerprint of those errors does not match
anymore between different versions.

With this commit, when the replacement string is two consecutive simple
quotes, the matching element is replaced by an empty sting, allowing to
remove unwanted characters.
2024-08-02 10:32:19 +02:00
Xavier-Do
958d07c57d [FIX] runbot: make requirements order deterministic
The order of requirements may have an impact on final outcome.
In documentation builds, we have two requirements with conflicting needs

Lets make the doc requirements install after the odoo ones
(sequence dependant)
2024-07-30 15:22:59 +02:00
Xavier-Do
5c537f822a [IMP] runbot: add a friendly colorblindness theme
The runbot colors can lack contrast in some case, especially for
colorblinded people.

This commit introduces a theme that should help a little.
Note that this was done without much analysis and should be tweaked in
the future.
2024-07-26 10:53:49 +02:00
Xavier-Do
f66f768024 [FIX] runbot: patch unidiff 0.7.3
There is an issue in unidiff 0.7.3 fixed in 0.7.4
a3faffc54e

This version is not able to parse a diff with removed files.

Since the unidiff packaged version in noble is 0.7.3 patching it looks
like the easiest solution
2024-07-09 08:21:55 +02:00
Christophe Monniez
3a4d432de9 [FIX] runbot: add another missing opcode
The COPY_FREE_VARS code was also added in python 3.11 and is needed by
some python steps.
2024-06-28 23:08:09 +02:00
Christophe Monniez
5740c93e38 [FIX] runbot: add missing opcode
The MAKE_CELL opcode appeared in python 3.11 and is needed in some
python steps when using closures and generators.

Like:
`(all(s > e for e in [1,2]) for s in [0,1])`
2024-06-28 15:27:39 +02:00
Xavier-Do
c562dac84d [FIX] higher sleep values 2024-06-17 12:52:58 +02:00
Xavier-Do
6fca88afa8 [IMP] allow to add single version in foreign trigger dependencies 2024-06-04 13:31:06 +02:00
Christophe Monniez
eded56a4ef [FIX] runbot: adapt docker template for debian control
The Debian control file was changed in odoo/odoo@55849aca in order to
work with Ubuntu Noble. Because of that, it was needed to have a more
robust parsing of the Debian Control file format.
2024-06-04 08:53:18 +02:00
Xavier-Do
8722aba511 [FIX] runbot: report access error on view 2024-05-16 10:04:54 +02:00
Xavier-Do
2a49ac4f8c [FIX] runbot: no result for subcommand 2024-05-15 12:11:24 +02:00
Xavier-Do
d5114c0062 [IMP] runbot: improve make_result
The "Error or traceback found in logs" message is sometimes confusing
since we don't know what is the cause of the issue.

The first idea is to split the two concept, error and traceback to have
a better idea of the cause of the issue

The second one will also log the content of the line in the error
message. For traceback, tries to get the complete traceback, getting all
indentend lines and one last non idented one.

While working on it, cleaning slighty to partially get rid of
returning a dict, artefact from odoo < 13.0
2024-05-14 16:21:15 +02:00
Xavier-Do
507f4e37e3 [IMP] runbot: custom trigger view
Somme trigger may have an important depth and nightly result can be long
to check.

A custom view was already done for upgrade nightly, but this is hidden
and the same logic could be applied to the distro builds, ...

This commit adds a custom view on the trigger and related controller to
display a custom view for a trigger.
2024-05-14 15:26:24 +02:00
Xavier-Do
2a18cd7f3d [FIX] *: remove invalid escape sequences 2024-04-30 16:05:21 +02:00
Xavier-Do
e7d1cc9c57 [FIX] runbot: small fix following BS 5.3 2024-04-23 11:53:46 +02:00
Xavier-Do
816af5a058 [IMP] runbot: better log listing
The current logic to define if a build has logs or not is based on
is_docker_step. This related logic is not always valid, since it is
based on step type and step content, but there are many way to have a
result that could be a docker start, one of them being to call another
step. The main issue is that we don't always now if this other step is a
docker step or not before runtime. Moreover the logic becamore more and
more complex adding more conditions like commands, _run_, docker_params,
...

Moreover, the log list is completed before the build start, meaning that
if the build is killed before completion, some logs may be missing but
will be listed. This is also an issue when trying to access logs before
the correspondong step ran.

This commit changes the logic by adding the log file to the list (and
start log) when starting a docker, wich should give a more consistent
result.
2024-04-23 10:17:34 +02:00
Xavier-Do
a3c85d87d0 [IMP] runbot: improve log message layout 2024-04-23 10:08:54 +02:00
Xavier-Do
c577b72a66 [IMP] runbot: rework preference menu 2024-04-23 10:08:54 +02:00
Xavier-Do
2c5f129969 [IMP] runbot: add dark theme 2024-04-23 10:08:54 +02:00
Xavier-Do
6bafea7c36 [IMP] runbot: adapt to bootstrap 5.3
The bootstrap version was freezed during a previous migration to avoid
loosing to much time adapting the style again to fit the previous
look and feel.

Anyway, the  latest version of bootsrap offers more flexibility about
themes, and it could be a good oportunity to modernise a little the
runbot interface and answer to long lasting requests.

The main part of the adaptation is to tweak colors to match the
previous style, and adapt some class in xml views.

Some css rules are also tweaked to keep the same looks without the need
to rewrite xml views too much, this could be done in a future commit.
2024-04-23 10:08:54 +02:00
Xavier-Do
0eae784a1e [FIX] runbot: don't kill build if build may be used.
The current stratey to check if a build can be killed is to ensure that
no slot unskipped slots points to the same build. Since the check is
only done after a prepare, it should ensure that the new slot have the
build linked before checking if the previous batch can kill it's build.
Since the slot are now filled latter on (after the minimal check) it is
possible that a build is considered killable an killed before being
linked again.

To fix this, we just need to consier params instead of builds to define
if a build is killable or not. If the params of a builds are linked
elsewere, don't kill them.
2024-04-17 16:21:21 +02:00
Xavier-Do
e241f03321 [FIX] runbot: only start triggers based on alive branches 2024-04-17 14:28:53 +02:00
Xavier-Do
9eb7f0d577 [IMP] runbot: add trigger cross dependency 2024-04-16 11:09:50 +02:00
Xavier-Do
86b88d86c8 [FIX] runbot: adapt tests 2024-04-16 08:39:51 +02:00
Xavier-Do
7277d402fb [FIX] runbot: fix typo in view 2024-04-15 23:10:27 +02:00
Xavier-Do
7bc26219c2 [FIX] runbot: add missing tree hash
Previous commit introduced commit tree hash, but only when calling
get_commit_infos. This fixes the get_ref and find_new_commit to have
the same infos.
2024-04-15 16:44:44 +02:00
Xavier-Do
c70aa57acb [FIX] runbot: incorrect label 2024-04-15 14:57:06 +02:00
Xavier-Do
c067384b8b [IMP] runbot: add tree hash to views 2024-04-15 14:41:02 +02:00
Xavier-Do
e521149d93 [IMP] runbot: add container_cpus to step views 2024-04-15 14:32:54 +02:00
Xavier-Do
83acc43a05 [IMP] runbot: save reference batches on base batches 2024-04-15 11:42:28 +02:00
Xavier-Do
c9e14a86ca [IMP] runbot: propagate batch to reference builds decision 2024-04-15 11:42:28 +02:00
Xavier-Do
082ecaafe9 [IMP] runbot: add tree hash info to commit
Right now the commit is considered as a part of the build uniquifier.

If it makes sence for a check on the commit metadata, it is not the case
for execution tests where only the content matters.

This will allow to mark a trigger as non depending on commit but tree
hash, and avoid rebuild when only fixing a commit message.
2024-04-15 10:53:48 +02:00
Xavier-Do
f191e8cc48 [IMP] runbot: optionnal container_name in python steps 2024-04-15 10:44:32 +02:00
Xavier-Do
7e848b8073 [FIX] runbot: fix stats.js 2024-04-15 10:35:20 +02:00
Xavier-Do
9ef850220b [IMP] runbot: add a way to add custom pre/post
Mainly usefull for custom triggers
2024-04-15 10:08:56 +02:00
Christophe Monniez
412baa3fad [REF] runbot: centralize cpu_limit in _run_step
In order to keep it coherent, the cpu_limit computation can be done in
one place instead of defining it in all _run* methods.

In python steps, it can still be overridden when returning
docker_params.
2024-04-15 10:06:28 +02:00
Christophe Monniez
d0a96faf84 [IMP] runbot: add cpus parameter
When build eats all the CPU's resources, it may interfere with other
builds and cause collateral damages.

With this commit, a new settings parameter `Containers CPUs` is added in
order to limit the usage of available CPU's on runbot instances.

If left to 0, no limts are applied.

Otherwise, the cpu_quota docker parameter is computed as Containers
CPU's * (logical cpu's count / nb parallel builds) * cpu period which defaults to 100000.

e.g.:
- on a host with 16 logical CPU's
- with 8 parallel builds allowed
- with Containers CPUs set to 1.5
- with the default cpu_period
cpu_quota will be:
    (16/8) * 1.5 * 100000 = 300000

This system parameter can be overridden by the `container_cpus` field on
steps.
2024-04-15 10:06:28 +02:00
Xavier-Do
0fc1daeac9 [IMP] runbot: enable user to toggle no_build 2024-04-10 16:23:50 +02:00
Xavier-Do
70f4fe23a5 [IMP] runbot: alow autorebase for external pr 2024-04-02 11:57:47 +02:00
Xavier-Do
9c4983f5b7 [IMP] runbot: always display autorebase for external pr 2024-04-02 08:33:36 -01:00
Xavier-Do
70532df2d6 [FIX] runbot: fix build error unlink 2024-03-20 09:50:03 +01:00
Xavier-Do
fd7f49aff8 [FIX] runbot: fix staging creation 2024-03-20 09:47:51 +01:00
Xavier-Do
a00fa04e07 [FIX] runbot: remove first db before ordering 2024-02-23 17:09:08 +01:00
Xavier-Do
9fa53d6581 [FIX] runbot: fix stats page following 17.0 upgrade
Some issue where remaining regarding stats click events and scales.
2024-02-23 11:10:39 +01:00
Xavier-Do
2ba3238a34 [FIX] runbot: fix repo.modules is False case 2024-02-21 11:04:25 +01:00
Xavier-Do
d661fd9fb4 [IMP] runbot: allow to hide project 2024-02-20 16:48:51 +01:00
Xavier-Do
c18bbecf37 [IMP] runbot: allow to customize repo filters from
When using a repo as a dependency for another trigger, the default
module filter for a repo is not always ideal

As an example, when using odoo as a dependency for another repo,
we may only want to install the module from the new repo.

This iss done right now by creating a custom config but this lead to
duplicates config and steps only to customize the module to install.

This commit proposes a new model to store the filters.

Note that this may be used later as module blacklist on repo too.
2024-02-20 16:48:51 +01:00
Christophe Monniez
f032428346 [FIX] runbot: use markup to post message on build errors 2024-02-20 10:22:35 -01:00
Xavier-Do
f7a1a6a11d [IMP] runbot: avoid long idle transaction
Git gc can last a few minutes, it's not a big deal since it's executed
once a day but the transaction is kept idele during this time wich is
not useful. This commit should help to avoid this.
2024-02-19 10:30:33 +01:00
Xavier-Do
60dbbcb72e [IMP] runbot: add total load time on builds 2024-02-19 10:12:28 +01:00
Xavier-Do
336e9525d5 [FIX] runbot: fix check token 2024-02-19 09:59:41 +01:00
Xavier-Do
9cf7a25cca [FIX] runbot: fix params view 2024-02-19 09:59:41 +01:00
Xavier-Do
ecfdb5693d [FIX] runbot: branch computation 2024-02-10 10:17:44 +01:00
Xavier-Do
2fa5a6aee7 [IMP] runbot: add has_pr filter 2024-02-09 16:32:25 +01:00
Xavier-Do
a0dd2fa560 [FIX] runbot: fix search limit 2024-02-09 16:32:25 +01:00
Xavier-Do
a7f6a9b157 [IMP] runbot: allow bundle access per name 2024-02-09 16:32:25 +01:00
Xavier-Do
db2da457ad [FIX] runbot: hide legend in new charjs version
The option namespace for label changed  in version 3.0
See https://www.chartjs.org/docs/3.0.2/configuration/legend.html
2024-02-09 16:32:25 +01:00
Christophe Monniez
36fdec11d5 [FIX] runbot: fix errors first and last seen dates
Because of a bad dependency on the compute, the first seen date and last
seen date are not always updated.

e.g.: a new build is scanned and a build is added to a linked error, the
parent error seen dates are not updated.

A test is added to reproduce the case.
2024-02-09 08:22:28 -01:00
Christophe Monniez
89a279d9dd [REL] 17.0 2024-02-09 08:22:28 -01:00
Xavier-Do
be95c8b364 [FIX] runbot: fix upgrade to 17.0 2024-02-05 10:25:31 +01:00
Christophe Monniez
6dd2580e21 [FIX] runbot: fix various js issues for 17.0 2024-02-05 07:17:31 -01:00
Xavier-Do
8f34312967 [FIX] runbot: adapt js for 17.0 2024-01-30 10:50:58 +01:00
Xavier-Do
3b50f172c0 [FIX] runbot: adapt settings to 17.0 2024-01-30 10:50:58 +01:00
Xavier-Do
a250794432 [FIX] runbot: adapt attrs
remove all attrs in xml views

To help with that, a scripts was written, minimal but sufficent

    #!/usr/bin/python3
    import glob
    import re
    from ast import literal_eval

    def leaf_to_python(leaf):
        if len(leaf) != 3:
            raise ValueError('This script doesnt support leaf', leaf)
        field, operator, value = leaf
        if operator == '=':
            return f'not {field}' if value is False else field if value is True else f'{field} == {value!r}'
        if operator == '!=':
            return f'not {field}' if value is True else field if value is False else f'{field} != {value!r}'
        if operator == 'in':
            return f'{field} in {value!r}'
        if operator == 'not in':
            return f'{field} not in {value!r}'
        raise ValueError('This script doesnt support operator', operator)

    for file in glob.glob('**/*.xml', recursive=True):
        with open(file) as f:
            content = f.read()
            attrs_list = re.findall(r'attrs="{.*}"', content)
            if attrs_list:
                for attrs in attrs_list:
                    match = re.match(r'''attrs="{'(invisible|readonly)': ?(\[.*\])}"''', attrs)
                    attr = match.groups()[0]
                    domain = literal_eval(match.groups()[1])
                    condition = ' and '.join([leaf_to_python(leaf) for leaf in domain])
                    replace = f'{attr}="{condition}"'
                    content = content.replace(attrs, replace)
                with open(file, 'w') as fw:
                    fw.write(content)
2024-01-30 10:50:58 +01:00
Xavier-Do
e83db83533 [REL] runbot: adapt for 17.0 2024-01-30 10:50:58 +01:00
Christophe Monniez
e9fc57816b [FIX] runbot: properly move builds to the merged error
When build errors are merged together, the builds of the merged errors
should be moved to the only error that will be kept.
It 's not the case because the merge method is assigning a compute field
and moreover it was hidden in the tests because the compute was not
triggered.

With this commit, the build_error_link is updated to point to the new
error. The test is modified to properly check the case and also to add a
case when the link already exists.

The access rights are updated to allow admin to unlink the
build_error_link records. Otherwise the action could fail when the link
already exists.
2024-01-30 08:50:12 -01:00
Christophe Monniez
3ff4787788 [FIX] runbot: allow build parsing with same error
When a build that contains the same error that appears two times is
parsed, it crashes because of the unique constraint on build error link.

With this commit, only one link with the same error is created.
Two tests are added for the two cases:
- a new error appearing two times in a same build
- an existing error appearing two times in a same build
2024-01-25 14:31:04 -01:00
Xavier-Do
0b20a834c7 [FIX] runbot: fix typos 2024-01-25 08:44:48 +01:00
Christophe Monniez
b949ea1817 [FIX] runbot: adapt build error search for link model
Since c6f9d1f0c a new model was added to link build errors and builds.
The _search_version and _search_trigger_ids were not adapted to work
with this new model.
2024-01-25 06:41:01 -01:00
Xavier-Do
5406d29f8f [FIX] runbot: fix prepare 2024-01-25 08:40:42 +01:00
Christophe Monniez
c6f9d1f0c5 [IMP] runbot: use real log date on build errors
In the build error view, a list of build is displayed with a confusing
create date. The create date in the list is the creation date of the
build, leading to a confusion with the creation of the build log
creation.

With this commit, the real log creation is used in this view.

To achieve that, the many2many relation is extended with a
log_date which is filled when a build log entry is parsed.
2024-01-24 13:02:22 -01:00
Christophe Monniez
d2bb42264e [IMP] runbot: give more information about known errors 2024-01-24 13:02:22 -01:00
Christophe Monniez
7c0c03753d [IMP] runbot: add warning ribbon when an error is a link
It happens that a user edits or annotates a build error form without
noticing that the error is linked to another one. In that case, the
modification or the note is probably useless. So a warning ribbon should
grab his attention.

While at it, we change the warning ribbon when an error has a test-tag
to not be confused with the link ribbon and also because removing a
test-tag can lead to a real danger (for the mergebot stagings).
2024-01-24 13:02:22 -01:00
Christophe Monniez
ef045c8540 [IMP] runbot: move builds on host page into a tab 2024-01-24 13:02:22 -01:00
Christophe Monniez
05d072d6eb [IMP] runbot: add a frontend button to view batch in backend 2024-01-24 13:02:22 -01:00
Christophe Monniez
c322458c5b [IMP] runbot: add a frontend btn to parse a single log 2024-01-24 13:02:22 -01:00
Christophe Monniez
68892398cd [FIX] runbot: fix error_log parse_log method 2024-01-24 13:02:22 -01:00
Christophe Monniez
c837e7330c [IMP] runbot: remove useless prefix on build errors 2024-01-24 08:54:39 -01:00
Christophe Monniez
6030988560 [IMP] runbot: allow to filter team errors by trigger 2024-01-24 08:54:39 -01:00
Christophe Monniez
bef9dc0e49 [FIX] runbot: properly compute error versions and triggers 2024-01-24 08:54:39 -01:00
Christophe Monniez
f47beb31ba [FIX] runbot: fix the favorites filter bug 2024-01-24 08:54:39 -01:00
Christophe Monniez
2999f92cdb [IMP] runbot: allow to search errors in a single version
When filtering the build error tree view based on the versions equality,
the results may not be what you expect.

e.g.: searching for `versions is equal to 16.0` gives the errors that
appeared in `16.0` (hopefully) but also those which appeared in other
versions too.

With this commit, this search will give the errors that appeared in the
specified version only. When the user wants to list errors that appeared
in `16.0` and other versions too, he has to use the `contains 16.0`
criteria.
2024-01-24 08:54:39 -01:00
Christophe Monniez
08c614a86e [IMP] runbot: add optional fingerprint in error list view 2024-01-24 07:43:33 -01:00
Christophe Monniez
bddac0a645 [IMP] runbot: allow to choose the replacement string
When defining cleaning regex, the replacement character is always the
percent sign as it's hard coded in various methods.

With this commit, a replacement string can be defined by cleaning regex
and fallback to the percent sign by default.
2024-01-24 07:43:33 -01:00
Christophe Monniez
7ce79b434e [IMP] runbot: improve build error cleaning action
With this commit, when build errors are re-cleaned, they are also merged
if the fingerprints when fingerprints are matching.

Also, this fixes the ir_logging compute that associate a build error
so that the active build error is preferred over an inactive one.
2024-01-24 07:43:33 -01:00
Xavier-Do
c513edc6ad [IMP] runbot: allow to use dev foreigh branches
Using dev branch from foreign project to fill missing commits looks like
a bad idea, mainly because the lifecycle of the branches is not the same

In some case, it can be useful to allow that to test a branch with a
future change in the base project that will be needed to make the branch
work. As an example introducing a small api change in odoo to make an
override easier, or introducing a module that may be needed to use
the feature.

This commit changes that by allowing to configure on the project or
bundle if we allow to use foreign bundle as reference *before* checking
the base bundle.
2024-01-24 09:30:10 +01:00
Xavier-Do
3f260ba08f [FIX] runbot: wakeup child without parent db
A check was add to avoid to wakeup a child if there is a parent database
Most of the time, it was a mistake.

In some case it can be legit, if the parent only creates subbuid without
installing any database.
This commit fixes that by allowing to wake up child if the parent has no
database.
2023-12-15 10:41:21 +01:00
Xavier-Do
fe87569044 [FIX] runbot: allow rebuild when killed
The feature was there but broken, lets fix it.
2023-12-05 10:06:18 +01:00
Xavier-Do
bf2528dbd0 [FIX] runbot: avoid non local redirect
The goal of this feature is to redirect to the build
2023-12-05 10:06:18 +01:00
Christophe Monniez
383352d88e [FIX] runbot: fix _get_commit_infos 2023-10-04 14:35:23 +02:00
Christophe Monniez
c2ef253ec9 [IMP] runbot: retrieve commit infos when needed
In some circumstances, a commit is created from scratch with only a hash
for sole information. This is not very convenient and can lead to issues
that may be difficult to investigate.

With this commit, when creating this kind of commit object, we try to
get commit informations from git repo.
2023-10-04 09:00:35 +02:00
Christophe Monniez
b5b90557fd [IMP] runbot: add a backend commit links tree view
It's sometimes usefull to find a commit in the commit links.
2023-10-04 09:00:35 +02:00
Christophe Monniez
929bc33569 [IMP] runbot: allow to call the commit route with a hash
With this commit, it's not possible to go directly to the commit
frontend page with a commit hash with its id or directly with the hash
value.
2023-10-04 09:00:35 +02:00
Christophe Monniez
fef9ae9801 [FIX] runbot: ensure that commit date is always set
When exporting a commit, the commit date is used in the `tar` command to
set the date of the exported folder. On the other hand it happens that a
commit is not found in the database and should be quickly created on the
fly. e.g.: with the `_get` method. In this case, if the commit needs to
be exported later, the method fails and may break a runbot build.

It happened with a custom python step.
2023-10-04 09:00:35 +02:00
Xavier-Do
ce47661bbc [IMP] runbot: allow to define a per host nb run 2023-10-03 10:38:55 +02:00
Xavier-Do
06e088f82e [IMP] runbot: check dir exists upgrade path 2023-09-28 22:55:55 +02:00
Xavier-Do
269e59538e [FIX] runbot: fix wake_up 2023-09-25 11:15:29 +02:00
Xavier-Do
435ac449f5 [FIX] runbot: various fixes and ref
- clean thread username
- allow to write on params for debug (was mainly usefull to forbid it
at the beginning)
- imrpove some guidelines about method and actions naming/ ordering
- move some code for a cleaner organisation.
- remove some useless request.env.user (not useful anymore)
2023-09-25 10:52:16 +02:00
Xavier-Do
8d619d11f8 [FIX] runbot: use upgrade path
Manage multiple upgrade_paths + small fix
2023-09-20 11:40:07 +02:00
Xavier-Do
f627ebaf0e [IMP] runbot: allow pr creation on closed pr
Even if we prefer to avoid this use case, some devs like to use the same
branch for multiple pr. This change allows to reopen a pr once one is
closed.
2023-09-15 13:23:06 +02:00
Christophe Monniez
b018e5a1a4 [IMP] runbot: resize the child extra parameters
When configuring a custom trigger on a bundle by using the wizard, the
child extra params field is often too small to display all the
parameters.
e.g., specify two long test-tags as it's often the case for
multi-builds.

With this commit, the field span over 4 columns.
2023-09-15 11:03:17 +02:00
Christophe Monniez
3c7f59a817 [REF] runbot: update settings view
The runbot settings view is a bit messy and the 16.0 upgrade added mess
on the existing one.

This commit is an attempt to make it a bit clearer and cleaner.
2023-09-15 11:03:17 +02:00
Xavier-Do
9923b18429 [IMP] runbot: use upgrade path
This commit will replace the symlink used for upgrade by the
upgrade-path.

The symlink was used before because old version does not support upgrade
path, but the decision was taken to now limit the testing to version
suporting upgrade paths in order to be able to support utils in another
repository latter.
2023-09-15 10:59:22 +02:00
Christophe Monniez
f50c099d93 [IMP] runbot: avoid modal in build error view
When trying to open a linked error or an error from the history, the
object is opened in a useless modal. With this commit, the object is
opened in a regular form view.
2023-09-06 10:58:21 +02:00
Christophe Monniez
c00579cdf2 [FIX] runbot: fix invalid condition in build template 2023-09-06 10:58:21 +02:00
Xavier-Do
e0795ffaea [IMP] runbot: customizable build access
This reverts commit and fixes some issues.
2023-08-07 15:07:30 +02:00
Xavier-Do
e2408a6869 [IMP] runbot: don't add child log if there are to much children 2023-08-07 15:07:30 +02:00
Xavier-Do
94828f300b [IMP] runbot: scan killed and warn build too 2023-08-07 15:07:30 +02:00
Xavier-Do
b99a1fefc0 [IMP] runbot: allow to define a base_batch manually 2023-08-07 15:07:30 +02:00
Xavier-Do
287be96775 [IMP] runbot: improve user view 2023-08-07 15:07:30 +02:00
Xavier-Do
3a0f045a4b [FIX] runbot: fix _local_pg_dropdb cleanup 2023-07-13 15:30:17 +02:00
Xavier-Do
4a99ee3ce3 [RVE] runbot: revert customizable build_access
This reverts commit 9e7441e098.

This doesn't work as expected because of db filter.
Will eb changed latter, reverting for now

Token field is kept, could still be used later.
2023-07-05 14:54:53 +02:00
Xavier-Do
ea490bffab [REF] runbot: cleanup old dead code 2023-07-04 10:46:41 +02:00
Xavier-Do
0c609ac533 [IMP] runbot: improve cleanup perfs
The current version will read ~100000 build

This one will avoid that by checking the date of the gcstamp instead
2023-07-04 10:46:41 +02:00
Christophe Monniez
7958375fb5 [IMP] runbot: log container short id
When investigating kernel logs e.g.: for finding oom killed containers,
the kernel does not log the name of the incriminated container but only
the id. With this commit the runbot will also log the container short id
which is enough to correlate the logs.
2023-07-04 10:32:54 +02:00
Christophe Monniez
207b890018 [FIX] runbot: frontendurl should use the right id 2023-07-03 16:30:06 +02:00
Xavier-Do
92157ad71a [IMP] runbot: make access to build customizable 2023-07-03 15:52:53 +02:00
Xavier-Do
91470e0cfa [IMP] runbot: add dedicated route for running 2023-07-03 15:52:53 +02:00
Xavier-Do
d09b9961cd [IMP] runbot: log user beside ip 2023-07-03 15:52:53 +02:00
Christophe Monniez
ab194610b0 [IMP] runbot: improve bundle backend page
With this commit trigger customization, branches and last batches are
put in tabs. That way, trigger customization are quickly accessible.
2023-06-30 10:57:01 +02:00
Christophe Monniez
6c41fbd8ae [FIX] runbot: fix build error form disposition
- Base info span over two columns
- Proper fixing info groups
2023-06-30 10:57:01 +02:00
Christophe Monniez
6ed23db655 [FIX] runbot: open frontend url's in a new tab by default 2023-06-30 10:57:01 +02:00
Christophe Monniez
3c8f821695 [IMP] runbot: unify build error wizards
There are two wizards for the runbot build errors:
- One to close an error with a reason
- One to update the team/user or PR

With this commit, the two wizards are merged into one wizard that helps
to update errors in bulk.

Also, a button is added in the list view that allow to save a mouse
click.

The `NEW` button is removed from the tree view as it should not be of
any use.
2023-06-30 10:57:01 +02:00
Christophe Monniez
1cc73a606d [IMP] runbot: allow only advanced users to wake up a child
As it happens that some users are waking up child builds, let's only
allow advanced users to do that.
2023-06-30 10:57:01 +02:00
Christophe Monniez
503afda3b6 [IMP] runbot: add a monthly filter on build errors 2023-06-30 10:57:01 +02:00
Christophe Monniez
041c4f242f [FIX] runbot: allow PR link only when it exists 2023-06-30 10:57:01 +02:00
Xavier-Do
506ff03e07 [IMP] runbot: pause, and profile.
Pausing a host can be usefull in some case, mainly when testing new code
The loop will have no effect avoiding to break some build wainting for
testing.

Profile will help to identify potential performance flows during the
loop.
2023-06-30 10:54:50 +02:00
Christophe Monniez
2b95e0dc01 [FIX] runbot: remove search_count hack
Since odoo/odoo@a01e8b5232 this hack should not be necessary anymore.
2023-06-30 10:37:28 +02:00
Xavier-Do
cd8cc1f77f [FIX] runbot: fix onchange 2023-06-27 14:42:00 +02:00
Xavier-Do
2a003f00de [FIX] runbot: fix dump_db 2023-06-27 14:28:08 +02:00
xdo
24d35988a4
[IMP] runbot: add auto restore for custom trigger (#784)
One of the most common custom trigger is to restore a build before
starting some test, either to create a multibuild or make the execution
and debug of some test faster.

It is somethimes tedious to use because we need to give an url of a
build to restore. This build must correspond to the right commits,
must still exixt, ... this means that the dump url must be adapted
everytime a branch is rebased.

The way the dump_url is defined is by going on the last batch, following
the link to the `base_reference_batch_id`, finding a slot corresponding
to the right repo set, (ex: Custom enterprise -> enterprise), and
copying the dump_url in this build.

The base_reference_batch_id is eay to automated but we have to find the
right trigger, this is now a parameter of the custom trigger wizard.
There are actually 2 strategy now to define how to download the dump:
- `url`, using `restore_ dump_url`
- `auto`,  using `restore_trigger_id` and `restore_database_suffix`

To ease the setup, a `restore_trigger_id` is added on a trigger, so that
when selecting a trigger, lets say `Custom enterprise`, the defined
`trigger.restore_trigger_id` is automatically chosen for the
`custom_trigger.restore_trigger_id` and the `restore_mode` is setted to
auto.

Two actions are also added to the header of a bundle, a shorcut to
setup a multi build (restore in children) or a restore and test build
(restore in parent).
2023-06-27 14:13:12 +02:00
Xavier-Do
465081e9f3 [FIX] runbot: fix jsonb in list view
In view list widget are not always instanciated and a formater is used
instead. This means that the t-esc will try to output a jsonb field
without nowing how to render it, making the page crash.

This is quickly fixed by forcing the widget on the field in tree view.
2023-06-26 13:36:20 +02:00
Xavier-Do
96808acb37 [IMP] runbot: diffable tracking in chatter
The python steps can be long and interresting to track but the change is
actually hard to see since a block of code is logged instead of the diff.

Also, the whitespaces are not preserverd since we are note in a <pre>
block making it hard to read.

This proposes an alternative to display python code tracking values as
a diff with options to copu the raw content of the old and new version.
(as well as showing unchanged lines or not)
2023-06-26 13:11:00 +02:00
Xavier-Do
727d71d103 [IMP] runbot: add an action to check token 2023-06-22 16:43:05 +02:00
Xavier-Do
f82478576d [FIX] runbot: fix bundle view 2023-06-22 16:35:19 +02:00
Christophe Monniez
37babcd526 [FIX] runbot: remove chatter widgets
Since odoo/odoo@3fea5b2136 the widgets specifications in the chatter are
not necessary anymore.
2023-06-22 16:30:03 +02:00
Christophe Monniez
7e92aaeae2 [FIX] runbot: chatter must be after sheet
The `oe_chatter` div must be defined after a sheet, otherwise two
chatters are visible on the form. One bellow and one on the right side.
2023-06-22 16:30:03 +02:00
Xavier-Do
ebb531f4a4 [FIX] runbot: adapt for 16.0 2023-06-22 14:27:32 +02:00
Xavier-Do
6107c0aefb [FIX] runbot: adapt widgets to owl
This is a first quick version to keep main functionalities, this will
need to be adapt and clean later.
2023-06-22 11:15:16 +02:00
Xavier-Do
70c468f821 [FIX] fix invalid t-att 2023-06-21 15:47:54 +02:00
Xavier-Do
614040720b [IMP] runbot: add useful indexes remaining from upgrade 2023-06-21 10:08:14 +02:00
Xavier-Do
8d724b157f [FIX] runbot: adapt frontend to 16.0
A first step to get more independant from web and website was done in
15.0 but some file were moved and it looks like the bootstrap version
changed breaking the frontend again. (4.3->5.1)

This commit simply copies the libs from odoo/15.0 to avoid losing time
fixing the frontend look and feel for the new bootstrap version for now.

A future refactoring could change the vendored version to addapt to 5.1
while modernizing the frontend style.
2023-06-20 14:15:26 +02:00
Xavier-Do
45104b635f [REL] adapt for 16.0 2023-06-20 12:34:32 +02:00
Xavier-Do
d3f998f88c [IMP] runbot: keep log create_date 2023-06-20 09:18:48 +02:00
Xavier-Do
a80dc25699 [FIX] runbot: only top parent is marked killable 2023-06-14 10:30:02 +02:00
Xavier-Do
9cf750119d [IMP] runbot: make search case insensitive 2023-06-02 17:11:54 +02:00
Xavier-Do
03667b703c [IMP] runbot: fallback on python step for upgrade step 2023-06-02 14:44:48 +02:00
Xavier-Do
ad4131789c [FIX] runbot: fix false positive message 2023-06-02 11:10:51 +02:00
Xavier-Do
7845a718b7 [IMP] runbot: add use_ssl to settings 2023-06-02 10:49:05 +02:00
Martin Trigaux
8153fdc4b2 [FIX] runbot: download over https
Firefox blocks downloads from http link if you are on an https page
Allow to deactivate via an ICP in case the runbot is configured over
HTTP (you shouldn't really)
2023-06-02 10:49:05 +02:00
Christophe Monniez
236554b588 [FIX] runbot: catch exceptions during db drop
When the runbot tries to drop a local database, if the that raises an
exception, it goes in a loop failure. It mays happen for example if
someone forgot to close a psql during an investigation :-)

With this commit, the exceptions are catched and at least the database
name is logged.
2023-06-01 16:16:06 +02:00
Xavier-Do
7523dc8000 [FIX] runbot: fix dockerfile choice order
Since all versions will have a defined dockerfile, the project one
will alway be ignored. The idea here is that for a project, we may
definea default dockerfile_id so that we don't have to set it on all
bundle to make it work.
2023-06-01 16:13:40 +02:00
Xavier-Do
db38794f9a [FIX] runbot: avoid sending sattus on running kill
The _kill method was called in multiple case, usually when something
wrong happen:
- exception initiating pending
- kill requested manually
- testing time exceeded
- exception running a job
- ...

But it will also be called when killing a running build.

It was usually not an issue since the status remains the same, but it is
not true if the same commit is used in two build, the new one is green,
the old one is red (enterprise commit remaining the same but community
commit changed as an example)

In this situation, the enterprise commit may receive the red ci from the
old build while the last one is green.

Since with the last version, the github status responsibility is left to
write method, this github status is not useful anymore, updating the
state and result is enough.

This commit also removes the commit since it is not always a god idea.
Most of the time the transaction will be comited quite fast after that
with the new scheduler.

Note that checking in github status if no status has a more recent build
may be a good idea. Only the most recent build using a commit could
sending a status? This would not alway be helpful Imagine a commit used
in 2 branches by mistake, the last build is not always the one we want
(usually fixed by rebuilding a subbuild of the good build)
2023-06-01 15:40:16 +02:00
Xavier-Do
64d3c59ed9 [FIX] runbot: limit max log size
In some case, a build can add a lot of info in a log, there
is already a limit to the number of entry but not to the size of an
entry. This will limit the database usage in case of mistake/abuse.
2023-06-01 15:33:47 +02:00
Xavier-Do
5a5e7693d4 [IMP] runbot: add an option on step to disable logdb 2023-06-01 15:13:01 +02:00
Christophe Monniez
f6eb23f896 [IMP] runbot: improve frontend search
When filtering bundles in the frontend, the user is not able to search
for its final trigram because of the `like`search.

With this commit, if the search contains a `%` symbol, the `=like`
operator is used permitting more accurate searches.
2023-06-01 15:03:13 +02:00
Christophe Monniez
2421a24f78 [IMP] runbot: add show builds the host form 2023-06-01 15:03:13 +02:00
Christophe Monniez
45fb4f8319 [IMP] runbot: add activities on runbot build errors 2023-06-01 15:03:13 +02:00
Christophe Monniez
86616ba88e [IMP] runbot: add a widget to go to runbot frontend
With this commit, a custom widget is added to go to the reunbot frontend
from a Char field. This allows to go from the bundle backend page to the
bundle frontend page wich is more useful in some situations.

e.g.: when creating a custom trigger with the wizard, this allows to
test the trigger with 2 clicks.
2023-06-01 15:03:13 +02:00
Christophe Monniez
e445ed27db [IMP] runbot: change fixing commit widget to url 2023-06-01 15:03:13 +02:00
Christophe Monniez
bdd98b07ec [IMP] runbot: rename active to active
Error is not fixed was too disturbing.
2023-06-01 15:03:13 +02:00
Christophe Monniez
6e145ff362 [IMP] runbot: improve build errors tabs
* show only the all builds tab
* hide linked errors tab when there is no linked errors
* hide error history tab when there is no history
* add some readonly
2023-06-01 15:03:13 +02:00
Christophe Monniez
ffe12182ab [IMP] runbot: open frontend_url in a new tab 2023-06-01 15:03:13 +02:00
Christophe Monniez
430a526b5c [FIX] runbot: limit triggers in additionnal_setup
When testing with a populated db, this assertion fails because of
additional repositories.
2023-06-01 15:03:13 +02:00
Christophe Monniez
82c6b22e77 [FIX] runbot: typo in test tag 2023-06-01 15:03:13 +02:00
Christophe Monniez
2e002c2dd7 [IMP] runbot: improve the build errors wizard
With this commit, the wizard now allows to set a fixing PR and/or a
commit text to multiple errors.
2023-06-01 15:03:13 +02:00
Christophe Monniez
388eeb377a [IMP] runbot: improve build errors views
* add a link to the fixing PR on github
* add a warning ribbon on test-tagged errors
* show different colors in tree view to spot fixed PR's
* add some search filters
2023-06-01 15:03:13 +02:00
Xavier-Do
231febab96 [FIX] runbot: fix stats.js 2023-05-08 13:26:07 +02:00
Xavier-Do
04760491a2 [FIX] runbot: hide wakeup action from public users
The initial idea to have a wakeup for public users stopped being viable
due to some abuse of the system, maybe unintentional crawling of
some build page but still, this feature will now be limited to internal
users only.
2023-05-02 13:32:43 +02:00
Xavier-Do
e3d87b5b5d [IMP] runbot: improve test-tags support
The current post_install build mecanism is using extra params to give
test-tags. Unfortunately this disables the support for auto tags
and this have to be done manually. This means that auto tags are in the
build extra-params and not dynamic at rebuild of a post_install.

Also, using extraparams in the post install creation was removing
extra_params comming from custom trigger.

With this commit, the test-tags can be given inside config_data
and will be combined with config step test-tags and auto-tags.

This was an opportunity to simplify the logic.

This commit also fixes the test_install_tags that was broken.
2023-04-06 11:33:01 +02:00
Xavier-Do
43d5cc9d7e [IMP] runbot: add some anchors in nginx 2023-04-04 10:43:58 +02:00
Xavier-Do
e88f679c87 [FIX] runbot: fix python step wakeup 2023-03-30 13:05:35 +02:00
Xavier-Do
8c2e7a5781 [FIX] runbot: manage case when there is no start
SInce the previous version the build end is written when going in any
done state. This means that when a build is skipped, it has a end
but no start.

Adapat the build dime to manage this use case.
2023-03-29 14:58:14 +02:00
Xavier-Do
9024594df2 [FIX] runbot: use request instead of self 2023-03-24 17:06:29 +01:00
Xavier-Do
f7d29f87a4 [FIX] runbot: fix build end
Previous fix was not enough because based on global_state, meaning
that the build if the build goes waiting, it need to update its end
anyway.
2023-03-24 13:09:19 +01:00
Xavier-Do
0edc0bce3a [FIX] limit /force route to advanced users
The force buttons were hidden because unfortunately miss used as a
rebuild in some case instead. The position of the button was to obvious
and used as a "magic fix" when the intended behavior was only for really
specific cases.

Unfortunately the routes were know and still used manually. This commit
blocs the access giving a message to ask for the group if needed.

Those feature would benefit for some documentation.
2023-03-24 11:30:13 +01:00
Xavier-Do
30c74e2434 [FIX] runbot: update build end 2023-03-24 10:26:20 +01:00
Xavier-Do
ee58a93e9a [IMP] runbot: avoid link to killed build
When a build is created, it will first check for another
build having the same params. It is usually a good idea to avoid
to much load. In some case, a build can be found, but a killed one.

This is not what we want:
The first scenario is to consecutive force push,
commit1 -> commit2 -> commit1

The build of commit1 may be killed because of commit2, then when
forcepushing commit1 again, it will be linked to a killed build.

A even more problematic problem was discovered because of a delay In
odoo/odoo repo hook. An odoo-dev/odoo 16.0-... branch was discovered
first using this commit, and a build was created.
Then, the branch was forcedpushed and the build was killed.
Finally, the 16.0 commit was discovered, and was linked to the killed
build. This was mainly an issue because the build was a template.

With this changes, the 16.0 would have created a new build, not linking
to a killed one.

Note that linking to a red build is not an error. Only a killed one.
2023-03-24 10:26:20 +01:00
Xavier-Do
3e5d5e88a1 [FIX] runbot: global_state not written in compute 2023-03-23 16:56:34 +01:00
Xavier-Do
0d29643d52 [IMP] runbot: add a separate pending count
The assigned build are in the same count of the pending build. This can
sometimes create a false queue, because you can have 1000 pending builds
on one host, this doesn't mean that a new standard build cannot be
immediatly taken by another host. This is mainly to hide the false queue
created by the full charge zfs build currently running and creating
~400 assigned build.
2023-03-23 16:33:24 +01:00
Xavier-Do
74f0c8e1ad [FIX] runbot: fix wakeup 2023-03-23 14:44:08 +01:00
Xavier-Do
2acca91a97 [FIX] runbot: fix missing return
This was missing to make python (and other non docker) steps really
blazing fast
2023-03-23 14:17:02 +01:00
Xavier-Do
b078275c94 [IMP] runbot: add commit_export_ids 2023-03-23 13:32:50 +01:00
Xavier-Do
2488228b7b [IMP] runbot: make params work in create multi
The main motivation is to allow to create params from data
the "new" method was called with a value list instead of a dict.

Also, makes it possible to update params when the registry is not laoded
2023-03-23 13:32:50 +01:00
Xavier-Do
d164d9a745 [IMP] runbot: host improvement 2023-03-23 13:32:50 +01:00
Xavier-Do
d2f9330043 [REF] runbot: various refactoring
The initial motivation is to remove the flush when a log_counter is
written. This flush was initially usefull when the limit was in a
psql trigger, but finally add a side effect to flush everything before
starting the docker. This was limiting concurrent update after starting
the docker, but we still have no garantee that the transaction is
commited after starting the docker. The use case where the docker is
started but the transaction is not commited was not handled well and was
leading to an infinite loop of trying to start a docker (while the
docker was already started)

This refactoring returns the docker to the scheduler so that the
schedulter can commit before starting the docker.

To achieve this, it is ideal to have only one method that could return
a callable in the _scheduler loop. This is done by removing the run_job
from the init_pending method. All satellite method like make result
are also modified and adapted to make direct write: the old way was
technical debt, useless optimization from pre-v13.

Other piece of code are moved arround to prepare for future changes,
mainly to make the last commit easier to revert if needed.

[FIX] runbot: adapt tests to previous refactoring
2023-03-23 13:32:50 +01:00
Xavier-Do
688900edb1 [FIX] runbot: remove hardcoded runbot_logs 2023-03-23 13:32:50 +01:00