Commit Graph

1628 Commits

Author SHA1 Message Date
Xavier-Do
9c6124d645 [IMP] runbot: faster docker update after change 2024-08-08 15:11:34 +02:00
Xavier-Do
d2872f8e16 [IMP] runbot: rework dockerfile generation 2024-08-08 15:11:34 +02:00
Xavier-Do
9a86941fcc [IMP] runbot: introduce a base test for docker_default content 2024-08-08 15:11:34 +02:00
Xavier-Do
e5ad7c6242 [IMP] runbot: store docker build output in a dedicated model
When a docker fails to build, the output is logged in the chatter
leading to a lot of noise and a not so readable output. Moreover, the
output tries to format markdown and don't render line break correctly.

This commit proposes to introduce a model to store this output, as well
as some other info like the image identifier, build time, ...

This will help to compare images versions between hosts and should be
useful later to have multiple version of the same image with variant
once the docker registry is introduced.
2024-08-08 15:11:34 +02:00
Xavier-Do
700c018b33 [FIX] runbot: single version fill missing
The batch prepare fill missing won't work if the repo is single version,
in a foreign bundle.

This commit simply adds the branches depending on their version instead
of the bundle version.
2024-08-06 09:33:22 +02:00
Christophe Monniez
6284baa1dc [IMP] runbot: allow removal when cleaning build errors
When cleaning build errors before fingerprinting, it's only possible to
replace the matching regex with something else but not an empty string.

Since the python 3.11 that may adds lines in error message in order to
visually improve them, the fingerprint of those errors does not match
anymore between different versions.

With this commit, when the replacement string is two consecutive simple
quotes, the matching element is replaced by an empty sting, allowing to
remove unwanted characters.
2024-08-02 10:32:19 +02:00
Xavier-Do
958d07c57d [FIX] runbot: make requirements order deterministic
The order of requirements may have an impact on final outcome.
In documentation builds, we have two requirements with conflicting needs

Lets make the doc requirements install after the odoo ones
(sequence dependant)
2024-07-30 15:22:59 +02:00
Xavier-Do
5c537f822a [IMP] runbot: add a friendly colorblindness theme
The runbot colors can lack contrast in some case, especially for
colorblinded people.

This commit introduces a theme that should help a little.
Note that this was done without much analysis and should be tweaked in
the future.
2024-07-26 10:53:49 +02:00
Xavier-Do
9d62213abf [FIX] runbot: pin matplotlib 1.22
See 66a1018918 (r144596695)

Looks like 1.21.5 is not available on some distribution

1.22 should be available on most system.

Note that the prefered solution would be to install matplotlib using
apt install python3-matplotlib as stated in the readme.
2024-07-25 08:23:54 +02:00
Xavier-Do
66a1018918 [FIX] runbot: mathplotlib packaging
A new version of numpy (2.0.0) was released in june 2024

Since matplotib requires a version of numpy >= 1.17, this new version
is installed instead of the 1.x.

Pinning the numpy version to the expected one in ubuntu jammy should
solve the issue.

Failed to initialize database `65188118-17-0-runbot`.
Traceback (most recent call last):
  File "/data/build/odoo/odoo/service/server.py", line 1313, in preload_registries
    registry = Registry.new(dbname, update_module=update_module)
  File "<decorator-gen-16>", line 2, in new
  File "/data/build/odoo/odoo/tools/func.py", line 87, in locked
    return func(inst, *args, **kwargs)
  File "/data/build/odoo/odoo/modules/registry.py", line 113, in new
    odoo.modules.load_modules(registry, force_demo, status, update_module)
  File "/data/build/odoo/odoo/modules/loading.py", line 480, in load_modules
    processed_modules += load_marked_modules(env, graph,
  File "/data/build/odoo/odoo/modules/loading.py", line 364, in load_marked_modules
    loaded, processed = load_module_graph(
  File "/data/build/odoo/odoo/modules/loading.py", line 185, in load_module_graph
    load_openerp_module(package.name)
  File "/data/build/odoo/odoo/modules/module.py", line 395, in load_openerp_module
    __import__(qualname)
  File "/data/build/runbot/runbot/__init__.py", line 3, in <module>
    from . import controllers
  File "/data/build/runbot/runbot/controllers/__init__.py", line 5, in <module>
    from . import badge
  File "/data/build/runbot/runbot/controllers/badge.py", line 5, in <module>
    from matplotlib.font_manager import FontProperties
  File "/home/odoo/.local/lib/python3.10/site-packages/matplotlib/__init__.py", line 109, in <module>
    from . import _api, _version, cbook, docstring, rcsetup
  File "/home/odoo/.local/lib/python3.10/site-packages/matplotlib/rcsetup.py", line 27, in <module>
    from matplotlib.colors import Colormap, is_color_like
  File "/home/odoo/.local/lib/python3.10/site-packages/matplotlib/colors.py", line 56, in <module>
    from matplotlib import _api, cbook, scale
  File "/home/odoo/.local/lib/python3.10/site-packages/matplotlib/scale.py", line 23, in <module>
    from matplotlib.ticker import (
  File "/home/odoo/.local/lib/python3.10/site-packages/matplotlib/ticker.py", line 136, in <module>
    from matplotlib import transforms as mtransforms
  File "/home/odoo/.local/lib/python3.10/site-packages/matplotlib/transforms.py", line 46, in <module>
    from matplotlib._path import (
ImportError: numpy.core.multiarray failed to import
2024-07-24 10:41:45 +02:00
Xavier-Do
f66f768024 [FIX] runbot: patch unidiff 0.7.3
There is an issue in unidiff 0.7.3 fixed in 0.7.4
a3faffc54e

This version is not able to parse a diff with removed files.

Since the unidiff packaged version in noble is 0.7.3 patching it looks
like the easiest solution
2024-07-09 08:21:55 +02:00
Christophe Monniez
3a4d432de9 [FIX] runbot: add another missing opcode
The COPY_FREE_VARS code was also added in python 3.11 and is needed by
some python steps.
2024-06-28 23:08:09 +02:00
Christophe Monniez
5740c93e38 [FIX] runbot: add missing opcode
The MAKE_CELL opcode appeared in python 3.11 and is needed in some
python steps when using closures and generators.

Like:
`(all(s > e for e in [1,2]) for s in [0,1])`
2024-06-28 15:27:39 +02:00
Xavier-Do
c562dac84d [FIX] higher sleep values 2024-06-17 12:52:58 +02:00
Xavier-Do
6fca88afa8 [IMP] allow to add single version in foreign trigger dependencies 2024-06-04 13:31:06 +02:00
Christophe Monniez
eded56a4ef [FIX] runbot: adapt docker template for debian control
The Debian control file was changed in odoo/odoo@55849aca in order to
work with Ubuntu Noble. Because of that, it was needed to have a more
robust parsing of the Debian Control file format.
2024-06-04 08:53:18 +02:00
Xavier-Do
8722aba511 [FIX] runbot: report access error on view 2024-05-16 10:04:54 +02:00
Xavier-Do
2a49ac4f8c [FIX] runbot: no result for subcommand 2024-05-15 12:11:24 +02:00
Xavier-Do
d5114c0062 [IMP] runbot: improve make_result
The "Error or traceback found in logs" message is sometimes confusing
since we don't know what is the cause of the issue.

The first idea is to split the two concept, error and traceback to have
a better idea of the cause of the issue

The second one will also log the content of the line in the error
message. For traceback, tries to get the complete traceback, getting all
indentend lines and one last non idented one.

While working on it, cleaning slighty to partially get rid of
returning a dict, artefact from odoo < 13.0
2024-05-14 16:21:15 +02:00
Xavier-Do
507f4e37e3 [IMP] runbot: custom trigger view
Somme trigger may have an important depth and nightly result can be long
to check.

A custom view was already done for upgrade nightly, but this is hidden
and the same logic could be applied to the distro builds, ...

This commit adds a custom view on the trigger and related controller to
display a custom view for a trigger.
2024-05-14 15:26:24 +02:00
Xavier-Do
2a18cd7f3d [FIX] *: remove invalid escape sequences 2024-04-30 16:05:21 +02:00
Xavier-Do
e7d1cc9c57 [FIX] runbot: small fix following BS 5.3 2024-04-23 11:53:46 +02:00
Xavier-Do
816af5a058 [IMP] runbot: better log listing
The current logic to define if a build has logs or not is based on
is_docker_step. This related logic is not always valid, since it is
based on step type and step content, but there are many way to have a
result that could be a docker start, one of them being to call another
step. The main issue is that we don't always now if this other step is a
docker step or not before runtime. Moreover the logic becamore more and
more complex adding more conditions like commands, _run_, docker_params,
...

Moreover, the log list is completed before the build start, meaning that
if the build is killed before completion, some logs may be missing but
will be listed. This is also an issue when trying to access logs before
the correspondong step ran.

This commit changes the logic by adding the log file to the list (and
start log) when starting a docker, wich should give a more consistent
result.
2024-04-23 10:17:34 +02:00
Xavier-Do
a3c85d87d0 [IMP] runbot: improve log message layout 2024-04-23 10:08:54 +02:00
Xavier-Do
c577b72a66 [IMP] runbot: rework preference menu 2024-04-23 10:08:54 +02:00
Xavier-Do
2c5f129969 [IMP] runbot: add dark theme 2024-04-23 10:08:54 +02:00
Xavier-Do
6bafea7c36 [IMP] runbot: adapt to bootstrap 5.3
The bootstrap version was freezed during a previous migration to avoid
loosing to much time adapting the style again to fit the previous
look and feel.

Anyway, the  latest version of bootsrap offers more flexibility about
themes, and it could be a good oportunity to modernise a little the
runbot interface and answer to long lasting requests.

The main part of the adaptation is to tweak colors to match the
previous style, and adapt some class in xml views.

Some css rules are also tweaked to keep the same looks without the need
to rewrite xml views too much, this could be done in a future commit.
2024-04-23 10:08:54 +02:00
Xavier-Do
0eae784a1e [FIX] runbot: don't kill build if build may be used.
The current stratey to check if a build can be killed is to ensure that
no slot unskipped slots points to the same build. Since the check is
only done after a prepare, it should ensure that the new slot have the
build linked before checking if the previous batch can kill it's build.
Since the slot are now filled latter on (after the minimal check) it is
possible that a build is considered killable an killed before being
linked again.

To fix this, we just need to consier params instead of builds to define
if a build is killable or not. If the params of a builds are linked
elsewere, don't kill them.
2024-04-17 16:21:21 +02:00
Xavier-Do
e241f03321 [FIX] runbot: only start triggers based on alive branches 2024-04-17 14:28:53 +02:00
Xavier-Do
9eb7f0d577 [IMP] runbot: add trigger cross dependency 2024-04-16 11:09:50 +02:00
Xavier-Do
86b88d86c8 [FIX] runbot: adapt tests 2024-04-16 08:39:51 +02:00
Xavier-Do
7277d402fb [FIX] runbot: fix typo in view 2024-04-15 23:10:27 +02:00
Xavier-Do
7bc26219c2 [FIX] runbot: add missing tree hash
Previous commit introduced commit tree hash, but only when calling
get_commit_infos. This fixes the get_ref and find_new_commit to have
the same infos.
2024-04-15 16:44:44 +02:00
Xavier-Do
c70aa57acb [FIX] runbot: incorrect label 2024-04-15 14:57:06 +02:00
Xavier-Do
c067384b8b [IMP] runbot: add tree hash to views 2024-04-15 14:41:02 +02:00
Xavier-Do
e521149d93 [IMP] runbot: add container_cpus to step views 2024-04-15 14:32:54 +02:00
Xavier-Do
83acc43a05 [IMP] runbot: save reference batches on base batches 2024-04-15 11:42:28 +02:00
Xavier-Do
c9e14a86ca [IMP] runbot: propagate batch to reference builds decision 2024-04-15 11:42:28 +02:00
Xavier-Do
082ecaafe9 [IMP] runbot: add tree hash info to commit
Right now the commit is considered as a part of the build uniquifier.

If it makes sence for a check on the commit metadata, it is not the case
for execution tests where only the content matters.

This will allow to mark a trigger as non depending on commit but tree
hash, and avoid rebuild when only fixing a commit message.
2024-04-15 10:53:48 +02:00
Xavier-Do
f191e8cc48 [IMP] runbot: optionnal container_name in python steps 2024-04-15 10:44:32 +02:00
Xavier-Do
7e848b8073 [FIX] runbot: fix stats.js 2024-04-15 10:35:20 +02:00
Xavier-Do
9ef850220b [IMP] runbot: add a way to add custom pre/post
Mainly usefull for custom triggers
2024-04-15 10:08:56 +02:00
Christophe Monniez
412baa3fad [REF] runbot: centralize cpu_limit in _run_step
In order to keep it coherent, the cpu_limit computation can be done in
one place instead of defining it in all _run* methods.

In python steps, it can still be overridden when returning
docker_params.
2024-04-15 10:06:28 +02:00
Christophe Monniez
d0a96faf84 [IMP] runbot: add cpus parameter
When build eats all the CPU's resources, it may interfere with other
builds and cause collateral damages.

With this commit, a new settings parameter `Containers CPUs` is added in
order to limit the usage of available CPU's on runbot instances.

If left to 0, no limts are applied.

Otherwise, the cpu_quota docker parameter is computed as Containers
CPU's * (logical cpu's count / nb parallel builds) * cpu period which defaults to 100000.

e.g.:
- on a host with 16 logical CPU's
- with 8 parallel builds allowed
- with Containers CPUs set to 1.5
- with the default cpu_period
cpu_quota will be:
    (16/8) * 1.5 * 100000 = 300000

This system parameter can be overridden by the `container_cpus` field on
steps.
2024-04-15 10:06:28 +02:00
Xavier-Do
0fc1daeac9 [IMP] runbot: enable user to toggle no_build 2024-04-10 16:23:50 +02:00
Xavier-Do
70f4fe23a5 [IMP] runbot: alow autorebase for external pr 2024-04-02 11:57:47 +02:00
Xavier-Do
9c4983f5b7 [IMP] runbot: always display autorebase for external pr 2024-04-02 08:33:36 -01:00
Xavier-Do
70532df2d6 [FIX] runbot: fix build error unlink 2024-03-20 09:50:03 +01:00
Xavier-Do
fd7f49aff8 [FIX] runbot: fix staging creation 2024-03-20 09:47:51 +01:00
Xavier-Do
a00fa04e07 [FIX] runbot: remove first db before ordering 2024-02-23 17:09:08 +01:00