Commit Graph

1939 Commits

Author SHA1 Message Date
Christophe Monniez
4a2c9d7491 [FIX] runbot: remove useless fields in test 2024-08-28 10:23:29 +02:00
Christophe Monniez
8ac21fb60b [IMP] runbot: allow to archive dockerfile 2024-08-28 10:23:08 +02:00
Christophe Monniez
5fb0e2f5ab [FIX] runbot: avoid builder crash when no docker to build
When there is no Dockerfile marked as `to_build`, the runbot builder
crashes in loop trying to find the max of an empty sequence.

With this commit, the builder does not even try to build Dockerfile in
that case.
2024-08-26 11:04:54 +02:00
Xavier Morel
ccca46c992 [FIX] runbot_merge: layout backend issues galore
- Odoo 17 seems to not be adjusting `nolabel` fields to be `colspan=2`
  by default, so every such occurrence has to be adjusted by hand or
  it gets squeezed in just the labels column.
- Because of the loss of readonly mode, some fields / setups which
  previously looked ugly during the rare edition (e.g. Pr titles) now
  look ugly all the time. Rework layout and force them to always be
  readonly (hopefully we won't need to edit those).
- This is compounded by unfortunate styling I can't find how to
  override e.g. char fields are 100% width even if readonly.
- `<header>` system requires some workarounds to have the right layout
  and spacing (notably `header` has a bunch of awful rules which we
  need to work around via an interstitial div to set up our own
  flexbox).
2024-08-16 15:12:04 +02:00
Xavier Morel
0d653620c2 [FIX] *: styling
- reduce font-size and bold-ness a hair as it causes issues in the
  backend
- remove font adjustment on root object
- add `text-bg-primary` in the oustanding FP view, apparently BS5 does
  not do anything like that by default when setting `bg-primary` for
  some reason so the current team or user appears in black on dark
  blue leading to sub-par readability
2024-08-16 14:20:39 +02:00
Xavier Morel
7eb25234f5 [FIX] runbot_merge: menus
Turns out having the same ids for the feedback template and feedback
object actions was kinda dumb, who'd have thought?

Split them apart so I can get both objects in my menu...
2024-08-16 14:19:13 +02:00
Xavier Morel
8ad186c204 [FIX] runbot_merge: styling on Odoo 17
Fix a few issues with migrated bootstrap classes (left -> start, right
-> end), revert a bunch of shitty colors from standard, fixup
backgrounds.

Tried to remove the background overrides what with having used
variables but it does completely wrong for info, and I can't be
arsed (also force primary alerts for the same reason, I don't
understand what either bootstrap or standard does and how to override
it properly but it's shit). It'll keep.
2024-08-13 14:39:09 +02:00
Xavier Morel
99ae369d69 [FIX] runbot_merge: update freeze wizard widget for 17.0
It was all borken and the web client couldn't even load.

The logic is basically the same, except the new client doesn't support
forcing onchanges maybe, so we need to `read()` the entire thing if
the form is not modified.
2024-08-13 12:50:13 +02:00
Xavier Morel
36786d51c8 [ADD] runbot_merge: 17.0 migration
`message_main_attachment_id` removed from `mail.thread` in
odoo/odoo@f8c7f2e5bb (16.2), however
precise inheritance tracking was only added in
odoo/odoo@b27eb20a41 (17.1), so
stock migration only handles stock models for this migration.
2024-08-13 08:00:21 +02:00
Xavier Morel
aa1df22657 [MERGE] bot from 16.0 to 17.0
Broken (can't run odoo at all):

- In Odoo 17.0, the `pre_init_hook` takes an env, not a cursor, update
  `_check_citext`.
- Odoo 17.0 rejects `@attrs` and doesn't say where they are or how to
  update them, fun, hunt down `attrs={'invisible': ...` and try to fix
  them.
- Odoo 17.0 warns on non-multi creates, update them, most were very
  reasonable, one very wasn't.

Test failures:

- Odoo 17.0 deprecates `name_get` and doesn't use it as a *source*
  anymore, replace overrides by overrides to `_compute_display_name`.
- Multiple tracking changes:
  - `_track_set_author` takes a `Partner` not an id.
  - `_message_compute_author` still requires overriding in order to
    handle record creation, which in standard doesn't support author
    overriding.
  - `mail.tracking.value.field_type` has been removed, the field type
    now needs to be retrieved from the `field_id`.
  - Some tracking ordering have changed and require adjusting a few
    tests.

Also added a few flushes before SQL queries which are not (obviously
at least) at the start of a cron or controller, no test failure
observed but better safe than sorry (probably).
2024-08-12 13:13:03 +02:00
Xavier-Do
2e2eb4e428 [FIX] runbot: move error message to the end
When building a docker image the error is part of the stream, and
at the end.

The current behaviour will append the error message at the begining of
the "result log" breaking the temporality of the output. Adding it at
the end should be more intuitive to read. This will also help to get a
more  usefull error sumary in some cases.
2024-08-12 09:40:48 +02:00
Xavier Morel
d7a78f89d0 [FIX] conftest: pytest warning
Forgot to declare `defaultstatuses` when it was introduced in
f367a64481. pytest righteously warns
when markers are not declated, so do that (turns out running pytest
with `-We` can be useful, though it also requires `-W
ignore::DeprecationWarning::importlib._boostrap` because reportlab
does weird shit).
2024-08-09 12:51:00 +02:00
Xavier Morel
61b92b2224 [IMP] runbot_merge: use native support for tracking messages
odoo/odoo@1341d52 added native support for tracking / overriding
tracking message so we don't need `change-message `anymore. The calls
had to be modified a bit as `_track_set_log_message` has to be invoked
on the records for which the tracking message is being set / updated.

Sadly the same for authors was only added in 16.3 *and* it's
unsuitable for our needs as we want to set the author without knowing
the scope of affected records (at least in the controller).

That way hopefully in 17.0 we can remove the `_message_compute_author`
override and things should work out. And maybe for 19.0 we can get the
ability to set a per-model (or even global) fallback author into
standard.
2024-08-09 10:10:01 +02:00
Xavier Morel
f378689a0d [FIX] runbot_merge: dumbshit 16.0 view validation complaints 2024-08-09 10:09:50 +02:00
Xavier Morel
a567960d52 [FIX] runbot_merge: 16.0 orm warning
Regular indexing doesn't work if unaccent is enabled, and the ORM
warns about it.
2024-08-09 10:08:36 +02:00
Xavier Morel
5dcdfb1138 [IMP] testing: avoid error on test failure / abort
It's not entirely clear but I assume if / when a test fails or is
aborted (via C-c) pytest kills the stdout somehow. Either way this
causes a bunch of `OSError` to be logged out as we try to write to
closed pipes.

If that occurs, assume the test is gone and just bail out of the
thread.
2024-08-09 10:05:42 +02:00
Xavier-Do
eaadd7886d [FIX] runbot: fix host id 2024-08-08 15:39:13 +02:00
Xavier-Do
917e39dd65 [IMP] runbot: adapt for noble
The migration to the new dockerfile also includes a change to the noble
docker image

This is done in a separate commit to ease the comparaison.

Notes:

The postgresql-client was initially pinned to version 16 and removed
from debian control, but this version proposes add the repositories and
let apt chose the latest version. This is experimental but was tested
on ubuntu:jammy, with 16 version installed as expected.

The requirements are installed as user instead of root to avoid the need
to use the ignore_installed flag (and solve a bunch of issues related
to this flag)
2024-08-08 15:11:34 +02:00
Xavier-Do
d051e68448 [IMP] runbot: official docker file route
This route was already existing on runbot using a website page.
Making it official
2024-08-08 15:11:34 +02:00
Xavier-Do
9c6124d645 [IMP] runbot: faster docker update after change 2024-08-08 15:11:34 +02:00
Xavier-Do
d2872f8e16 [IMP] runbot: rework dockerfile generation 2024-08-08 15:11:34 +02:00
Xavier-Do
9a86941fcc [IMP] runbot: introduce a base test for docker_default content 2024-08-08 15:11:34 +02:00
Xavier-Do
e5ad7c6242 [IMP] runbot: store docker build output in a dedicated model
When a docker fails to build, the output is logged in the chatter
leading to a lot of noise and a not so readable output. Moreover, the
output tries to format markdown and don't render line break correctly.

This commit proposes to introduce a model to store this output, as well
as some other info like the image identifier, build time, ...

This will help to compare images versions between hosts and should be
useful later to have multiple version of the same image with variant
once the docker registry is introduced.
2024-08-08 15:11:34 +02:00
Xavier Morel
d5bda3d3e2 [MERGE] bot from 15.0 to 16.0
Breakages:

- the entire http.py API was updated requiring fixing the uses of
  `request.jsonrequest` and the patches to `WebRequest` to hook in
  sentry
- `fontawesome` was moved
- `*[@groups]` are now completely removed from the view if not
  matching, so any field inside of them which needs to be used outside
  (e.g. attrs) has to be added as invisible outside the element
- discuss removed the mail tracking value helpers from RPC in
  odoo/odoo#88547, so reimplement locally (and better)
2024-08-08 10:37:42 +02:00
Xavier-Do
700c018b33 [FIX] runbot: single version fill missing
The batch prepare fill missing won't work if the repo is single version,
in a foreign bundle.

This commit simply adds the branches depending on their version instead
of the bundle version.
2024-08-06 09:33:22 +02:00
Xavier Morel
0334225114 [FIX] runbot_merge: read sha first in commit statuses broadcast
Because the first operation the notification task performs is updating
the commit, this has to be flushed in order to read-back the commit
hash (probably fixed in later version).

This means if updating the flag triggers a serialization
failure (which happens and is normal) we transition to the `exception`
handler, which *tries to retrieve the sha again* and we get an "in
failed transaction" error on top of the current "serialization
failure", leading to a giant traceback.

Also an unhelpful one since a serialization failure is expected in
this cron.

- Move the reads with no write dependency out of the `try`, there's no
  reason for them to fail, and notably memoize the `sha`.
- Split the handling of serialization failures out of the normal one,
  and just log an `info` (we could actually log nothing, probably).
- Also set the precommit data just before the commit, in case staging
  tracking is ever a thing which could use it.

Fixes #909
2024-08-05 16:16:17 +02:00
Xavier Morel
52f2b1e381 [FIX] runbot_merge: don't show details of merged PR on GH
Detailed statuses are useful in the actual PR dashboard as that allows
direct access to the builds, however in the PR where it's only a
picture it's useless, so fold that information. Also fold it when a PR
is staged.

And while at it add a note / sub-title that the PR is staged.

Fixes #919
2024-08-05 15:18:19 +02:00
Xavier Morel
ff6f046811 [CHG] runbot_merge: deskill mergebot feedback
twas heehee in the moment but it's not really a long term hoohoo
haahaa.
2024-08-05 09:09:23 +02:00
Xavier Morel
ec01523875 [IMP] runbot_merge: prune repo during maintenance
The weekly maintenance would not prune refs. This is not an issue on
odoo/odoo because development branches are in a separate repository,
thus never fetched (we push to them but only using local commits and
remote refs).

However on repos like odoo/documentation the reference and development
branches are collocated, the lack of pruning thus keeps every
development branch alive locally, even years after the branch has been
deleted in the repository.

By pruning remote-tracking refs before GC-ing, we should have cleaner
local clones, and better packing.
2024-08-05 09:03:39 +02:00
Xavier Morel
8131271a9c [FIX] runbot_merge: flaky test
`test_inconsistent_target` was, appropriately, inconsistent, but would
only fail a very small fraction of the time: the issue is that a PR
would switch target between `other` and `master` assuming neither was
an intrinsic blocker *but* the branches are created independently,
just with the same content.

This means if a second ticked over between the creation of the
`master` branch's commit and that of `other`, they would get different
commit hashes (because different timestamp), thus the PR would get 2
commits (or complete nonsense) when targeted to `other`, and the PR
itself would be blocked for lack of a merge method.

The solution is to be slightly less lazy, and create `other` from
`master` instead of copy/pasting the `make_commits` directive. This
means the PR has the exact same number of commits whether targeted to
`master` or `other`, and we now test what we want to test 60 seconds
out of every minute.
2024-08-05 08:58:05 +02:00
Xavier Morel
229ae45e11 [MERGE] *: triggerify mergebot and forwardport crons
A few crons (e.g.database maintenance) remain on timers, but most of
the work crons should be migrated to triggers.

The purpose of this change is mostly to reduce log spam, as crons get
triggered very frequently (most of them every minute) but with little
to do. Secondary purposes are getting experience with cron triggers,
and better sussing out dependencies between actions and crons /
queues.

Fixes #797
2024-08-05 08:54:07 +02:00
Xavier Morel
78cc8835ce [IMP] rubnbot_merge: avoid triggering every cron on every test
Since every cron runs on a fresh database, on the first `run_crons`
every single cron in the db will run even though almost none of them
is relevant.

Aside from the slight inefficiency, this creates unnecessary extra
garbage in the test logs.

By setting the `nextcall` of all crons to infinity in the template we
avoid this issue, only triggered crons (or the crons whose nextcall we
set ourselves) will trigger during calls.

This requires adjusting the branch cleanup cron slightly: it didn't
correctly handle the initial run (`lastcall` being false).
2024-08-05 08:03:56 +02:00
Xavier Morel
157657af49 [REM] *: default_crons fixture
With the trigger-ification pretty much complete the only cron that's
still routinely triggered explicitly is the cross-pr check, and it's
that in all modules, so there's no cause to keep an overridable
fixture.
2024-08-02 15:14:50 +02:00
Xavier Morel
3ee3e9cc81 [IMP] *: trigger-ify staging cron
The staging cron turns out to be pretty reasonable to trigger, as we
already have a handler on the transition of a batch to `not blocked`,
which is exactly when we want to create a staging (that and the
completion of the previous staging).

The batch transition is in a compute which is not awesome, but on the
flip side we also cancel active stagings in that exact scenario (if it
applies), so that matches.

The only real finesse is that one of the tests wants to observe the
instant between the end of a staging (and creation of splits) and the
start of the next one, which because the staging cron is triggered by
the failure of the previous staging is now "atomic", requiring
disabling the staging cron, which means the trigger is skipped
entirely. So this requires triggering the staging cron by hand.
2024-08-02 15:14:50 +02:00
Xavier Morel
f367a64481 [IMP] *: trigger-ify merge cron
The merge cron is the one in charge of checking staging state and
either integrating the staging into the reference branch (if
successful) or cancelling the staging (if failed).

The most obvious trigger for the merge cron is a change in staging
state from the states computation (transition from pending to either
success or failure). Explicitly cancelling / failing a staging marks
it as inactive so the merge cron isn't actually needed.

However an other major trigger is *timeout*, which doesn't have a
trivial signal. Instead, it needs to be hooked on the `timeout_limit`,
and has to be re-triggered at every update to the `timeout_limit`,
which in normal operations is mostly from "pending" statuses bumping
the timeout limit. In that case, `_trigger` to the `timeout_limit` as
that's where / when we expect a status change.

Worst case scenario with this is we have parasitic wakeups of this
cron, but having half a dozen wakeups unnecessary wakeups in an hour
is still probably better than having a wakeup every minute.
2024-08-02 15:14:50 +02:00
Xavier Morel
029957dbeb [IMP] *: trigger-ify task queue type crons
These are pretty simple to convert as they are straightforward: an
item is added  to a work queue (table), then a cron regularly scans
through the table executing the items and deleting them.

That means the cron trigger can just be added on `create` and things
should work out fine.

There's just two wrinkles in the port_forward cron:

- It can be requeued in the future, so needs a conditional trigger-ing
  in `write`.
- It is disabled during freeze (maybe something to change), as a
  result triggers don't enqueue at all, so we need to immediately
  trigger after freeze to force the cron re-enabling it.
2024-08-02 15:14:50 +02:00
Xavier Morel
57a82235d9 [IMP] *: rely only on triggers to run statuses propagation
The cron had been converted to using mostly triggers in
7cd9afe7f2 but I forgot to update
the *tests* to avoid explicitly triggering it.
2024-08-02 12:12:20 +02:00
Christophe Monniez
6284baa1dc [IMP] runbot: allow removal when cleaning build errors
When cleaning build errors before fingerprinting, it's only possible to
replace the matching regex with something else but not an empty string.

Since the python 3.11 that may adds lines in error message in order to
visually improve them, the fingerprint of those errors does not match
anymore between different versions.

With this commit, when the replacement string is two consecutive simple
quotes, the matching element is replaced by an empty sting, allowing to
remove unwanted characters.
2024-08-02 10:32:19 +02:00
Xavier Morel
dd17730f4c [IMP] *: crons tests running for better triggered compatibility
Mergebot / forwardport crons need to run in a specific ordering in
order to flow into one another correctly. The default ordering being
unspecified, it was not possible to use the normal cron
runner (instead of the external driver running crons in sequence one
at a time). This can be fixed by setting *sequences* on crons, as the
cron runner (`_process_jobs`) will use that order to acquire and run
crons.

Also override `_process_jobs` however: the built-in cron runner
fetches a static list of ready crons, then runs that.

This is fine for normal situation where the cron runner runs in a loop
anyway but it's any issue for the tests, as we expect that cron A can
trigger cron B, and we want cron B to run *right now* even if it
hadn't been triggered before cron A ran.

We can replace `_process_job` with a cut down version which does
that (cut down because we don't need most of the error handling /
resilience, there's no concurrent workers, there's no module being
installed, versions must match, ...). This allows e.g. the cron
propagating commit statuses to trigger the staging cron, and both will
run within the same `run_crons` session.

Something I didn't touch is that `_process_jobs` internally creates
completely new environments so there is no way to pass context into
the cron jobs anymore (whereas it works for `method_direct_trigger`),
this means the context values have to be shunted elsewhere for that
purpose which is gross. But even though I'm replacing `_process_jobs`,
this seems a bit too much of a change in cron execution semantics. So
left it out.

While at it tho, silence the spammy `py.warnings` stuff I can't do
much about.
2024-08-02 09:00:34 +02:00
Xavier Morel
32871c3896 [FIX] *: stray prints
Found a bunch of old leftover `print` calls which should not be in the
repo.
2024-08-02 08:59:45 +02:00
Xavier-Do
958d07c57d [FIX] runbot: make requirements order deterministic
The order of requirements may have an impact on final outcome.
In documentation builds, we have two requirements with conflicting needs

Lets make the doc requirements install after the odoo ones
(sequence dependant)
2024-07-30 15:22:59 +02:00
Xavier Morel
48d22b920a [FIX] mergebot coverage
- add context, in case that allows tracking overlap between tests
  eventually
- use `-m` to run odoo, requires backporting `odoo/__main__.py`
  but I'm not quite sure how I was running it previously, possibly
  via odoo-bin? It certainly doesn't seem to work out with a global
  `odoo` helper
2024-07-30 11:04:45 +02:00
Xavier Morel
cabab210de [FIX] *: don't send merge errors to logging
Merge errors are logical failures, not technical, it doesn't make
sense to log them out because there's nothing to be done technically,
a PR having consistency issues or a conflict is "normal". As such
those messages are completely useless and just take unnecessary space
in the logs, making their use more difficult.

Instead of sending them to logging, log staging attempts to the PR
itself, and only do normal logging of the operation as an indicative
item. And remove a bunch of `expect_log_errors` which don't stand
anymore.

While at it, fix a missed issue in forward porting: if the `root.head`
doesn't exist in the repo its `fetch` will immediately fail before
`cat-file` can even run, so the second one is redundant and the first
one needs to be handled properly. Do that. And leave checking
for *that* specific condition as a logged-out error as it should mean
there's something very odd with the repository (how can a pull request
have a head we can't fetch?)
2024-07-26 14:48:59 +02:00
Xavier Morel
6f0aea799f [IMP] *: add test for r- on forward ports
Apparently I'd already fixed that in
286c1fdaee but it has yet to be
deployed.

While at it, add a feedback message to clarify that, unlike `r+`, `r-`
on forward ports does *not* propagate.

Fixes #912
2024-07-26 12:29:52 +02:00
Xavier-Do
5c537f822a [IMP] runbot: add a friendly colorblindness theme
The runbot colors can lack contrast in some case, especially for
colorblinded people.

This commit introduces a theme that should help a little.
Note that this was done without much analysis and should be tweaked in
the future.
2024-07-26 10:53:49 +02:00
Xavier Morel
a0d4bb0d0c [ADD] runbot_merge: expanded view for current batch
The PR dashboard picture provides a great overview of the batch state
both horizontally and vertically *but* apparently people can't for the
life of them go check the actual dashboard when things don't line
up. So expand the "current batch" to a view more similar to
dashboard *page*, which gives details of the sub-checks being
performed and whether they are or are not fulfilled.

Fixes #908
2024-07-26 10:01:33 +02:00
Xavier Morel
f5eb7447fb [IMP] runbot_merge: reorganise composition of PR dashboard pic
The previous version worked but was extremely plodding and
procedural. Initially I wanted to compose the table in a single pass
but that turns out not to really be possible as the goal for #908 is
to have a "drawer" for extended information about the current batch:
this means different cells of the same row can have different heights,
so we can't one-pass the image either vertically (later cells of the
current column might be wider) or horizontally (later cells of the
current row might be taller).

However what can be done is give the entire thing *structure*,
essentially defining a very cut down and ad-hoc layout system before
committing the layout to raster.

This also deduplicates formatting and labelling information which was
previously in both the computation first step and the rasterisation
second step.
2024-07-25 15:34:23 +02:00
Xavier-Do
9d62213abf [FIX] runbot: pin matplotlib 1.22
See 66a1018918 (r144596695)

Looks like 1.21.5 is not available on some distribution

1.22 should be available on most system.

Note that the prefered solution would be to install matplotlib using
apt install python3-matplotlib as stated in the readme.
2024-07-25 08:23:54 +02:00
Xavier Morel
6cc9a6ca11 [IMP] runbot_merge: show batch inconsistency in PR dash picture
Extract current table generation into a separate function, add an
other function to render an alert / list of PR targets if the batch is
not consistent.

This means an extra pass on the table contents to precompute the image
size, but we can delay loading fonts until after etag computation
which might be a bigger gain all things considered: there aren't many
cells in most PR tables, but fonts are rather expensive to
load (I should probably load them at import and cache them in the module...)
2024-07-24 13:27:26 +02:00
Xavier Morel
bca8adbdc4 [IMP] runbot_merge: move inconsistency block higher in batches
Should probably take priority over a PR being misconfigured.
2024-07-24 12:35:21 +02:00