Commit Graph

1884 Commits

Author SHA1 Message Date
Xavier-Do
94828f300b [IMP] runbot: scan killed and warn build too 2023-08-07 15:07:30 +02:00
Xavier-Do
b99a1fefc0 [IMP] runbot: allow to define a base_batch manually 2023-08-07 15:07:30 +02:00
Xavier-Do
287be96775 [IMP] runbot: improve user view 2023-08-07 15:07:30 +02:00
Xavier-Do
3a0f045a4b [FIX] runbot: fix _local_pg_dropdb cleanup 2023-07-13 15:30:17 +02:00
Xavier Morel
a692163f6e [IMP] runbot_merge: add quick jump from stagings to PRs
In the backend, the intermediate jump through batches is really not
convenient (even if we kinda have to jump through batches *anyway*).

Fixes #751
2023-07-10 15:23:31 +02:00
Xavier Morel
780e20bfd6 [IMP] runbot_merge: filtering options and UX on stagings list
Allow filtering stagings by state (success or failure), and provide a
control to explicitly update the staging date limit.

Should make it easier to drill through stagings when looking for
specific information.

Related to #751
2023-07-10 15:23:31 +02:00
Xavier Morel
5bce73c97d [IMP] *: optimise loading of home page
Fix outstanding query to make a positive `state` filtering, instead of
negative, matching 3b52b1aace8674259812a76b1566260937dbcacb.

Also manually create a map of stagings (grouped by branch) sharing a
single prefetch set.

For odoo the mergebot home page has 12 branches in the odoo project
and 8 in spreadsheet, 6 stagings each. This means 120 queries to
retrieve all the heads (Odoo stagings have 5 heads and spreadsheet
have 1, but that seems immaterial).

By fixing `_compute_statuses` and creating a single prefetch set for
all stagings of all branches we can fetch all the commits in a single
query instead of 120.
2023-07-10 15:23:31 +02:00
Xavier Morel
81ce4ea02b [IMP] rewrite /forwardport/outstanding
- add support for authorship (not just approval)
- make display counts directly
- fix `state` filter: postgres can't do negative index lookups
- add indexes for author and reviewed_by as we look them up
- ensure we handle the entire source filtering via a single subquery

Closes #778
2023-07-10 15:23:31 +02:00
Xavier Morel
aefbdaf974 [IMP] *: client side sorted implementation
Allow sorting by a callback. Sort remains client-side.
2023-07-10 15:23:31 +02:00
Xavier Morel
d748d4b215 [IMP] *: create a single template db per module to test
Before this, when testing in parallel (using xdist) each worker would
create its own template database (per module, so 2) then would copy
the database for each test.

This is pretty inefficient as the init of a db is quite expensive in
CPU, and when increasing the number of workers (as the test suite is
rather IO bound) this would trigger a stampede as every worker would
try to create a template at the start of the test suite, leading to
extremely high loads and degraded host performances (e.g. 16 workers
would cause a load of 20 on a 4 cores 8 thread machine, which makes
its use difficult).

Instead we can have a lockfile at a known location of the filesystem,
the first worker to need a template for a module install locks it,
creates the templates, then writes the template's name to the
lockfile.

Every other worker can then lock the lockfile and read the name out,
using the db for duplication.

Note: needs to use `os.open` because the modes of `open` apparently
can't express "open at offset 0 for reading or create for writing",
`r+` refuses to create the file, `w+` still truncates, and `a+` is
undocumented and might not allow seeking back to the start on all
systems so better avoid it.

The implementation would be simplified by using `lockfile` but that's
an additional dependency plus it's deprecated. It recommends
`fasteners` but that seems to suck (not clear if storing stuff in the
lockfile is supported, it opens the lockfile in append mode). Here the
lockfiles are sufficient to do the entire thing.

Conveniently, this turns out to improve *both* walltime CPU time
compared to the original version, likely because while workers now
have to wait on whoever is creating the template they're not competing
for resources with it.
2023-07-10 15:23:31 +02:00
Xavier-Do
4a99ee3ce3 [RVE] runbot: revert customizable build_access
This reverts commit 9e7441e098.

This doesn't work as expected because of db filter.
Will eb changed latter, reverting for now

Token field is kept, could still be used later.
2023-07-05 14:54:53 +02:00
Xavier-Do
ea490bffab [REF] runbot: cleanup old dead code 2023-07-04 10:46:41 +02:00
Xavier-Do
0c609ac533 [IMP] runbot: improve cleanup perfs
The current version will read ~100000 build

This one will avoid that by checking the date of the gcstamp instead
2023-07-04 10:46:41 +02:00
Christophe Monniez
7958375fb5 [IMP] runbot: log container short id
When investigating kernel logs e.g.: for finding oom killed containers,
the kernel does not log the name of the incriminated container but only
the id. With this commit the runbot will also log the container short id
which is enough to correlate the logs.
2023-07-04 10:32:54 +02:00
Christophe Monniez
207b890018 [FIX] runbot: frontendurl should use the right id 2023-07-03 16:30:06 +02:00
Xavier-Do
92157ad71a [IMP] runbot: make access to build customizable 2023-07-03 15:52:53 +02:00
Xavier-Do
91470e0cfa [IMP] runbot: add dedicated route for running 2023-07-03 15:52:53 +02:00
Xavier-Do
d09b9961cd [IMP] runbot: log user beside ip 2023-07-03 15:52:53 +02:00
Christophe Monniez
ab194610b0 [IMP] runbot: improve bundle backend page
With this commit trigger customization, branches and last batches are
put in tabs. That way, trigger customization are quickly accessible.
2023-06-30 10:57:01 +02:00
Christophe Monniez
6c41fbd8ae [FIX] runbot: fix build error form disposition
- Base info span over two columns
- Proper fixing info groups
2023-06-30 10:57:01 +02:00
Christophe Monniez
6ed23db655 [FIX] runbot: open frontend url's in a new tab by default 2023-06-30 10:57:01 +02:00
Christophe Monniez
3c8f821695 [IMP] runbot: unify build error wizards
There are two wizards for the runbot build errors:
- One to close an error with a reason
- One to update the team/user or PR

With this commit, the two wizards are merged into one wizard that helps
to update errors in bulk.

Also, a button is added in the list view that allow to save a mouse
click.

The `NEW` button is removed from the tree view as it should not be of
any use.
2023-06-30 10:57:01 +02:00
Christophe Monniez
1cc73a606d [IMP] runbot: allow only advanced users to wake up a child
As it happens that some users are waking up child builds, let's only
allow advanced users to do that.
2023-06-30 10:57:01 +02:00
Christophe Monniez
503afda3b6 [IMP] runbot: add a monthly filter on build errors 2023-06-30 10:57:01 +02:00
Christophe Monniez
041c4f242f [FIX] runbot: allow PR link only when it exists 2023-06-30 10:57:01 +02:00
Xavier-Do
506ff03e07 [IMP] runbot: pause, and profile.
Pausing a host can be usefull in some case, mainly when testing new code
The loop will have no effect avoiding to break some build wainting for
testing.

Profile will help to identify potential performance flows during the
loop.
2023-06-30 10:54:50 +02:00
Christophe Monniez
2b95e0dc01 [FIX] runbot: remove search_count hack
Since odoo/odoo@a01e8b5232 this hack should not be necessary anymore.
2023-06-30 10:37:28 +02:00
Xavier-Do
cd8cc1f77f [FIX] runbot: fix onchange 2023-06-27 14:42:00 +02:00
Xavier-Do
2a003f00de [FIX] runbot: fix dump_db 2023-06-27 14:28:08 +02:00
xdo
24d35988a4
[IMP] runbot: add auto restore for custom trigger (#784)
One of the most common custom trigger is to restore a build before
starting some test, either to create a multibuild or make the execution
and debug of some test faster.

It is somethimes tedious to use because we need to give an url of a
build to restore. This build must correspond to the right commits,
must still exixt, ... this means that the dump url must be adapted
everytime a branch is rebased.

The way the dump_url is defined is by going on the last batch, following
the link to the `base_reference_batch_id`, finding a slot corresponding
to the right repo set, (ex: Custom enterprise -> enterprise), and
copying the dump_url in this build.

The base_reference_batch_id is eay to automated but we have to find the
right trigger, this is now a parameter of the custom trigger wizard.
There are actually 2 strategy now to define how to download the dump:
- `url`, using `restore_ dump_url`
- `auto`,  using `restore_trigger_id` and `restore_database_suffix`

To ease the setup, a `restore_trigger_id` is added on a trigger, so that
when selecting a trigger, lets say `Custom enterprise`, the defined
`trigger.restore_trigger_id` is automatically chosen for the
`custom_trigger.restore_trigger_id` and the `restore_mode` is setted to
auto.

Two actions are also added to the header of a bundle, a shorcut to
setup a multi build (restore in children) or a restore and test build
(restore in parent).
2023-06-27 14:13:12 +02:00
Xavier-Do
465081e9f3 [FIX] runbot: fix jsonb in list view
In view list widget are not always instanciated and a formater is used
instead. This means that the t-esc will try to output a jsonb field
without nowing how to render it, making the page crash.

This is quickly fixed by forcing the widget on the field in tree view.
2023-06-26 13:36:20 +02:00
Xavier-Do
96808acb37 [IMP] runbot: diffable tracking in chatter
The python steps can be long and interresting to track but the change is
actually hard to see since a block of code is logged instead of the diff.

Also, the whitespaces are not preserverd since we are note in a <pre>
block making it hard to read.

This proposes an alternative to display python code tracking values as
a diff with options to copu the raw content of the old and new version.
(as well as showing unchanged lines or not)
2023-06-26 13:11:00 +02:00
Xavier-Do
727d71d103 [IMP] runbot: add an action to check token 2023-06-22 16:43:05 +02:00
Xavier-Do
f82478576d [FIX] runbot: fix bundle view 2023-06-22 16:35:19 +02:00
Christophe Monniez
37babcd526 [FIX] runbot: remove chatter widgets
Since odoo/odoo@3fea5b2136 the widgets specifications in the chatter are
not necessary anymore.
2023-06-22 16:30:03 +02:00
Christophe Monniez
7e92aaeae2 [FIX] runbot: chatter must be after sheet
The `oe_chatter` div must be defined after a sheet, otherwise two
chatters are visible on the form. One bellow and one on the right side.
2023-06-22 16:30:03 +02:00
Xavier-Do
f22ae357e3 [DEL] runbot: 15.0 eol 2023-06-22 15:59:50 +02:00
Xavier-Do
b72fdff01d [DEL] runbot: 15.0 eol 2023-06-22 15:59:36 +02:00
Xavier Morel
72281b0c63 [IMP] runbot_merge: allow running test suite without an explicit addons path 2023-06-22 14:38:10 +02:00
Xavier-Do
ebb531f4a4 [FIX] runbot: adapt for 16.0 2023-06-22 14:27:32 +02:00
Xavier-Do
fc41e09f44 [IMP] runbot: add useful indexes remaining from upgrade 2023-06-22 11:23:43 +02:00
Xavier-Do
6107c0aefb [FIX] runbot: adapt widgets to owl
This is a first quick version to keep main functionalities, this will
need to be adapt and clean later.
2023-06-22 11:15:16 +02:00
Xavier-Do
70c468f821 [FIX] fix invalid t-att 2023-06-21 15:47:54 +02:00
Xavier Morel
765281a665 [FIX] runbot_merge: make provisioning more resilient
A few cases of conflict were missing from the provisioning
handler.

They can't really be auto-fixed, so just output a warning and ignore
the entry, that way the rest of the provisioning succeeds.
2023-06-21 14:26:19 +02:00
Xavier Morel
9260384284 [FIX] runbot_merge: concurrency error in freeze wizard (hopefully)
During the 16.3 freeze an issue was noticed with the concurrency
safety of the freeze wizard (because it blew up, which caused a few
issues): it is possible for the cancelling of an active staging to the
master branch to fail, which causes the mergebot side of the freeze to
fail, but the github state is completed, which puts the entire thing
in a less than ideal state.

Especially with the additional issue that the branch inserter has its
own concurrency issue (which maybe I should fix): if there are
branches *being* forward-ported across the new branch, it's unable to
see them, and thus can not create the now-missing PRs.

Try to make the freeze wizard more resilient:

1. Take a lock on the master staging (if any) early on, this means if
   we can acquire it we should be able to cancel it, and it won't
   suffer a concurrency error.
2. Add the `process_updated_commits` cron to the set of locked crons,
   trying to read the log timeline it looks like the issue was commits
   being impacted on that staging while the action had started:
   REPEATABLE READ meant the freeze's transaction was unable to see
   the update from the commit statuses, therefore creating a diverging
   update when it cancelled the staging, which postgres then reported
   as a serialization error.

I'd like to relax the locking of the cron (to just FOR SHARE), but I
think it would work, per postgres:

> SELECT FOR UPDATE, and SELECT FOR SHARE commands behave the same as
> SELECT in terms of searching for target rows: they will only find
> target rows that were committed as of the transaction start
> time. However, such a target row might have already been updated (or
> deleted or locked) by another concurrent transaction by the time it
> is found. In this case, the repeatable read transaction will wait
> for the first updating transaction to commit or roll back (if it is
> still in progress). If the first updater rolls back, then its
> effects are negated and the repeatable read transaction can proceed
> with updating the originally found row. But if the first updater
> commits (and actually updated or deleted the row, not just locked
> it) then the repeatable read transaction will be rolled back with
> the message

This means it would be possible to lock the cron, and then get a
transaction error because the cron modified one of the records we're
going to hit while it was running: as far as the above is concerned
the cron's worker had "just locked" the row so it's fine to
continue. However this makes it more and more likely an error will be
hit when trying to freeze (to no issue, but still). We'll have to see
how that ends up.

Fixes #766 maybe
2023-06-21 14:26:19 +02:00
Xavier Morel
ed0fd88854 [ADD] runbot_merge: sentry instrumentation
Currently sentry is only hooked from the outside, which doesn't
necessarily provide sufficiently actionable information.

Add some a few hooks to (try and) report odoo / mergebot metadata:

- add the user to WSGI transactions
- add a transaction (with users) around crons
- add the webhook event info to webhook requests
- add a few spans to the long-running crons, when they cover multiple
  units per iteration (e.g. a span per branch being staged)

Closes #544
2023-06-21 14:26:19 +02:00
Xavier-Do
614040720b [IMP] runbot: add useful indexes remaining from upgrade 2023-06-21 10:08:14 +02:00
Xavier-Do
8d724b157f [FIX] runbot: adapt frontend to 16.0
A first step to get more independant from web and website was done in
15.0 but some file were moved and it looks like the bootstrap version
changed breaking the frontend again. (4.3->5.1)

This commit simply copies the libs from odoo/15.0 to avoid losing time
fixing the frontend look and feel for the new bootstrap version for now.

A future refactoring could change the vendored version to addapt to 5.1
while modernizing the frontend style.
2023-06-20 14:15:26 +02:00
Xavier-Do
45104b635f [REL] adapt for 16.0 2023-06-20 12:34:32 +02:00
Xavier-Do
d3f998f88c [IMP] runbot: keep log create_date 2023-06-20 09:18:48 +02:00