Commit Graph

1584 Commits

Author SHA1 Message Date
Xavier-Do
47c12fe00b [FIX] runbot: add MULTILINE to ReProxy 2022-06-15 12:30:41 +02:00
Xavier-Do
bf0df23a3d [FIX] runbot: add findall to ReProxy 2022-06-15 12:13:34 +02:00
Xavier-Do
15e4fa76c5 [FIX] runbot: add VERBOSE to ReProxy 2022-06-15 09:12:17 +02:00
Xavier-Do
2ac70f5ab4 [FIX] runbot: fix test_all test for restore 2022-06-13 14:03:16 +02:00
Xavier-Do
d466887f08 [REF] runbot: adapt statistic models
Initial model was a (build_id, key, value) where key is in fact a two
part information: `category.key` (where key is usually a module)

This means that for each module, we will have one entry per
modules*category.
We have between 200 and 400 modules per build * 4 keys -> around 1000
entries per build.

The hudge amount of total entries lead to a fast overflow of the table
sequence + this create important indexes.

Also, most of the time, the js will manage the display of stats meaning
that python will transform
(build_id, category.key, value)
into
(build_id, {key: value}) for one category

A new model makes use of a json field to store values for different
modules in a json dict, on entry per category*build. (4 entry per build)

The table will be renamed and migrate later
2022-06-13 12:42:13 +02:00
Xavier-Do
87366de213 [IMP] runbot: add log on separator 2022-06-10 14:56:13 +02:00
Xavier-Do
77f8c29091 [IMP] runbot: check restore result 2022-06-10 14:02:58 +02:00
Xavier-Do
1879f2c235 [IMP] runbot: forced host improvement 2022-06-10 13:26:47 +02:00
Xavier-Do
f654738b04 [FIX] runbot: fix nginx static 2022-06-09 15:53:41 +02:00
Xavier-Do
a6fa7b8023 [FIX] runbot: fix leader pull_info_failures
The initial pull_info_failures feature was introduced before the separate
process for the leader. Since this is here for a rare case, this
wasn't tested since then.

Testing the new infrastructure without token, there was many
pull_info_failure, leading to an error on cleanup.
2022-06-09 08:43:28 +02:00
Christophe Monniez
33ab9cdce6 [FIX] runbot: open nginx file as text
Continuation of previous fix 6902a7e24.
2022-06-08 15:33:26 +02:00
Christophe Monniez
6902a7e242 [FIX] runbot: cast markup into str before saving
When rendering templates for git configuration and nginx configuration,
the templates are rendered as Markup, while a byte-like object was
expected for the saving.

With this commit, the Markup is casted into str and the files are saved
as text files.
2022-06-08 15:19:08 +02:00
Christophe Monniez
a051568213 [FIX] runbot: avoid werkzeug deprecated escape
The `utils.escape` utility was deprecated in werkzeug 2.0.0, so it can
be replaced with our `html_escape` which is itself `markupsafe.escape`.

Also, with this change, the double quotes are now escaped in `"`
instead of `"`, so we fix the test too.
2022-06-08 10:53:50 +02:00
Xavier-Do
ca1246b87a [FIX] runbot: fix pseudo_markdown 2022-06-07 15:56:26 +02:00
Xavier Morel
56898df93f [ADD] runbot_merge: remote user provisioning
New accounts endpoint such that the SSO can push new pre-configured
users / employees directly. This lowers maintenance burden.

Also remove one of the source partners from the merge test, as
ordering seems wonky for unclear reasons leading to random failures of
that test.
2022-06-07 13:48:17 +02:00
Xavier-Do
bf4167670d [IMP] runbot: cleanup 2022-06-07 11:18:34 +02:00
Xavier-Do
6b79d5b96c [REM] runbot: remove runbot_domain
This parameter can be replaced by web.base.url when needed.
2022-06-07 10:54:48 +02:00
Xavier-Do
272096b810 [REM] runbot: remove non nginx support
Some non tested code was making running build available without
an nginx config (using port and local address instead of using a proper
dns/nginx config).
Removing this part to reduce complexity.
2022-06-07 10:31:09 +02:00
Xavier-Do
02cbc0d090 [FIX] runbot: adapt for jammy 2022-06-03 14:51:49 +02:00
Xavier Morel
1dcab0744d [IMP] runbot_merge: remove dead code
apparently at one point I wanted to provide an atom feed or something
2022-06-02 12:55:22 +02:00
Xavier Morel
48c9754906 [IMP] runbot_merge: acknowledge check command
The command works, but the 'bot would always reply it was not allowed
2022-06-02 12:55:22 +02:00
Xavier-Do
3071f5a6db [IMP] runbot: adapt for odoo 15.0 2022-06-02 10:50:42 +02:00
Xavier-Do
91d176f561 [FIX] runbot: fix pull_info_failures 2022-03-30 08:11:45 +02:00
Xavier-Do
78f050b132 [IMP] runbot: adapt for 14.0 2022-03-23 09:49:53 +01:00
Xavier-Do
eda014f724 [DEL] runbot: remove old upgrade scripts 2022-03-18 11:09:58 +01:00
Xavier-Do
8c120ac0d9 [FIX] runbot: don't log invalid bases on autorebase 2022-03-17 09:30:03 +01:00
Xavier-Do
c4d0b56972 [FIX] runbot: dont duplicate export
Even if it isn't really problematic, avoiding to duplicate exports
will make the table slighly smaller, especially when considering running
builds.
2022-03-16 09:44:58 +01:00
Xavier-Do
3985332a34 [IMP] runbot: commit exports
Source cleanup will check in multiple place for potentially used
commits. This maked the cleanup logic complex, plus limit the usable
commits in python steps. The current use case is to export the mergebase
commits.

The poposed solution is to save the exported commit and clean this list
when the build is done. All commit in this list cannot be cleaned.
2022-03-16 09:05:39 +01:00
Xavier-Do
ea3191a8d0 [IMP] runbot: add javascript support for JsonDictFields
The main motivation is to be able to see existing config_data on params
and edit configuration on triggers.

A simple version would be to use the `FieldChar` with a simple
`JSON.stringify` formater, but having some indent on field make it
easier to read and edit.

The multiline need make it closet to the `FieldText` than the `FieldChar`
but the reset makes the browser freeze.
2022-03-14 15:27:11 +01:00
Xavier-Do
623255b0d8 [IMP] runbot: reorganize menus
and add missing fields...
2022-03-11 15:12:36 +01:00
Xavier-Do
45cc0b1675 [REF] runbot: remove useless names 2022-03-11 15:12:36 +01:00
Xavier-Do
ca8ee428f8 [REF] runbot: move all menu to a single file 2022-03-11 15:12:36 +01:00
Xavier-Do
4e6ae00f23 [IMP] runbot: custom trigger wizard
The actual way to add a custom multibuild on a dev bundle will be:
- use the multibuild wizard to create 2 configs and 2 steps
- add this config on the bundle
- customize this config to make it fatser by installing/restoring...

The usefull parameters are the number of build, the test tags and
modules to install.

Another usefull operation is to restore a dump instead of installing,
especially for post install test only breaking on full databases.

This commit proposes to replace the multi build wizard with a custom
trigger wizard. The main idea is to have generic config, parametrized by
config_data. The wizard will only help to generate the corresponding
config_data.
2022-03-11 14:00:47 +01:00
Xavier Morel
2337bd8518 [FIX] forwardport: chain crash on insert in forward-port cron
On two of the freezes, thereafter the logs showed serial crashes in
the forwardport cron when trying to find the insertion point for a new
forward-port.

The first time was not really diagnosed, the second time the cause was
found to be a retargeted PR which led to a failure of the "insertion"
forward port, which did not take that possibility in account (it
assumed -- sensibly I believed -- that an intermediate FP following a
branch insertion would always succeed, sadly the malevolent universe
had other plans).

So only insert the new forward port inside its sequence (if necessary)
if the forward port actually succeeded, otherwise ignore it.

Fixes #551
2022-02-10 13:51:33 +01:00
Xavier Morel
2898c7edd4 [FIX] runbot_merge: updating of commit date on rebase
On #509, the rebasing process was changed to forcefully update the
commit date of the commits, in order to force trigger builds.

However when squashing was re-enabled for #539 for some fool reason it
implemented its own bespoke rebasing (despite that not actually saving
any API call that I can see), meaning it did *not* update the commit
date. As such, an old commit being squashed would not get picked up by
the runbot, which is what happened to odoo/documentation#1226 (which
ultimately had to be hand-rebased after some confusion as to why it
did not work).

Update `_stage_squash` to go through `rebase` the normal way, also
update `rebase` to pop the commit date entirely instead of setting it
manually, and update the squashing test to check that the commit date
gets properly updated.

Fixes #579, closes #582
2022-02-10 13:51:08 +01:00
Xavier Morel
2285965e22 [MRG] runbot_merge, forwardport: multiple fixes to freeze wizard
Multiple fixes to various issues of the freeze wizard.

Some of the less fix-oriented sub-tasks were moved to #586 for discussion instead.

Closes #559, #587
2022-02-10 13:50:21 +01:00
Christophe Monniez
a931af2fed [IMP] runbot: fully garbage collect old builds
When a build is garbage collected and the local folders are cleaned, the
log files are kept to allow build errors investigations.

With this commit, a full local cleanup will be done 365 days after the
garbage collect.  This means that the build dir will be completely
removed.  The default number of days can be tweaked with the
`runbot.full_gc_days` ir.config_parameter.

Also, the _local_cleanup method can be called with a `full` boolean
parameter to force a full cleanup. e.g.: when called in a config step.

While at it, the res_config_settings view is a bit reorganized.
2022-02-10 13:39:50 +01:00
Christophe Monniez
2f7c752793 [FIX] runbot: avoid hook_time update on builders 2022-02-10 13:39:18 +01:00
Xavier Morel
de70bd6f83 [IMP] runbot_merge: show PR titles in freeze wizard
Currently limited to release/freeze PRs: it can be difficult to be
sure the right PR was selected then, and a mistake there seems more
impactful than in the PRs being waited for?

Note: adds a test to make sure I don't break the check that all
      release PRs must have the same label (be linked). This was
      already safe, and in a way this PR adds convenience but not
      really safety, but better sure than sorry.
2022-02-08 12:28:10 +01:00
Xavier Morel
e05cc77a57 [FIX] forwardport: don't forwardport freeze PRs
The freeze wizard has support for merging freeze / release PRs on each
of the newly created branches. But since this would be done by, well,
merging, those PRs would get forward-ported to master, and would have
to be closed there.

This creates additional work for the freeze master, and noise /
parasitic PRs.

Obviously it's possible for the freeze master to set some nonsense `up
to` (nonsense because the "real" limit doesn't exist yet at that
point), but really it never makes any sense to forward port release
PRs, so the wizard should do it.
2022-02-08 10:11:57 +01:00
Xavier Morel
e2887a7473 [IMP] runbot_merge: allow only freezing a subset of a project
- add flag to not select repos for freezing
- allow removing more repositories from the wizard
- when performing the freeze, only create branches for the selected
  repos
2022-02-07 15:15:13 +01:00
Xavier Morel
1add3d4854 [FIX] runbot_merge: freeze being triggered upon reopening the wizard
The freeze wizard was implemented using a single action to open and
validate the dialog.

This was a mistake, as it means if there are no errors left (e.g. all
the PRs being waited for are now validated) trying to view the freeze
wizard will immediately validate it and commit the freeze, which is
unexpected, surprising, and unsafe e.g.

- open wizard
- add freeze prs
- add a required pr or two
- close and go do something else
- be told that more PRs need to be waited for
- reopen wizard
- oops freeze is done

So split the "open action" part of `action_freeze` into opening the
action and performing the freeze. The "freeze" / "view freeze" button
on the project only activates the latter, and the actual freeze
operation is only triggered from the wizard's "Freeze" button.

Part of #559.
2022-02-07 13:23:41 +01:00
Xavier-Do
ea5feb114a [FIX] runbot: fix codowner match 2022-02-07 13:03:26 +01:00
Xavier-Do
2c29a6b94b [IMP] runbot: add validation for domain 2022-02-07 09:31:19 +01:00
Xavier-Do
ebb342ea0c [IMP] runbot: link on reference batch 2022-02-07 08:59:28 +01:00
Xavier Morel
8e8a238a66 [FIX] forwarport: mark branches as fp-targets by default
otherwise the freeze wizard gets very confused
2022-02-07 08:10:03 +01:00
Xavier Morel
4c8ab63af8 [REM] runbot_merge: leftover debug print
Found by @d-fence
2022-02-07 08:08:59 +01:00
Christophe Monniez
ffd432311f [FIX] runbot: removing failed git export
When a git archive fails, the partially exported source tree is left in
place. If another builds tries to use the same commit, the tree is not
exported anymore as the directory exists. This leads to non
deterministic behaviors.
2022-01-27 10:19:54 +01:00
Xavier-Do
8cdb8c9e5b [FIX] runbot: small fixed to ease deployment 2022-01-27 10:08:39 +01:00
Christophe Monniez
ec2f12ef5f [FIX] runbot: manage or in debian/controll
Since odoo/odoo@3d6043a735
the controll file can contain fallbacks `dep1 | dep2 | dep3`.
Taking the first column will work in most cases.

closes #573
2022-01-27 10:08:39 +01:00