Commit Graph

1315 Commits

Author SHA1 Message Date
Xavier-Do
9fb7e94de9 [IMP] add load info page 2022-07-25 12:10:48 +02:00
Xavier-Do
a52c14957e [IMP] runbot: log manual batch and redirect
1. Log who started a manull build
2. redirect if the batch given in a build url is not containing this
build (manual url)
2022-07-12 13:39:44 +02:00
Xavier-Do
af8a5b8e8e [FIX] runbot: avoid error when there is no build 2022-07-08 15:53:43 +02:00
Christophe Monniez
e1161a03e6 [FIX] runbot: avoid sudo when using pip
In bd4cf76b7 a new argument `--progress-bar off` was added to the pip
command line. Unfortunately, it appears that this argument is not
available in some pip versions, typically the one from Ubuntu Bionic
(9.0.1) [0]. The argument appeared in version 10.0.0 [1].

Also, in the DockerFile, we allow the user to use `sudo` for
`/usr/bin/pip3` and the main reason for that was to avoid having `pip`
to re-install the packages that are pre-installed by the distribution
package manager.

That said, it appears that since pip 10.0.0 when installing packages
locally for the user, pip now properly detects distribution installed
packages.

So, as the solution to fix the progress bar issue is to upgrade pip to a
newer version, we can take advantage of it to get rid of the `sudo`.

Besides the fact that `--user` is the default on Debian based
distributions [2] , we enforce it in case another distribution is used
in the Dockerfile.

[0] https://packages.ubuntu.com/bionic-updates/python3-pip
[1] https://pip.pypa.io/en/stable/news/#id744
[2] https://wiki.debian.org/Python#Deviations_from_upstream
2022-07-06 09:48:20 +02:00
Xavier-Do
bd4cf76b76 [FIX] runbot: hide progressbar for pip 2022-07-01 15:47:13 +02:00
Xavier-Do
03732638f3 [IMP] runbot: add an automatic generated message for upgrade exceptions 2022-07-01 15:47:13 +02:00
Xavier Morel
2a268d777c [IMP] forwardport: check commit liveness on the PR we're porting from
Check remains a tad dodgy, but since we're actually porting from
`root` (the earliest ancestor of `self`) it makes more sense to
sanity-check that *its* commits remain visible.

Also log that as it makes a tad more sense, hopefully.

Closes #600
2022-06-30 15:07:49 +02:00
Xavier Morel
2b3ee48ebd [IMP] runbot_merge: staging logging
The log message only indicated whether the PR was squashed or not, but
that's not actually useful.

Improve the message to log the actual merge method, for
information. The old "squash" (aka squash flag set and no merge
method, since an actual squash merge method was reintroduced a while
ago) has been renamed to "single" for the purpose of this display.
2022-06-30 15:07:49 +02:00
Xavier Morel
e8ae5ec263 [IMP] forwardport: flag detached PRs in their dashboard
This is an important bit of information but it was not visible without
going into the backend.

`user-select-none` doesn't work in BS3 but that way it'll be ready for
an eventual update. Currently when hovering the badge the cursor
switches to text selection, and the text is selectable, which is
useless.

Complements 4e235a2 and finishes the fixes for #617
2022-06-30 15:07:49 +02:00
Xavier Morel
4e70b1acbb [FIX] forwardport: detached / closed PR clarifications
- trying to r+ a detached PR *via the forwardbot* should warn, same as
  a non-forwardport PR
- the following sibling of a closed PR should be detached from
  it (probably)
- when a closed forward-port PR is reopened, there should be a
  notification that it is detached and merged via mergebot

Fixes #617
2022-06-30 15:07:49 +02:00
Xavier Morel
0016404474 [IMP] runbot_merge: better styling for missing statuses on stagings
Before this, until the first status for a required context the status
would appear as pending, but would be have oddly (e.g. not clickable).

Update the style of such statuses for clarity:

- use a light background to show them as inactive
- use the `wait` cursor to show their status as oddball (and not clickable)

Setting this styling on the link (or even `li`) doesn't seem to work,
so set it on the `ul`, the actual active links will set the relevant
"active" cursor instead, which seems to work fine.

While at it, extract the status menu to its own template and unify the
disparate bits, mainly in that both the main dashboard and the
per-branch list display the staging instant in UTC on hover: before,
the main dashboard would display a relative delta and provide the
UTC-formatted instant on hover, but the branch would only show a zoned
ISO-8601 instant.

While adjusting is easy, it's unnecessary, we can easily provide the UTC
staging instant there).
2022-06-30 15:07:49 +02:00
Xavier Morel
fb45f089b0 [FIX] forwardport: use diff3 for conflict style
Existing conflict style is the local default ("merge", most
likely). `diff3` is a lot more informative as it provides the common
ancestor's code for the hunk, which helps see how the two branches
diverged and thus resolve the conflict.

Even better would be zdiff3 but that's a bit too recent...

Fixes #619
2022-06-30 15:07:49 +02:00
Xavier Morel
f430c014c1 [IMP] *: review mergebot & forwardbot messages for pinging
Old messages were quite inconsistent in their pinging of the PR author
and reviewer.

Reviewed messages (probably missed some but...) and try to more
consistently ping when the feedback requires some sort of action in
order to proceed.

Fixes #592
2022-06-30 15:07:49 +02:00
Xavier Morel
4a3cde2faa [IMP] runbot_merge: provisioning features
A few fixes and improvements after testing the feature:

- ensure the provisioned users are created as internal (not portal)
- assume oauth is installed and just crash if it's not
- handle a user not having an email (ignore)
- return value from json handler, otherwise JsonRequest sends no
  payload which is *weird*
2022-06-30 15:07:49 +02:00
Xavier Morel
96f8f4e688 [FIX] runbot_merge: unstage on staging-relevant PR modifications
Stagings would be cancelled automatically if the PR's commits were
updated, but not if the target (base) was changed, even though that
has a drastic impact on staging.

Add hooks to unstage PRs if their base is updated, or if their message
is updated and relevant to staging (merge or rebase-merge methods).

Fixes #604
2022-06-30 15:07:49 +02:00
Xavier Morel
a5fae548fe [IMP] runbot_merge: extract dashboard pr linking
Extract the creation of a PR link (to github) as a dedicated
template for easier updates, and to use `display_name`
everywhere (instead of reimplementing it by name).

Also implement support for repo-level groups setting for information
hiding.

Fixes #590
2022-06-30 15:07:49 +02:00
Xavier Morel
66c2bdc25b [IMP] runbot_merge: error reporting on fast-forward failure
When a staging's fast-forward (to the target branch) fails, the
mergebot would provide no useful information on the staging or the
dashboard.

This is because the reason was set to the HTTP status, which in case
of a fast-forward error is just "422 client error: unprocessable
entity".

Improve this by trying to parse github's response in that case, and
using the JSON error message as failure reason. This provides more
useful failure information like "update is not a fast forward",
"reference does not exist", or a branch protection failure.

Closes #591
2022-06-30 15:07:49 +02:00
Xavier Morel
2204c0410a [IMP] mergebot, forwardbot: various UI bits
- code in the various menus added over time through the UI (queues,
  configuration, ...)
- update / improve PR layout a tick
- fix "outstanding forward ports" count on the dashboard
- improve hover title / help on dashboard
  - add date of last modification (usually date of success / failure)
  - make casing more coherent (everything lowercase)
  - add explicit note that UTC date on staged at label is staged at datetime
  - rediscover yet again that the staging information is when hovering
    on the staging *except the staged at label*
- improve `PullRequest.unstage` to always insert the PR at the start of the
  reason when cancelling the staging, for clarity / traceability

Closes #560, closes #609
2022-06-30 15:07:49 +02:00
Christophe Monniez
b25aa52fb9 [IMP] runbot: avoid core dumps in containers
In some conditions, the google-chrome crashes and a core dump is written
in the bind mounted directory.

With this commit, the core dumps are disabled in the containers.
2022-06-30 10:58:38 +02:00
Christophe Monniez
78697d2acb [IMP] runbot: use docker python lib
While using the docker cli as a subprocess was KISS and convenient,
the python Docker SDK is mature, easy to use, available as a Debian
package and much more powerful.

It will permit to monitor the containers memory consumption and will
help to spot memory leaks.
2022-06-30 10:58:38 +02:00
Christophe Monniez
9fb48f48cf [IMP] runbot: improve git gc warning 2022-06-27 09:11:02 +02:00
Christophe Monniez
3f042c44a8 [FIX] runbot: fix gc warning 2022-06-24 08:41:32 +02:00
Christophe Monniez
d027d1c7de [IMP] runbot: git gc only once a day 2022-06-23 09:45:47 +02:00
Christophe Monniez
c006bf953d [FIX] runbot: apply small fixes 2022-06-20 14:37:02 +02:00
Christophe Monniez
d534f5a5e2 [FIX] runbot: count only really active hosts 2022-06-20 10:32:10 +02:00
Christophe Monniez
250d48e266 [IMP] runbot: schedule git gc on repositories
On the actual runbot deployments, the `git gc` command is handled by a
unix cron. From time to time, some repositories get corrupted and we
suspect that some concurrent action may be involved as stated in
documentation [0].

For those reasons, with this commit, the `git gc` will be run by the
runbot clients themselves in order to avoid concurrent operations.

By default, the first gc will occur a few minutes after the start of the
client and the next gc are scheduled a two hours and a few minutes later.

Also, this commit ensures that the git config is written regularly in
case of change.

[0] https://git-scm.com/docs/git-gc
2022-06-20 10:32:10 +02:00
Xavier-Do
cdaae9b3ed [IMP] runbot: speedup get_refs
Regarding the number of refs in odoo repo (arround 18 million at this
time), the parsing of the date was significant when filtering old refs.

Using unix time allows a direct comparaison without parsing the date,
and improved performance, going from ~7 seconds to ~1.3 seconds.
2022-06-20 10:20:37 +02:00
Xavier-Do
cbfc8401a8 [FIX] runbot: various FIX 2022-06-17 13:52:37 +02:00
Xavier-Do
26a3ad20f1 [IMP] runbot: status management
The status are currently sent by leader when build are created and by
workers on post_commit.

If the leader fails during the preparing of a batch (while creating
builds) the transaction is rollbacked and the status are send again.

The number of status to send makes it quite slow, making the transaction
longuer, and the retry even more expensive. This leads to preparing time
being quite important, sometimes ten minutes after many retries.

This commit proposes to send status in another dedicated transaction.
Since status are sent in batch, we can also try to use a unique session,
and uniquify commit+context status.

This allows to remove the postcommit logic

A further improvement would be to wait before sending status in order to
skip the pending status if the build is verry fast.

An option is also added on the remote to skip the status: if the remote
is a fork, sending the satus on the main remote should be enough.
2022-06-17 12:32:14 +02:00
Christophe Monniez
b997119588 [FIX] runbot: avoid children in team errors
When an error is assigned to a team and the error is a child of another
one, the two errors are displayed on the team dashboard and adds
confusion.
2022-06-17 10:40:57 +02:00
Christophe Monniez
69cd228b6b [IMP] runbot: use the lovely boolean_toggle widget 2022-06-16 13:39:46 +02:00
Xavier-Do
47c12fe00b [FIX] runbot: add MULTILINE to ReProxy 2022-06-15 12:30:41 +02:00
Xavier-Do
bf0df23a3d [FIX] runbot: add findall to ReProxy 2022-06-15 12:13:34 +02:00
Xavier-Do
15e4fa76c5 [FIX] runbot: add VERBOSE to ReProxy 2022-06-15 09:12:17 +02:00
Xavier-Do
2ac70f5ab4 [FIX] runbot: fix test_all test for restore 2022-06-13 14:03:16 +02:00
Xavier-Do
d466887f08 [REF] runbot: adapt statistic models
Initial model was a (build_id, key, value) where key is in fact a two
part information: `category.key` (where key is usually a module)

This means that for each module, we will have one entry per
modules*category.
We have between 200 and 400 modules per build * 4 keys -> around 1000
entries per build.

The hudge amount of total entries lead to a fast overflow of the table
sequence + this create important indexes.

Also, most of the time, the js will manage the display of stats meaning
that python will transform
(build_id, category.key, value)
into
(build_id, {key: value}) for one category

A new model makes use of a json field to store values for different
modules in a json dict, on entry per category*build. (4 entry per build)

The table will be renamed and migrate later
2022-06-13 12:42:13 +02:00
Xavier-Do
87366de213 [IMP] runbot: add log on separator 2022-06-10 14:56:13 +02:00
Xavier-Do
77f8c29091 [IMP] runbot: check restore result 2022-06-10 14:02:58 +02:00
Xavier-Do
1879f2c235 [IMP] runbot: forced host improvement 2022-06-10 13:26:47 +02:00
Xavier-Do
f654738b04 [FIX] runbot: fix nginx static 2022-06-09 15:53:41 +02:00
Xavier-Do
a6fa7b8023 [FIX] runbot: fix leader pull_info_failures
The initial pull_info_failures feature was introduced before the separate
process for the leader. Since this is here for a rare case, this
wasn't tested since then.

Testing the new infrastructure without token, there was many
pull_info_failure, leading to an error on cleanup.
2022-06-09 08:43:28 +02:00
Christophe Monniez
33ab9cdce6 [FIX] runbot: open nginx file as text
Continuation of previous fix 6902a7e24.
2022-06-08 15:33:26 +02:00
Christophe Monniez
6902a7e242 [FIX] runbot: cast markup into str before saving
When rendering templates for git configuration and nginx configuration,
the templates are rendered as Markup, while a byte-like object was
expected for the saving.

With this commit, the Markup is casted into str and the files are saved
as text files.
2022-06-08 15:19:08 +02:00
Christophe Monniez
a051568213 [FIX] runbot: avoid werkzeug deprecated escape
The `utils.escape` utility was deprecated in werkzeug 2.0.0, so it can
be replaced with our `html_escape` which is itself `markupsafe.escape`.

Also, with this change, the double quotes are now escaped in `"`
instead of `"`, so we fix the test too.
2022-06-08 10:53:50 +02:00
Xavier-Do
ca1246b87a [FIX] runbot: fix pseudo_markdown 2022-06-07 15:56:26 +02:00
Xavier Morel
56898df93f [ADD] runbot_merge: remote user provisioning
New accounts endpoint such that the SSO can push new pre-configured
users / employees directly. This lowers maintenance burden.

Also remove one of the source partners from the merge test, as
ordering seems wonky for unclear reasons leading to random failures of
that test.
2022-06-07 13:48:17 +02:00
Xavier-Do
bf4167670d [IMP] runbot: cleanup 2022-06-07 11:18:34 +02:00
Xavier-Do
6b79d5b96c [REM] runbot: remove runbot_domain
This parameter can be replaced by web.base.url when needed.
2022-06-07 10:54:48 +02:00
Xavier-Do
272096b810 [REM] runbot: remove non nginx support
Some non tested code was making running build available without
an nginx config (using port and local address instead of using a proper
dns/nginx config).
Removing this part to reduce complexity.
2022-06-07 10:31:09 +02:00
Xavier-Do
02cbc0d090 [FIX] runbot: adapt for jammy 2022-06-03 14:51:49 +02:00