Commit Graph

1508 Commits

Author SHA1 Message Date
Christophe Monniez
bdd98b07ec [IMP] runbot: rename active to active
Error is not fixed was too disturbing.
2023-06-01 15:03:13 +02:00
Christophe Monniez
6e145ff362 [IMP] runbot: improve build errors tabs
* show only the all builds tab
* hide linked errors tab when there is no linked errors
* hide error history tab when there is no history
* add some readonly
2023-06-01 15:03:13 +02:00
Christophe Monniez
ffe12182ab [IMP] runbot: open frontend_url in a new tab 2023-06-01 15:03:13 +02:00
Christophe Monniez
430a526b5c [FIX] runbot: limit triggers in additionnal_setup
When testing with a populated db, this assertion fails because of
additional repositories.
2023-06-01 15:03:13 +02:00
Christophe Monniez
82c6b22e77 [FIX] runbot: typo in test tag 2023-06-01 15:03:13 +02:00
Christophe Monniez
2e002c2dd7 [IMP] runbot: improve the build errors wizard
With this commit, the wizard now allows to set a fixing PR and/or a
commit text to multiple errors.
2023-06-01 15:03:13 +02:00
Christophe Monniez
388eeb377a [IMP] runbot: improve build errors views
* add a link to the fixing PR on github
* add a warning ribbon on test-tagged errors
* show different colors in tree view to spot fixed PR's
* add some search filters
2023-06-01 15:03:13 +02:00
Xavier-Do
231febab96 [FIX] runbot: fix stats.js 2023-05-08 13:26:07 +02:00
Xavier-Do
04760491a2 [FIX] runbot: hide wakeup action from public users
The initial idea to have a wakeup for public users stopped being viable
due to some abuse of the system, maybe unintentional crawling of
some build page but still, this feature will now be limited to internal
users only.
2023-05-02 13:32:43 +02:00
Xavier Morel
6bc6dd77ab [FIX] runbot_merge: mismatch can contain non-str values
The mismatch diff attribute contains values from the in-db object and
the github PR structure, some of which are explicitly *not*
strings (e.g. the squash flag, possibly the commits # in the future).

As a result, when the squash-flag of a PR differs from the actual the
formatting for diffing blows up, because difflib can't handle
non-strings.

Stringify values between passing them to `format_items`, this way the
string operations on names and values should work correctly.
2023-04-17 08:27:57 +02:00
Xavier-Do
e3d87b5b5d [IMP] runbot: improve test-tags support
The current post_install build mecanism is using extra params to give
test-tags. Unfortunately this disables the support for auto tags
and this have to be done manually. This means that auto tags are in the
build extra-params and not dynamic at rebuild of a post_install.

Also, using extraparams in the post install creation was removing
extra_params comming from custom trigger.

With this commit, the test-tags can be given inside config_data
and will be combined with config step test-tags and auto-tags.

This was an opportunity to simplify the logic.

This commit also fixes the test_install_tags that was broken.
2023-04-06 11:33:01 +02:00
Xavier-Do
2ad188201b [IMP] runbot_merge: speedup frontend page
The mergebot page become a bit slow with the years, it is time to make
small optimisation to speed up thinks a little.

Note: all changes where applied modifying the views or adding index by
hand. There is still room for improvement but it would need more in
depth refactoring, mainly adding specialized computed fields to
enable a better batching.

The first issue was using branch.staging_ids

    branch.staging_ids.sorted(lambda s: s.staged_at, reverse=True)[:6]

The number of staging_ids is increasing and prefetching + sorting all
of them is slow.

The proposed solution is to replace it by a search, not ideal, a
specialized compute field may be a good idea, but this is a quick fix
that can be done editing a view.

    branch.env['runbot_merge.stagings'].search([('target', '=', branch.id)],order='staged_at desc', limit=6)

Other changes are just index on critical columns.

Before changes, /runbot_merge page takes ~5s to load
After changes,  /runbot_merge page takes ~1s to load

Small note: note 100% sure that runbot_merge.batch.target was useful
2023-04-05 09:05:50 +02:00
Xavier-Do
43d5cc9d7e [IMP] runbot: add some anchors in nginx 2023-04-04 10:43:58 +02:00
Xavier-Do
e88f679c87 [FIX] runbot: fix python step wakeup 2023-03-30 13:05:35 +02:00
Xavier-Do
8c2e7a5781 [FIX] runbot: manage case when there is no start
SInce the previous version the build end is written when going in any
done state. This means that when a build is skipped, it has a end
but no start.

Adapat the build dime to manage this use case.
2023-03-29 14:58:14 +02:00
Xavier-Do
9024594df2 [FIX] runbot: use request instead of self 2023-03-24 17:06:29 +01:00
Xavier-Do
f7d29f87a4 [FIX] runbot: fix build end
Previous fix was not enough because based on global_state, meaning
that the build if the build goes waiting, it need to update its end
anyway.
2023-03-24 13:09:19 +01:00
Xavier-Do
0edc0bce3a [FIX] limit /force route to advanced users
The force buttons were hidden because unfortunately miss used as a
rebuild in some case instead. The position of the button was to obvious
and used as a "magic fix" when the intended behavior was only for really
specific cases.

Unfortunately the routes were know and still used manually. This commit
blocs the access giving a message to ask for the group if needed.

Those feature would benefit for some documentation.
2023-03-24 11:30:13 +01:00
Xavier-Do
30c74e2434 [FIX] runbot: update build end 2023-03-24 10:26:20 +01:00
Xavier-Do
ee58a93e9a [IMP] runbot: avoid link to killed build
When a build is created, it will first check for another
build having the same params. It is usually a good idea to avoid
to much load. In some case, a build can be found, but a killed one.

This is not what we want:
The first scenario is to consecutive force push,
commit1 -> commit2 -> commit1

The build of commit1 may be killed because of commit2, then when
forcepushing commit1 again, it will be linked to a killed build.

A even more problematic problem was discovered because of a delay In
odoo/odoo repo hook. An odoo-dev/odoo 16.0-... branch was discovered
first using this commit, and a build was created.
Then, the branch was forcedpushed and the build was killed.
Finally, the 16.0 commit was discovered, and was linked to the killed
build. This was mainly an issue because the build was a template.

With this changes, the 16.0 would have created a new build, not linking
to a killed one.

Note that linking to a red build is not an error. Only a killed one.
2023-03-24 10:26:20 +01:00
Xavier-Do
3e5d5e88a1 [FIX] runbot: global_state not written in compute 2023-03-23 16:56:34 +01:00
Xavier-Do
0d29643d52 [IMP] runbot: add a separate pending count
The assigned build are in the same count of the pending build. This can
sometimes create a false queue, because you can have 1000 pending builds
on one host, this doesn't mean that a new standard build cannot be
immediatly taken by another host. This is mainly to hide the false queue
created by the full charge zfs build currently running and creating
~400 assigned build.
2023-03-23 16:33:24 +01:00
Xavier-Do
74f0c8e1ad [FIX] runbot: fix wakeup 2023-03-23 14:44:08 +01:00
Xavier-Do
2acca91a97 [FIX] runbot: fix missing return
This was missing to make python (and other non docker) steps really
blazing fast
2023-03-23 14:17:02 +01:00
Xavier-Do
b078275c94 [IMP] runbot: add commit_export_ids 2023-03-23 13:32:50 +01:00
Xavier-Do
2488228b7b [IMP] runbot: make params work in create multi
The main motivation is to allow to create params from data
the "new" method was called with a value list instead of a dict.

Also, makes it possible to update params when the registry is not laoded
2023-03-23 13:32:50 +01:00
Xavier-Do
d164d9a745 [IMP] runbot: host improvement 2023-03-23 13:32:50 +01:00
Xavier-Do
d2f9330043 [REF] runbot: various refactoring
The initial motivation is to remove the flush when a log_counter is
written. This flush was initially usefull when the limit was in a
psql trigger, but finally add a side effect to flush everything before
starting the docker. This was limiting concurrent update after starting
the docker, but we still have no garantee that the transaction is
commited after starting the docker. The use case where the docker is
started but the transaction is not commited was not handled well and was
leading to an infinite loop of trying to start a docker (while the
docker was already started)

This refactoring returns the docker to the scheduler so that the
schedulter can commit before starting the docker.

To achieve this, it is ideal to have only one method that could return
a callable in the _scheduler loop. This is done by removing the run_job
from the init_pending method. All satellite method like make result
are also modified and adapted to make direct write: the old way was
technical debt, useless optimization from pre-v13.

Other piece of code are moved arround to prepare for future changes,
mainly to make the last commit easier to revert if needed.

[FIX] runbot: adapt tests to previous refactoring
2023-03-23 13:32:50 +01:00
Xavier-Do
688900edb1 [FIX] runbot: remove hardcoded runbot_logs 2023-03-23 13:32:50 +01:00
Xavier-Do
44ec541e32 [IMP] runbot: docker failure management 2023-03-23 13:32:50 +01:00
Xavier-Do
0e14e6d922 [IMP] runbot: serealisation imps
Trying to log when the transaction is in error is useless and create
noise in the logs.

Flushing is also useless there now that we have the local logs,
and it makes the error confusing since the error does not come from the
log_counter update but from the update of the global state on the
parents global_results.
2023-03-23 13:32:50 +01:00
Xavier-Do
937747caec [IMP] runbot: remove dead code 2023-03-23 13:32:50 +01:00
Xavier-Do
c965c2c35a [IMP] runbot: add message queue
Message queue squeletton for future changes
2023-03-23 10:09:12 +01:00
Xavier-Do
2579a2d3fe [FIX] runbot: fix run step for no install config 2023-03-22 16:17:30 +01:00
David James
7477b4cde9 [IMP] runbot_builder: support addons_path parameter
Currently, the addons_path is intuited from the location of the script, with the assumption that the only custom addons path will be runbot itself.

This commit introduces an addons_path parameter to support custom deployments. If the parameter is not set, the existing behaviour is unchanged.
2023-03-13 10:27:59 +01:00
Christophe Monniez
6fdd35ed50 [FIX] runbot: avoid empty negative test tags
When parenting a build error, if a test_tag is set on it, the tag is
transferred to the parent and cleared to an empty string.
In that case, a single `-` appears in the disabling tags and leads to an
apocalyptic situation ... the runbot builds don't not run any tests.

With this commit, the test_tags is set to `False`.
2023-02-28 12:16:07 +01:00
David James
1532bcab0f [FIX] fix typo in selection field value 2023-02-28 11:58:34 +01:00
David James
fa3808bf1e [FIX] runbot: fix dictionary changed size during iteration 2023-02-28 11:16:46 +01:00
Xavier-Do
f5caa87da1 [FIX] runbot: fix time related test 2023-02-28 10:59:47 +01:00
Xavier-Do
d3e3228921 [FIX] runbot: get_module remove prefix
Since removeprefix was not available in ubuntu 20.04, a easier alternative
using rebase was used.

The initial assumption was that the prefix would look like `odoo/addons/`
and won't be in the filename.

When the repo is enterprise, the prefix is `enterprise/` meaning that
module name ending with `enterprise` will be truncated

`repo._get_module('enterprise/mail_enterprise/static/src/widgets/form_renderer/form_renderer.js')`

will output

`mail_static`
2023-02-28 10:59:47 +01:00
Xavier Morel
5e1048c6da [IMP] runbot_merge: logging of gh requests
The loggers would only print the "tail" of the path, not including the
repo name, or the `/repos` prefix.

While this made logs shorter, it was not intentional and made
debugging some issues on endpoints harder than necessary as the calls
had to be adjusted mentally, which is completely unnecessary.
2023-02-21 15:11:59 +01:00
Xavier Morel
6e1fc61781 [IMP] runbot_merge: add json & requests to server actions context 2023-02-20 10:13:05 +01:00
Xavier Morel
23e1b93465 [FIX] runbot_merge: a few issues with updated staging check
1cea247e6c tried to improve staging
checks to avoid staging PRs in the wrong state, however it had two
issues:

PR state
--------

The process would reset the PR's state to open, but unless the head
was being resync'd it wouldn't re-apply the statuses on the state,
leading to a PR with all-valid statuses, but a missing CI.

Message
-------

The message check didn't compose the PR message the same way PR
creation / update did (it did not trim the title and description
individually, only after concatenation), resulting in a
not-actually-existing divergence getting signaled in the case where
the PR title ends or the description starts with whitespace.

Expand relevant test, add a utility function to compose a PR message
and use it everywhere for coherence.

Also update the logging and reporting to show a diff of all the
updated items (hidden behind a `details` element).
2023-02-14 13:45:28 +01:00
Xavier Morel
9c6380c480 [FIX] conftest: there can be some delay on repo content initialization
Seems to be a new github weirdness: when forking a repo, when hitting
it fast enough it's apparently possible to see the repository in an
incomplete state (without its content).

Obviously this causes tests to fail.

Complete the post-fork testing by listing the branches of the fork,
and considering the repo created once we see a non-empty list (the
source repo should always have at least one branch, which should be
copied over when forking).
2023-02-14 13:45:28 +01:00
Xavier-Do
f7a12c6359 [FIX] runbot: fix fetch tests 2023-02-06 10:16:07 +01:00
Xavier Morel
907c6072d1 [FIX] runbot_merge: cancel master staging on freeze
If there are bump PRs anyway: the bump commits will cause the
forward-port of the staging to fail, so might as well clearly notify
everybody of the issue if there is a pending staging, and not waste
too much time waiting for a staging which can not succeed.

We could also cancel stagings when there's no bump PR, but it's not
clear that there's any reason to do so: if we didn't touch any master
branch, there's no reason for the staging to fail, or to otherwise
cancel it.

And obviously we can't have staged anything on the new branch so
there's nothing to cancel.

Part-Of: #718
2023-01-25 12:25:45 +01:00
Xavier Morel
8d7d6302d3 [FIX] runbot_merge: make freeze wizard labels lookup not shit
I DECLARE BANKRUPTCY!!!

The previous implementation of labels lookup was really not
intuitive (it was just a char field, and matched labels by equality
including the owner tag), and was also full of broken edge
cases (e.g. traceback if a label matched multiple PRs in the same repo
because people reuse branch names).

Tried messing about with contextual `display_name` and `name_search`
on PRs but the client goes wonky in that case, and there is no clean
autocomplete for non-relational fields.

So created a view which reifies labels, and that can be used as the
basis for our search. It doesn't have to be maintained by hand, can be
searched somewhat flexibly, we can add new view fields in the future
if desirable, and it seems to work fine providing a nice
understandable UX, with the reliability of using a normal Odoo model
the normal way.

Also fixed the handling of bump PRs, clearly clearing the entire field
before trying to update existing records (even with a link_to
inbetween) is not the web client's fancy, re-selecting the current
label would just empty the thing entirely.

So use a two-step process slightly closer to the release PRs instead:

- first update or delete the existing bump PRs
- then add the new ones

The second part is because bump PRs are somewhat less critical than
release, so it can be a bit more DWIM compared to the more deliberate
process of release PRs where first the list of repositories involved
has to be set up just so, then the PRs can be filled in each of them.

Fixes #697
2023-01-25 12:25:45 +01:00
Xavier Morel
ae53c87fc9 [FIX] runbot_merge: allow adding and removing release PR lines
In order to support partial freezing, we need the ability to remove
some of the release lines for the repos we don't want to
freeze (e.g. because they don't use per-version branches).

This subsequently means we need the ability to *create* new lines if
we fucked up and removed one we should not have. Alternatively the
freeze meat-bot could cancel the entire thing and redo the wizard but
that seems harsh and mean, so don't do that.

Fixes 0f3647b7c7 which specifically
mentioned partial freeze then proceeded to make them entirely
impossible anyway.

Part of #718
2023-01-25 12:25:45 +01:00
Xavier Morel
fb60c38731 [IMP] runbot_merge: add color key to freeze wizard
Was difficult to understand what the colors meant on the required PRs.

Part of #718
2023-01-25 12:25:45 +01:00
Xavier Morel
1cea247e6c [FIX] runbot_merge: sync PR target and message on check
Previously the mergebot would only sync the head commit, but synching
more is useful.

Also update the final sanity check on staging:

- as with check, update the message & target branch
- reset PR state and post a message when updating message instead of
  doing so silently

Note: maybe only fail the staging if the message is updated *and*
relevant to staging (aka there's a merge method and it's not
`rebase`)?

Fixes #680
2023-01-25 12:25:45 +01:00