Commit Graph

546 Commits

Author SHA1 Message Date
Xavier-Do
ecd105218c [IMP] runbot: parse warning and critical 2023-01-13 14:27:17 +01:00
Xavier-Do
2b8242cf42 [IMP] runbot: default empty value for result 2023-01-13 14:27:17 +01:00
Xavier-Do
6892546d94 [IMP] runbot: fail before contacting github if user cannot write on team 2023-01-13 14:27:17 +01:00
Xavier-Do
cd4c6b5e43 [IMP] runbot: better configure upgrade descriptions 2023-01-13 14:27:17 +01:00
Xavier-Do
37df18d54d [FIX] runbot: use params db_name when provided for upgrades 2023-01-12 11:54:53 +01:00
Xavier-Do
c9c64bccc7 [FIX] runbot: fetch member should be able to access token for users 2023-01-11 13:35:02 +01:00
Xavier-Do
06f9f62b1f [FIX] runbot: fix was alive 2023-01-11 12:41:39 +01:00
Xavier-Do
16fa0e1ad8 [IMP] runbot: add freeze list feature 2023-01-11 11:20:29 +01:00
Xavier-Do
fdb144a3af [IMP] runbot: allow to ignore forward-ports for a team 2023-01-10 10:47:59 +01:00
Xavier-Do
56c08be631 [FIX] runbot: filter numerical branch names
Right now, a branch with a numerical name will be added to the database,
but it can conflict with pr since the name of a pr is a number.
This means that a unique (name, repo_id) constraints can be broken.
We could use the 'is_pr' in the unicity constraints to avoid this issue
but searching on branch name will give confusing result if some of them
can be numerical.

Moreover, a runbot branch name should start with the version name
meaning that a numerical branch name was a bad idea from the begining.
2022-12-27 10:37:30 +01:00
Xavier-Do
6d3663e9f6 [IMP] runbot: disable auto-tags from custom trigger 2022-12-26 12:31:09 +01:00
Xavier-Do
910843f06d [FIX]: fix codeowner
A corner case was discovered when a pr is open but the pr is not in the
codeowner commits.
2022-12-24 12:39:55 +01:00
Xavier-Do
ad573d64cf [FIX] runbot: manage falsy path_glob 2022-12-22 14:40:13 +01:00
Xavier-Do
5540f84667 [IMP] runbot: add priority queue for debuging
The main idea is to allow some build to use an extra slot from all host
if a bundle is in priority mode.

This is mainly for quick step debuging, mainly when modifying python
steps when the runbot is fully loaded.

This will also be usefull to concurrency test, by starting a build on
each host at the same time, even when some host are already fully loaded
2022-12-22 14:26:10 +01:00
Xavier-Do
356843c680 [IMP] runbot: reduce concurrency
Right now, multiple build are read when managing build to schedule.

This is not usefull since the transaction is commited between each
of them. Moreover, the read build can be written from another host
adding another possibility to have a conccurent update.

Removing the prefetch_ids may help a little.
2022-12-22 11:25:04 +01:00
Xavier-Do
205d299c7d [IMP] runbot: add trigger list to team 2022-12-22 10:47:24 +01:00
Christophe Monniez
9ccf5a2c0e [IMP] runbot: add a reassign team wizard 2022-12-21 15:20:42 +01:00
Xavier-Do
c83f89fc23 [IMP] runbot: documentation and views imps
Add a small documentation for users, mainly about teams and codeowners.
Improves some views and hide some menu_items to keep interface easier
to navigate.
2022-12-21 13:00:42 +01:00
Xavier-Do
ada69222c9 [IMP] runbot: add an action to reassign team automaticaly 2022-12-21 10:58:30 +01:00
Xavier-Do
af7a0751fc [FIX] runbot: avoid recursion in create steps
Since the custom create was introduced, if a config is added in the
config data of a create step, the config can be dynamic. If the given
config contains a create step, this become recursive.

This is fixed in this commit by:
- Checking the parent_path depth in add_child. This will also work for
  python config.
- Consuming the params when adding the child
- Also cleanup the base custom multi config to use a specific step
2022-12-20 09:30:08 +01:00
Xavier-Do
92ed23c237 [FIX] runbot: manage team without github logins 2022-12-19 11:14:07 +01:00
Xavier-Do
25557e6116 [FIX] runbot: manage team without github logins 2022-12-19 11:09:15 +01:00
Xavier-Do
a67e51e600 [FIX] runbot: revert log_access and write in cache
Removing log_access has as side effect to add the foreign key to the
create_uid and write_uid fields. This is quite slow and will slow the
insert

Removing the fields is also not an good idea on such a large table

Puthing the value in cache and flushing should do the trick.
2022-12-16 16:59:59 +01:00
Xavier-Do
0d61f6c93b [IMP] runbot: add github login
This idea was postpone for a while since this was most a mergebot
responsability but having the github login of the user will help
for some team feature requests.

The main one is to only ping a team if the pr was not opened by a member
of the team. We want to let the team manager manage that as much as
possible so the team manager group will be able to write the user
github login (as well as the user himself) and add a list of non user
github_login to consider if not all user have a account on runbot.

This commit also improves the views for team managers.
2022-12-16 15:32:39 +01:00
Xavier-Do
3f2bf0112f [FIX] runbot: custom log_access fields shouldn't be automatic 2022-12-15 14:20:11 +01:00
Xavier-Do
02ed33c498 [FIX] runbot: keep log create date 2022-12-15 13:38:55 +01:00
Xavier-Do
0d6ec3e038 [FIX] runbot: fix invalid db_name log local
If the db_name does not stat with a build ind (or at least an int)
the query will fail because of 'local_pg_cursor'

Since a database can be create with an invalid name from the db manager
but still log in the runbot_logs, we need to manage all format.
Also add a limit to catchup if the db is full of logs, to avoid a Memory
error.
2022-12-15 12:54:50 +01:00
Xavier-Do
a6ce3dceb0 [IMP] runbot: add step description 2022-12-12 10:35:23 +01:00
Xavier-Do
2a82f3c1f7 [FIX] runbot: fw-bot codeowner fix 2022-12-01 11:29:25 +01:00
Xavier-Do
ee9b3b7570 [FIX] runbot: avoid excessive log_counter updates 2022-11-30 14:07:55 +01:00
Xavier-Do
3664eabd90 [FIX] runbot: manage empty dbname 2022-11-28 14:38:05 +01:00
Christophe Monniez
2cad0542f4 [IMP] runbot: queue build logs in a local database
Before the commit the build ir_logging was sent from the build instance
to the main runbot ir.logging table. As the number of runbot hosts
increases, it introduce a lot of concurrency.
e.g.: 80 hosts with 8 builds means 640 instances trying to insert
records in the ir.logging table.

With this commit, a special database is used on the builder host in
order to receive ir.logging's from the build instances.

Regulary, the table is emptied by the builder and the logs are inserted
in the runbot leader ir.logging table.
2022-11-28 06:46:49 +01:00
Xavier-Do
66e37b9323 [FIX] runbot: renable blacklist 2022-11-25 10:41:52 +01:00
Xavier-Do
891d2d71e8 [IMP] runbot: add draft detection form titl 2022-11-24 16:24:42 +01:00
Christophe Monniez
3a9832d747 [IMP] runbot: add a close error wizard
When marking multiple build error as fixed, it's sometimes necessary to
explain why it was decided to close the error. When working with a few
errors, this can be done manually ... But most of the time we want to
close a lot of false negatives in batch.

With this commit, a simple wizard is made available that will post a
reason in the chatter of the build_errors.
2022-11-24 15:18:14 +01:00
Xavier-Do
1e8e059734 [FIX] fix codeowner 2022-11-24 15:17:53 +01:00
Xavier-Do
f2d71a0b79 [FIX] runbot: fix pull info 2022-11-21 16:48:54 +01:00
Xavier-Do
2e77a55ddb [IMP] runbot: add codeowner management 2022-11-21 16:32:25 +01:00
Xavier-Do
410a01d13b [REF] runbot: move teams stuff 2022-11-21 16:32:25 +01:00
Xavier-Do
2ca7a3de6e [FIX] runbot: trigger with no config fallback 2022-11-09 12:29:00 +01:00
Xavier-Do
f72c4a8baa [FIX] runbot: fix cleanup 2022-11-08 15:19:33 +01:00
Xavier-Do
e0856b2245 [IMP] runbot: improve build_error management
The build error view was unstructured and contains to much information.

This commit organize fields in groups and also validate some
modification on records in order to avoid build error manager to
disable test-tags by mistake.

An error cannot be deactivated if it appeared less than 24 hours ago to
avoid disabling a non forxardported pr that will fail the next nightly
generating another build error.

Test tags can only be added or removed by administrators.

Also adds a menu for easier User managerment

Also fixed the dname search and display.
2022-11-08 14:43:43 +01:00
Christophe Monniez
964a88cb36 [FIX] runbot: take number of builds from config data
Oversight from previous improvement in b79c4a5a52.
2022-11-07 10:12:15 +01:00
Xavier ALT
fffc27d2fa [FIX] runbot: fix creation of new runbot.version from the backend
Traceback (most recent call last):
  File "/home/odoo/src/odoo/15.0/odoo/addons/base/models/ir_http.py", line 237, in _dispatch
    result = request.dispatch()
  File "/home/odoo/src/odoo/15.0/odoo/http.py", line 687, in dispatch
    result = self._call_function(**self.params)
  File "/home/odoo/src/odoo/15.0/odoo/http.py", line 359, in _call_function
    return checked_call(self.db, *args, **kwargs)
  File "/home/odoo/src/odoo/15.0/odoo/service/model.py", line 94, in wrapper
    return f(dbname, *args, **kwargs)
  File "/home/odoo/src/odoo/15.0/odoo/http.py", line 348, in checked_call
    result = self.endpoint(*a, **kw)
  File "/home/odoo/src/odoo/15.0/odoo/http.py", line 916, in __call__
    return self.method(*args, **kw)
  File "/home/odoo/src/odoo/15.0/odoo/http.py", line 535, in response_wrap
    response = f(*args, **kw)
  File "/home/odoo/src/odoo/15.0/addons/web/controllers/main.py", line 1347, in call_kw
    return self._call_kw(model, method, args, kwargs)
  File "/home/odoo/src/odoo/15.0/addons/web/controllers/main.py", line 1339, in _call_kw
    return call_kw(request.env[model], method, args, kwargs)
  File "/home/odoo/src/odoo/15.0/odoo/api.py", line 464, in call_kw
    result = _call_kw_multi(method, model, args, kwargs)
  File "/home/odoo/src/odoo/15.0/odoo/api.py", line 451, in _call_kw_multi
    result = method(recs, *args, **kwargs)
  File "/home/odoo/src/odoo/15.0/odoo/models.py", line 6489, in onchange
    snapshot1 = Snapshot(record, nametree)
  File "/home/odoo/src/odoo/15.0/odoo/models.py", line 6271, in __init__
    self.fetch(name)
  File "/home/odoo/src/odoo/15.0/odoo/models.py", line 6281, in fetch
    self[name] = record[name]
  File "/home/odoo/src/odoo/15.0/odoo/models.py", line 5888, in __getitem__
    return self._fields[key].__get__(self, type(self))
  File "/home/odoo/src/odoo/15.0/odoo/fields.py", line 1054, in __get__
    self.recompute(record)
  File "/home/odoo/src/odoo/15.0/odoo/fields.py", line 1243, in recompute
    self.compute_value(recs)
  File "/home/odoo/src/odoo/15.0/odoo/fields.py", line 1265, in compute_value
    records._compute_field_value(self)
  File "/home/odoo/src/odoo/15.0/odoo/models.py", line 4255, in _compute_field_value
    getattr(self, field.compute)()
  File "/home/odoo/runbot/extra/runbot/models/version.py", line 36, in _compute_version_number
    version.number = '.'.join([elem.zfill(2) for elem in re.sub(r'[^0-9\.]', '', version.name).split('.')])
  File "/usr/lib/python3.8/re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/odoo/src/odoo/15.0/odoo/http.py", line 643, in _handle_exception
    return super(JsonRequest, self)._handle_exception(exception)
  File "/home/odoo/src/odoo/15.0/odoo/http.py", line 301, in _handle_exception
    raise exception.with_traceback(None) from new_cause
TypeError: expected string or bytes-like object
2022-10-21 11:59:55 +02:00
Xavier ALT
1095b25270 [FIX] runbot: fix help msg, 'additionnal_env' is split on semi-column 2022-10-21 11:59:55 +02:00
Christophe Monniez
309aeaa32e [IMP] runbot: speedup build garbage collecting
When the builds directory is filled with a lot of build directories
(around 100000) the garbage collection process may take up to 2 minutes.
The root cause is that each build directory is scanned to clean it up
even if it was already cleaned.

With this commit, a stamp file is used to mark directories that were
already garbage collected.
2022-10-21 11:32:46 +02:00
Xavier-Do
e27a1b8f71 [IMP] runbot: cleaup settings view 2022-10-21 10:32:40 +02:00
Xavier-Do
0287dcaab7 [IMP] runbot: update install documentation 2022-10-21 10:32:40 +02:00
Xavier-Do
303638e507 [FIX] runbot: don't hardcode user odoo 2022-10-21 10:32:40 +02:00
Xavier-Do
e4af8d7b6b [FIX] runbot: remove fqdn calls 2022-10-21 10:32:40 +02:00
Xavier-Do
785e2dc3bc [IMP] runbot: don't use local odoorc 2022-10-21 10:32:40 +02:00
Xavier-Do
16e71d064a [FIX] runbot: remove useless/invalid check 2022-10-21 10:32:40 +02:00
Xavier-Do
b79c4a5a52 [IMP] runbot: various build_config improvements 2022-10-21 10:32:40 +02:00
Xavier-Do
eaf8964701 [IMP] runbot: change trigger start logic
and add sequence on project
2022-10-21 10:32:40 +02:00
Xavier-Do
02eb8014a3 [IMP] runbot: make build_error triggers searchable 2022-10-21 10:32:40 +02:00
Xavier-Do
70e8b15690 [IMP] runbot: add websocket support 2022-08-23 16:30:24 +02:00
Xavier-Do
e91611b0f8 [FIX] runbot: reduce load when builder are idle
The sleep 1 was ok with a few builder bur regarding the number of
request on the database when no build are running, this become
problematic.

An ideal solution would be to detect if
-> me managed some testing build
-> There is load (pendings)

In both case, we don't want to sleep to much.
In other cases, we may want to wait a little longer.

A simple quick fix will just wait longer in all cases.
2022-08-23 15:54:50 +02:00
Xavier-Do
b44ed5f7a6 [IMP] runbot: active trigger and filter manual
- dont display manual trigger having a team if not member of that team
- add an active field on triggers
2022-08-17 16:50:23 +02:00
Xavier-Do
a382977d39 [IMP] runbot: add teams on trigger.
This will be mainy usefull in trigger categories custom view and for
dashboards.
2022-08-17 11:40:56 +02:00
Christophe Monniez
2dce6d048e [IMP] runbot: freeze files modification time at export
When exporting a commit, it will be useful to freeze the modification
time of the exported files. The idea behind that is to pre-generate
bundles at the install time of the Odoo instances, that way when running
the post install tests, the bundles does not have to be generated for
each test.
2022-07-26 13:57:18 +02:00
Xavier-Do
9fb7e94de9 [IMP] add load info page 2022-07-25 12:10:48 +02:00
Xavier-Do
a52c14957e [IMP] runbot: log manual batch and redirect
1. Log who started a manull build
2. redirect if the batch given in a build url is not containing this
build (manual url)
2022-07-12 13:39:44 +02:00
Christophe Monniez
e1161a03e6 [FIX] runbot: avoid sudo when using pip
In bd4cf76b7 a new argument `--progress-bar off` was added to the pip
command line. Unfortunately, it appears that this argument is not
available in some pip versions, typically the one from Ubuntu Bionic
(9.0.1) [0]. The argument appeared in version 10.0.0 [1].

Also, in the DockerFile, we allow the user to use `sudo` for
`/usr/bin/pip3` and the main reason for that was to avoid having `pip`
to re-install the packages that are pre-installed by the distribution
package manager.

That said, it appears that since pip 10.0.0 when installing packages
locally for the user, pip now properly detects distribution installed
packages.

So, as the solution to fix the progress bar issue is to upgrade pip to a
newer version, we can take advantage of it to get rid of the `sudo`.

Besides the fact that `--user` is the default on Debian based
distributions [2] , we enforce it in case another distribution is used
in the Dockerfile.

[0] https://packages.ubuntu.com/bionic-updates/python3-pip
[1] https://pip.pypa.io/en/stable/news/#id744
[2] https://wiki.debian.org/Python#Deviations_from_upstream
2022-07-06 09:48:20 +02:00
Xavier-Do
bd4cf76b76 [FIX] runbot: hide progressbar for pip 2022-07-01 15:47:13 +02:00
Xavier-Do
03732638f3 [IMP] runbot: add an automatic generated message for upgrade exceptions 2022-07-01 15:47:13 +02:00
Christophe Monniez
78697d2acb [IMP] runbot: use docker python lib
While using the docker cli as a subprocess was KISS and convenient,
the python Docker SDK is mature, easy to use, available as a Debian
package and much more powerful.

It will permit to monitor the containers memory consumption and will
help to spot memory leaks.
2022-06-30 10:58:38 +02:00
Christophe Monniez
9fb48f48cf [IMP] runbot: improve git gc warning 2022-06-27 09:11:02 +02:00
Christophe Monniez
3f042c44a8 [FIX] runbot: fix gc warning 2022-06-24 08:41:32 +02:00
Christophe Monniez
c006bf953d [FIX] runbot: apply small fixes 2022-06-20 14:37:02 +02:00
Christophe Monniez
250d48e266 [IMP] runbot: schedule git gc on repositories
On the actual runbot deployments, the `git gc` command is handled by a
unix cron. From time to time, some repositories get corrupted and we
suspect that some concurrent action may be involved as stated in
documentation [0].

For those reasons, with this commit, the `git gc` will be run by the
runbot clients themselves in order to avoid concurrent operations.

By default, the first gc will occur a few minutes after the start of the
client and the next gc are scheduled a two hours and a few minutes later.

Also, this commit ensures that the git config is written regularly in
case of change.

[0] https://git-scm.com/docs/git-gc
2022-06-20 10:32:10 +02:00
Xavier-Do
cdaae9b3ed [IMP] runbot: speedup get_refs
Regarding the number of refs in odoo repo (arround 18 million at this
time), the parsing of the date was significant when filtering old refs.

Using unix time allows a direct comparaison without parsing the date,
and improved performance, going from ~7 seconds to ~1.3 seconds.
2022-06-20 10:20:37 +02:00
Xavier-Do
cbfc8401a8 [FIX] runbot: various FIX 2022-06-17 13:52:37 +02:00
Xavier-Do
26a3ad20f1 [IMP] runbot: status management
The status are currently sent by leader when build are created and by
workers on post_commit.

If the leader fails during the preparing of a batch (while creating
builds) the transaction is rollbacked and the status are send again.

The number of status to send makes it quite slow, making the transaction
longuer, and the retry even more expensive. This leads to preparing time
being quite important, sometimes ten minutes after many retries.

This commit proposes to send status in another dedicated transaction.
Since status are sent in batch, we can also try to use a unique session,
and uniquify commit+context status.

This allows to remove the postcommit logic

A further improvement would be to wait before sending status in order to
skip the pending status if the build is verry fast.

An option is also added on the remote to skip the status: if the remote
is a fork, sending the satus on the main remote should be enough.
2022-06-17 12:32:14 +02:00
Christophe Monniez
b997119588 [FIX] runbot: avoid children in team errors
When an error is assigned to a team and the error is a child of another
one, the two errors are displayed on the team dashboard and adds
confusion.
2022-06-17 10:40:57 +02:00
Xavier-Do
47c12fe00b [FIX] runbot: add MULTILINE to ReProxy 2022-06-15 12:30:41 +02:00
Xavier-Do
bf0df23a3d [FIX] runbot: add findall to ReProxy 2022-06-15 12:13:34 +02:00
Xavier-Do
15e4fa76c5 [FIX] runbot: add VERBOSE to ReProxy 2022-06-15 09:12:17 +02:00
Xavier-Do
d466887f08 [REF] runbot: adapt statistic models
Initial model was a (build_id, key, value) where key is in fact a two
part information: `category.key` (where key is usually a module)

This means that for each module, we will have one entry per
modules*category.
We have between 200 and 400 modules per build * 4 keys -> around 1000
entries per build.

The hudge amount of total entries lead to a fast overflow of the table
sequence + this create important indexes.

Also, most of the time, the js will manage the display of stats meaning
that python will transform
(build_id, category.key, value)
into
(build_id, {key: value}) for one category

A new model makes use of a json field to store values for different
modules in a json dict, on entry per category*build. (4 entry per build)

The table will be renamed and migrate later
2022-06-13 12:42:13 +02:00
Xavier-Do
87366de213 [IMP] runbot: add log on separator 2022-06-10 14:56:13 +02:00
Xavier-Do
77f8c29091 [IMP] runbot: check restore result 2022-06-10 14:02:58 +02:00
Xavier-Do
1879f2c235 [IMP] runbot: forced host improvement 2022-06-10 13:26:47 +02:00
Xavier-Do
f654738b04 [FIX] runbot: fix nginx static 2022-06-09 15:53:41 +02:00
Xavier-Do
a6fa7b8023 [FIX] runbot: fix leader pull_info_failures
The initial pull_info_failures feature was introduced before the separate
process for the leader. Since this is here for a rare case, this
wasn't tested since then.

Testing the new infrastructure without token, there was many
pull_info_failure, leading to an error on cleanup.
2022-06-09 08:43:28 +02:00
Christophe Monniez
33ab9cdce6 [FIX] runbot: open nginx file as text
Continuation of previous fix 6902a7e24.
2022-06-08 15:33:26 +02:00
Christophe Monniez
6902a7e242 [FIX] runbot: cast markup into str before saving
When rendering templates for git configuration and nginx configuration,
the templates are rendered as Markup, while a byte-like object was
expected for the saving.

With this commit, the Markup is casted into str and the files are saved
as text files.
2022-06-08 15:19:08 +02:00
Xavier-Do
6b79d5b96c [REM] runbot: remove runbot_domain
This parameter can be replaced by web.base.url when needed.
2022-06-07 10:54:48 +02:00
Xavier-Do
272096b810 [REM] runbot: remove non nginx support
Some non tested code was making running build available without
an nginx config (using port and local address instead of using a proper
dns/nginx config).
Removing this part to reduce complexity.
2022-06-07 10:31:09 +02:00
Xavier-Do
3071f5a6db [IMP] runbot: adapt for odoo 15.0 2022-06-02 10:50:42 +02:00
Xavier-Do
78f050b132 [IMP] runbot: adapt for 14.0 2022-03-23 09:49:53 +01:00
Xavier-Do
8c120ac0d9 [FIX] runbot: don't log invalid bases on autorebase 2022-03-17 09:30:03 +01:00
Xavier-Do
c4d0b56972 [FIX] runbot: dont duplicate export
Even if it isn't really problematic, avoiding to duplicate exports
will make the table slighly smaller, especially when considering running
builds.
2022-03-16 09:44:58 +01:00
Xavier-Do
3985332a34 [IMP] runbot: commit exports
Source cleanup will check in multiple place for potentially used
commits. This maked the cleanup logic complex, plus limit the usable
commits in python steps. The current use case is to export the mergebase
commits.

The poposed solution is to save the exported commit and clean this list
when the build is done. All commit in this list cannot be cleaned.
2022-03-16 09:05:39 +01:00
Xavier-Do
ea3191a8d0 [IMP] runbot: add javascript support for JsonDictFields
The main motivation is to be able to see existing config_data on params
and edit configuration on triggers.

A simple version would be to use the `FieldChar` with a simple
`JSON.stringify` formater, but having some indent on field make it
easier to read and edit.

The multiline need make it closet to the `FieldText` than the `FieldChar`
but the reset makes the browser freeze.
2022-03-14 15:27:11 +01:00
Xavier-Do
4e6ae00f23 [IMP] runbot: custom trigger wizard
The actual way to add a custom multibuild on a dev bundle will be:
- use the multibuild wizard to create 2 configs and 2 steps
- add this config on the bundle
- customize this config to make it fatser by installing/restoring...

The usefull parameters are the number of build, the test tags and
modules to install.

Another usefull operation is to restore a dump instead of installing,
especially for post install test only breaking on full databases.

This commit proposes to replace the multi build wizard with a custom
trigger wizard. The main idea is to have generic config, parametrized by
config_data. The wizard will only help to generate the corresponding
config_data.
2022-03-11 14:00:47 +01:00
Christophe Monniez
a931af2fed [IMP] runbot: fully garbage collect old builds
When a build is garbage collected and the local folders are cleaned, the
log files are kept to allow build errors investigations.

With this commit, a full local cleanup will be done 365 days after the
garbage collect.  This means that the build dir will be completely
removed.  The default number of days can be tweaked with the
`runbot.full_gc_days` ir.config_parameter.

Also, the _local_cleanup method can be called with a `full` boolean
parameter to force a full cleanup. e.g.: when called in a config step.

While at it, the res_config_settings view is a bit reorganized.
2022-02-10 13:39:50 +01:00
Xavier-Do
ea5feb114a [FIX] runbot: fix codowner match 2022-02-07 13:03:26 +01:00
Xavier-Do
2c29a6b94b [IMP] runbot: add validation for domain 2022-02-07 09:31:19 +01:00
Xavier-Do
ebb342ea0c [IMP] runbot: link on reference batch 2022-02-07 08:59:28 +01:00
Christophe Monniez
ffd432311f [FIX] runbot: removing failed git export
When a git archive fails, the partially exported source tree is left in
place. If another builds tries to use the same commit, the tree is not
exported anymore as the directory exists. This leads to non
deterministic behaviors.
2022-01-27 10:19:54 +01:00
Xavier-Do
8cdb8c9e5b [FIX] runbot: small fixed to ease deployment 2022-01-27 10:08:39 +01:00
Xavier-Do
b5d01797cb [IMP] runbot: add single branch repo
Right now single version repo like upgrade are managed using
a regex to limit name prefix, this avoid grouping branches when
mergebot wont be able to merge them together but the ci can be painfull
since the branch needs to be renamed (closing existing pr) or a manual
operation to move the branch into a new bundle must be performed.
This commit proposes to replace the forbidden_regex mechanism with
an explicit single_version mechanism.
In this case the reference name will be automatically prefixes with
the version. The prefix contains `---` to indicate that some
magic was applied.
2022-01-11 15:14:08 +01:00
Xavier-Do
9a3c11b09f [IMP] runbot: use custom layout and improve views
Runbot layout modifies the website/portal base layout to remove navbar,
footer, overides some custom styles. A lot of assets are loaded but not
used. The only real usefull elements are base assets (bootstrap, ...)
and the login button.

Migrating to the next version of odoo is usually painfull because some
xpath may break, extra element added, or some style change may break the
page, needing to add more and more xpath, css rules, ... for very little
benefits.

This cleanup creates a custom base layout for runbot independant from
base odoo templates.

Also add a breadcrumb, navigation arrow, and improve batch links
2022-01-11 09:11:48 +01:00
Xavier-Do
460e26bafc [IMP] runbot: add populated data 2022-01-11 09:11:48 +01:00
Xavier-Do
fe987cd0f3 [IMP] runbot: some fixes for ps runbot
- searching on number will search for both pr and branche name
- hooks are now using payload to define repo when not given in url
- fixes .git cleaning in repo
    (remove rstrip since it can fail for repo starting with g, i, t)
- recompute base on prepare if base was not found
- remove local_result form write values if there is a single record
    (instead of raising, makes python step easier to write).
- avoid stucked build/loop after removing a step from a config.
- avoid to send ci for linked base_commit
- add a fallback mechanism for base if no master branch is found
- add option on project to avoid to keep sticky running, usefull
    when using a lots of projects
    WARNING: this is a change of default behaviour, need to update
    existing projects.
- always discover new commits for branch matching base paterns.
    This is especially usefull to discover old versions on project with
    low merge frequency.
- always create a batch, event if there is now trigger. This helps to
    notice that commits are discovered
- add line-through on death branches/pr
- manual trigger are now displayed on main page
2022-01-10 15:19:44 +01:00
Xavier-Do
ee715360ff [FIX] runbot: avoid docker state UNKNOWN after slow fetch 2021-12-08 15:06:49 +01:00
Xavier-Do
e22c403102 [IMP] runbot: avoid concurrent write and useless still in use messages 2021-12-08 15:06:49 +01:00
Xavier-Do
0b30b9c104 [IMP] runbot: create a separate process for cron
As for the builder, this give the ability to run the discovery of new
commits and all related logic in a separate process.

This will mainly be usefull to restart frontend without waiting for cron
or restart "leader" without stoping the frontend. This will also be
usefull for optimisation purpose.
2021-12-08 15:06:49 +01:00
Christophe Monniez
91efbec943 [IMP] runbot: add a codeowner model
As a custom codeowner system was successfully implemented in a python
step on our runbot instance, it's now time to have a real model for
that.

This commit adds a skeleton Codeowner model in order to be used for a
basic usage.

This should be improved in the future after some battle testing.
2021-11-25 15:19:26 +01:00
Christophe Monniez
89dcb52215 [IMP] runbot: add a runbot team model
With the increasing usage of runbot to test various things and to take
care of random bugs in tests, the need of a team dashboard arose.

This commit adds a `runbot.team` model. Internal users can be
linked to the team. Module wildcards can be used to automatically assign
build errors to a team at 'build.error` creation.

Also, an upgrade exception can be assigned to a team in order to display
it on a dashboard.

A dashboard model is used to create custom dashboards on the team
frontend page. By default, a dashboard is meant to display a list of
failed builds. The failed builds are selected by specifying a project, a
trigger category (e.g. nightly), a config and a domain (which select
failed builds by default).

The dashboard can be customized by specifying a custom view.

Each created team has a frontend page that displays all the team
dashboards and the errors assigned to the team.

A few other improvement also come with this commit:

 * The cleaned error is now in a tab on the build error form
 * Known errors are displayed as "known" on the build log page
 * The build form shows the config used for the build
2021-11-25 15:19:26 +01:00
Xavier-Do
2b8e5c5a91 [IMP] runbot: remove _log_access on build_stat
_log_access is not usefull, and increase disk space used by stats.
2021-08-04 10:46:59 +02:00
Xavier-Do
39eeb73f71 [FIX] runbot: fix keep_running
Keep running was broken for since 5.0.
This commit fixes the broken logic and adds an index
2021-08-04 10:46:59 +02:00
Christophe Monniez
01371c5270 [REM] runbot: remove dead code 2021-08-04 10:40:13 +02:00
Andrii Skrypka
5b9c20877e [FIX] runbot: correctly get number type value from icp 2021-08-03 09:47:11 +02:00
Xavier-Do
de927d89e7 [IMP] runbot: cleanup and improve hook
When getting pull info, the alive state can be determined easily,
meaning that this field can join the "_compute_branch_infos" familly

Hook was catching some changes made on pr and was conditionnaly updating some fields
and triggering some other operations conditionaly depending on the action flag.
All of the information needed to update the pull info should always be present in the
 payload body, meaning that all fields can be updated at once in case some hook was missed,
 and additionnal operation can be triggered based on fields changes.
2021-07-20 13:24:01 +02:00
Xavier-Do
d9a09806e2 [IMP] runbot: add a static run link for documentation builds 2021-07-20 13:22:57 +02:00
Christophe Monniez
847622552f [IMP] runbot: limit memory usage of containers
In some conditions, it appears that a containerized build can eat up
all memory of the container host. This leads to disturbance of other
builds as the kernel OOM killer enters the dance.

With this commit, the docker ability to limit memory usage of a
container is used. The OOM killer will choose its victim among the
container processes.

The containers memory limit has to be set in the runbot settings. If not
set, no memory limit is used.
2021-07-15 11:15:38 +02:00
Christophe Monniez
363c488c10 [IMP] runbot: add a reviewers field 2021-07-07 11:05:51 +02:00
Xavier-Do
87214a82a6 [IMP] info: log level debug to info
runbot servers are running with a log-level debug in order to have usefull
debug information, but this also causes some noise comming from odoo.

This pr changes most debug to info.
2021-07-02 15:05:42 +02:00
Xavier-Do
011bdc00e2 [IMP] add draft pr management + multiple custom fields
- Add draft pr management to avoid to trigger code owner on draft pr.
- Add check on falsy config on trigger id (avoid crash, usefull to disable trigger)
- Add extra params on custom trigger to avoid to write specific config every time.
- Trigger a new batch automaticaly when updating target/draft
2021-06-29 10:13:34 +02:00
Christophe Monniez
6e5d722882 [IMP] runbot: add a Dockerfile on project
Some projects may use a totally different Dockerfile. In order to avoid
new branches of those projects to automatically build with the generic
default Dockerfile, this commit adds the possibility to configure a
Default Dockerfile on a project.
2021-06-29 09:22:41 +02:00
Christophe Monniez
b015ec6840 [IMP] runbot: show bundles that use a Dockerfile
Before manipulating a Dockerfile, it can be useful to know which bundle
is using it.
2021-06-29 09:22:41 +02:00
Alexandre Fayolle
3c760870b5 [FIX] runbot: bug in duplicate module detection
The test in the original code will never fire because the value searched
for is not in the keys of the dictionary, but in one of the lists which
are in the values. Work around this by maintaining a reverse dictionary
module name -> commit and use this for the test.
2021-05-18 12:37:42 +02:00
Xavier-Do
0a37ff2f90 [FIX] runbot: don't log twice 2021-05-17 14:43:48 +02:00
Xavier-Do
6d4efcb470 [FIX] runbot: avoid failure for duplicate warnings 2021-05-17 14:23:18 +02:00
Xavier-Do
5d3a2de698 [FIX] runbot: don't fail all branch discovery if one pull info fails.
Sometimes a pr pull info can fail.
- Most of the time it is only temporary and it will be successfull on next try.
- In some rare case the pr will always fail (github inconsistency) The pr exist in git but not on github api.
For this rare case, we store the pr in memory in order to unstuck other pr/branches update.
We consider that this error should not remain, in this case github needs to fix the inconsistency.
This is why the runbot model don't handle such a case for now.
Another solution would be to create the pr with fake pull info. This idea is not the best one
since we want to avoid to have many pr with fake pull_info in case of temporary failure of giothub services.
With this solution, the pr will be retried once every cron loop.
We dont except to have pr with this kind of persistent failure more than every few mounths/years.
2021-05-17 14:07:55 +02:00
Xavier-Do
84d9425bdc [FIX] runbot: various view fixes
- fix missing batch references on builds
- fix group on action buttons
2021-04-09 14:58:17 +02:00
Christophe Monniez
57bd00672d [IMP] runbot: add a chart page for build stats
Since 360e31ade4, it's possible to add statistics values to build
results but there was no practical way to analyze them.

With this commit, there is a new button on the bundle page that leads to
a chart page that displays those values.

The default reference build is last known good build of the bundle.
Values are filtered by key and only the most significant values are
displayed. The user can then refine the chart by changing the reference
build or the key and a few other options.

Co-author: Xavier-Do <xdo@odoo.com>
2021-04-09 14:10:37 +02:00
David James
19c312d92c [IMP] runbot: add an option to exclude paths from coverage report 2021-04-07 15:47:11 +02:00
Andrius Laukavičius
5337ecd11f [FIX] runbot: _compute_host_id
If you try to manually create bundle, Odoo will crash, because it will
try to use `name` value that is not set yet. For that we start computing
host_id once `name` is entered.
2021-04-07 11:00:55 +02:00
Xavier-Do
0ed4728518 [IMP] runbot: use last step in wakeup when possible 2021-04-07 10:31:05 +02:00
Xavier-Do
46269ada70 [FIX] runbot: fix logged message 2021-03-15 13:12:48 +01:00
Christophe Monniez
e769a9d2af [FIX] runbot: improve the way a database name is choosen for running
When a build is reaching the run_run_odoo step, a database has to be
set. If none are found in the build params, the one from the last step
is choosen. Historically, the last one in the `Split` config was `all`
but now, the last one is `base`.

With this commit, if none are found in build params, `all` is choosen if
found in any install config steps. As a default, the one from the first
step is choosen.
2020-12-04 15:03:37 +01:00
Christophe Monniez
3b6487b969 [FIX] runbot: fix _is_docker_step to incidentallly fix log menu
Since 3657a65b20 docker_run is called outside of the step and python
steps have to set the `docker_params` variable. This breaks the computed
`log_list` because the string `docker_run(` is searched in python
steps code to determine if it's a docker_step.

With this commit, the `docker_params = ` is searched instead.
2020-11-26 18:05:00 +01:00
Xavier-Do
7519406e3f [FIX] runbot: to_upgrade for complement should use base_id 2020-11-26 13:35:09 +01:00
Xavier-Do
4ab669e371 [IMP] runbot: add to_upgrade option
For now, the sticky flag is used to define bundle to use as target to upgrade.
The plan for future is to continue to test upgrade, even if the bundle is not sticky anymore.
This new flag will allow to remove sticky for old branche.
This will also allow to disable upgrda tests for still sticky branches when needed.

This commit also filter source bundle based on this flag
(before that all base bundle where used as source, even if not sticky)
2020-11-24 15:14:23 +01:00
Xavier-Do
d76a41537c [IMP] runbot: add batch_dependent triggers
Params mechanism are a way to avoid to rebuild with the exact same params.
Commit is the main identifier to know if two builds are the same, ususally
commit_link are not usefull to uniquify params, exept when the mergebase is
in use as in security check (based on diff).

We would also like to avoid to have red diff-based build in sticky/base branches when the
ci is ignored on a pr, but we want to keep the diff display on new batches
(lines and files changed).
We need a way to know if it is a is_base bundle. This was an initially forbidden behaviour
because of duplicate detection.

This commit add a batch_dependent flag. This will enable the use of the batch in the
fingerprint, making it unique per batch and disabling duplicate detection/ in this
case we can use the "is_base" information from the bundle + avoid false duplicate
detection if trigger is based on mergebase-commit diff. (pr based on another pr now merged)
2020-11-24 15:13:00 +01:00
Christophe Monniez
9c7cee657d [IMP] runbot: improve build error frontend page
On the build error web page, a regular assigned error is not shown to
all users.

With this commit, a regular build error (not only random) will be shown
if the error is assigned.
2020-11-18 16:49:45 +01:00
Christophe Monniez
c6a1d202a1 [IMP] runbot: parse log redirects to the created error
When builds logs are parsed by using the contextual button, the client
stays on the same page even if a build error is created.

With this commit, the client is now redirected to the created/found
build error(s).
2020-11-18 16:49:45 +01:00
Christophe Monniez
e51a27bbbb [FIX] runbot: fix build error creation form 2020-11-18 16:49:45 +01:00
Xavier-Do
87512ae78c [IMP] runbot: improve coverage
- execute coverage results in case of failure
- fix coverage links
- use coverage directly
2020-11-18 13:36:41 +01:00
Xavier-Do
559e10754e [FIX] runbot: fix wakeup 2020-11-12 14:02:00 +01:00
Xavier-Do
3657a65b20 [IMP] runbot: call docker_run outside steps
When calling a step from a python step, it is impossible to alter some parameter, the only solution is to copy all step code.
With this change, python step are now able to override docker_run parameter of another step by modifying the dict returned by run_* steps
and assigning the result to "docker_params".

This will mainly be used to set the correct docker_image for migration pre and post test. This can also be used to alter a command,
like removing the pip install or adding extra pre/post operation on the command.
2020-11-11 10:47:26 +01:00
Christophe Monniez
79f7fc9ed2 [FIX] runbot: fix default image_tag
Also fix Docker build process return code.
2020-11-10 11:03:30 +01:00
Christophe Monniez
981cb7e1b6 [IMP] runbot: use a Dockerfile model
Currently, runbot is using a single Dockerfile maintained in a data file
in the source code. This situation is not convenient for testing Odoo in
different environments.

With this commit, a Dockerfile Odoo model is used to allow usage of
multiple Docker containers.

This model comes with a pre-defined Dockerfile that can be used to build
the current Odoo supported versions (12.0 up to 14.0).
2020-11-10 09:52:48 +01:00
Xavier-Do
92ac1d4737 [FIX] runbot: fix field label 2020-10-20 15:26:41 +02:00
David James
237e931e57 [IMP] runbot: add extra parameters and environment variables to run stages 2020-10-19 11:29:39 +02:00
Xavier-Do
7e2d7c65a1 [FIX] runbot: fix message formating 2020-10-19 01:30:39 +02:00
Xavier-Do
43075a59e6 [IMP] runbot: allow to define errors policy for _git decode 2020-10-15 16:54:47 +02:00
Xavier-Do
1985917702 [FIX] runbot: add limit on log length 2020-10-15 15:55:33 +02:00
Xavier-Do
f9272cb207 [IMP]: add linitng tools (+bunch of small fixes) 2020-10-15 10:38:36 +02:00