Commit Graph

1866 Commits

Author SHA1 Message Date
Xavier-Do
e521149d93 [IMP] runbot: add container_cpus to step views 2024-04-15 14:32:54 +02:00
Xavier-Do
83acc43a05 [IMP] runbot: save reference batches on base batches 2024-04-15 11:42:28 +02:00
Xavier-Do
c9e14a86ca [IMP] runbot: propagate batch to reference builds decision 2024-04-15 11:42:28 +02:00
Xavier-Do
082ecaafe9 [IMP] runbot: add tree hash info to commit
Right now the commit is considered as a part of the build uniquifier.

If it makes sence for a check on the commit metadata, it is not the case
for execution tests where only the content matters.

This will allow to mark a trigger as non depending on commit but tree
hash, and avoid rebuild when only fixing a commit message.
2024-04-15 10:53:48 +02:00
Xavier-Do
f191e8cc48 [IMP] runbot: optionnal container_name in python steps 2024-04-15 10:44:32 +02:00
Xavier-Do
7e848b8073 [FIX] runbot: fix stats.js 2024-04-15 10:35:20 +02:00
Xavier-Do
9ef850220b [IMP] runbot: add a way to add custom pre/post
Mainly usefull for custom triggers
2024-04-15 10:08:56 +02:00
Christophe Monniez
412baa3fad [REF] runbot: centralize cpu_limit in _run_step
In order to keep it coherent, the cpu_limit computation can be done in
one place instead of defining it in all _run* methods.

In python steps, it can still be overridden when returning
docker_params.
2024-04-15 10:06:28 +02:00
Christophe Monniez
d0a96faf84 [IMP] runbot: add cpus parameter
When build eats all the CPU's resources, it may interfere with other
builds and cause collateral damages.

With this commit, a new settings parameter `Containers CPUs` is added in
order to limit the usage of available CPU's on runbot instances.

If left to 0, no limts are applied.

Otherwise, the cpu_quota docker parameter is computed as Containers
CPU's * (logical cpu's count / nb parallel builds) * cpu period which defaults to 100000.

e.g.:
- on a host with 16 logical CPU's
- with 8 parallel builds allowed
- with Containers CPUs set to 1.5
- with the default cpu_period
cpu_quota will be:
    (16/8) * 1.5 * 100000 = 300000

This system parameter can be overridden by the `container_cpus` field on
steps.
2024-04-15 10:06:28 +02:00
Xavier-Do
0fc1daeac9 [IMP] runbot: enable user to toggle no_build 2024-04-10 16:23:50 +02:00
Xavier-Do
70f4fe23a5 [IMP] runbot: alow autorebase for external pr 2024-04-02 11:57:47 +02:00
Xavier-Do
9c4983f5b7 [IMP] runbot: always display autorebase for external pr 2024-04-02 08:33:36 -01:00
Xavier-Do
70532df2d6 [FIX] runbot: fix build error unlink 2024-03-20 09:50:03 +01:00
Xavier-Do
fd7f49aff8 [FIX] runbot: fix staging creation 2024-03-20 09:47:51 +01:00
Xavier Morel
9f22305903 [IMP] runbot_merge: view warnings around ACLs
Eventually we might want to add a proper "sensitive" flag on overrides
and compute the flag based on that. For now just check for
`ci/security`.
2024-03-19 12:54:20 +01:00
Xavier Morel
5024c2e27b [FIX] forwardport: suppress warning when closing unmanaged PR
Continuation of 327500bc83 for an other
edge case of closing a PR to a detached branch with a merged
descendant. The mergebot would:

- warn on the parent about it being detached due to being closed
- then warn on the child about it being detached due to the parent
  being closed (despite it being merged already)
- then warn the parent *again* due to the child being detached

At least some of those messages were still produced by the test case,
stop them.

Issue was noticed on odoo/odoo#145969 and odoo/odoo#145984 due to 16.2
being deactivated.
2024-03-19 11:46:36 +01:00
Xavier Morel
953bf86044 [FIX] forwardport: don't notify of detached child on merged parents
The notification is both noise and confusing: we're telling the
author (and reviewer, and anyone else subscribed) that they need to
merge a merged PR.

Fixes #855
2024-03-12 14:57:50 +01:00
Xavier Morel
327500bc83 [FIX] runbot_merge: don't notify on closing unknown PRs
If an untracked PR is closed, especially on an inactive or untracked
branch, the closer (or author) almost certainly don't care to receive
3 different notifications on the subject.

The fix requires a schema change in order to track that we're fetching
the PR due to a `closed` event, as in other cases we may still want to
notify the user that we received the request (and it just happened to
resolve to a closed PR).

Fixes #857
2024-03-12 12:17:30 +01:00
Xavier Morel
721b769039 [IMP] runbot_merge: handling of signatures
- correctly handle projects without a secret set, we don't want the
  requests to blow up by trying to `strip()` a `False` or `None`, that
  is dumb, who would do that?
- provide better reporting on signature mismatch: which repo we tried
  to access, and the full list of headers
- log when there was no signature matching, either because there was
  no signature in the request and no secret on the project, or because
  the request is signed but no secret is configured on the repo
2024-02-26 10:11:53 +01:00
Xavier Morel
bcf6074153 [FIX] runbot_merge: maintenance gc command
`gc --prune` can not take a *separate* parameter, it has to be part of
the same arg (the `=` is not optional), otherwise the `gc` call blows
up.

So use the positional form of the git command to generate the correct
invocation, Python-level `foo=bar` generates a split-style option in
two args which does not please git.
2024-02-26 09:58:22 +01:00
Xavier-Do
a00fa04e07 [FIX] runbot: remove first db before ordering 2024-02-23 17:09:08 +01:00
Xavier Morel
de32b54090 [FIX] runbot_merge: error in maintenance, and tracking
Before this, we would check if a repository had a name and run
maintenance on it, leading to repeated (but unnoticed until now
because I didn't monitor it) tracebacks as the maintenance cron would
fail to find the local repo then run maintenance on nowhere anyway.

Also augment the repo-finding process to try and get better
information about what it's doing when it fails, rather than failing
completely silently.
2024-02-23 13:58:31 +01:00
Xavier-Do
9fa53d6581 [FIX] runbot: fix stats page following 17.0 upgrade
Some issue where remaining regarding stats click events and scales.
2024-02-23 11:10:39 +01:00
Xavier-Do
2ba3238a34 [FIX] runbot: fix repo.modules is False case 2024-02-21 11:04:25 +01:00
Christophe Monniez
f26065c4dc [FIX] runbot: append a tuple in git gc command list 2024-02-21 07:57:18 +01:00
Xavier-Do
d661fd9fb4 [IMP] runbot: allow to hide project 2024-02-20 16:48:51 +01:00
Xavier-Do
c18bbecf37 [IMP] runbot: allow to customize repo filters from
When using a repo as a dependency for another trigger, the default
module filter for a repo is not always ideal

As an example, when using odoo as a dependency for another repo,
we may only want to install the module from the new repo.

This iss done right now by creating a custom config but this lead to
duplicates config and steps only to customize the module to install.

This commit proposes a new model to store the filters.

Note that this may be used later as module blacklist on repo too.
2024-02-20 16:48:51 +01:00
Christophe Monniez
f032428346 [FIX] runbot: use markup to post message on build errors 2024-02-20 10:22:35 -01:00
Xavier-Do
f7a1a6a11d [IMP] runbot: avoid long idle transaction
Git gc can last a few minutes, it's not a big deal since it's executed
once a day but the transaction is kept idele during this time wich is
not useful. This commit should help to avoid this.
2024-02-19 10:30:33 +01:00
Xavier-Do
60dbbcb72e [IMP] runbot: add total load time on builds 2024-02-19 10:12:28 +01:00
Xavier-Do
336e9525d5 [FIX] runbot: fix check token 2024-02-19 09:59:41 +01:00
Xavier-Do
9cf7a25cca [FIX] runbot: fix params view 2024-02-19 09:59:41 +01:00
Xavier Morel
5d615bd733 [IMP] runbot_merge: logging around webhook body & signature
The signature validation code seems correct, but there are validation
failure in production, increase logging around webhook requests to
try and diagnose things better:

- dump the *entire* body to the github_requests logfile
- add the received & computed signatures to the log error
2024-02-12 10:19:53 +01:00
Xavier Morel
65c303a750 [FIX] runbot_merge: bot info fetch
`r0` is still used afterwards as the response object, so don't
overwrite it when parsing the JSON body.
2024-02-12 10:18:59 +01:00
Xavier Morel
3a4fa494f8 [FIX] runbot_merge: incorrectly named endpoint
Method has the same name as its preceding sibling, so it overwrites
it and one of the endpoints is not accessible.
2024-02-12 10:18:25 +01:00
Xavier-Do
ecfdb5693d [FIX] runbot: branch computation 2024-02-10 10:17:44 +01:00
Xavier-Do
2fa5a6aee7 [IMP] runbot: add has_pr filter 2024-02-09 16:32:25 +01:00
Xavier-Do
a0dd2fa560 [FIX] runbot: fix search limit 2024-02-09 16:32:25 +01:00
Xavier-Do
a7f6a9b157 [IMP] runbot: allow bundle access per name 2024-02-09 16:32:25 +01:00
Xavier-Do
db2da457ad [FIX] runbot: hide legend in new charjs version
The option namespace for label changed  in version 3.0
See https://www.chartjs.org/docs/3.0.2/configuration/legend.html
2024-02-09 16:32:25 +01:00
Christophe Monniez
36fdec11d5 [FIX] runbot: fix errors first and last seen dates
Because of a bad dependency on the compute, the first seen date and last
seen date are not always updated.

e.g.: a new build is scanned and a build is added to a linked error, the
parent error seen dates are not updated.

A test is added to reproduce the case.
2024-02-09 08:22:28 -01:00
Christophe Monniez
89a279d9dd [REL] 17.0 2024-02-09 08:22:28 -01:00
Xavier-Do
b6bc0e3911 [IMP] runbot: allow to disable fetch on start 2024-02-05 12:00:28 +01:00
Xavier-Do
7bbd1271c6 [FIX] runbot: fix tools.py 2024-02-05 11:48:37 +01:00
Xavier-Do
be95c8b364 [FIX] runbot: fix upgrade to 17.0 2024-02-05 10:25:31 +01:00
Christophe Monniez
6dd2580e21 [FIX] runbot: fix various js issues for 17.0 2024-02-05 07:17:31 -01:00
Xavier-Do
8f34312967 [FIX] runbot: adapt js for 17.0 2024-01-30 10:50:58 +01:00
Xavier-Do
3b50f172c0 [FIX] runbot: adapt settings to 17.0 2024-01-30 10:50:58 +01:00
Xavier-Do
a250794432 [FIX] runbot: adapt attrs
remove all attrs in xml views

To help with that, a scripts was written, minimal but sufficent

    #!/usr/bin/python3
    import glob
    import re
    from ast import literal_eval

    def leaf_to_python(leaf):
        if len(leaf) != 3:
            raise ValueError('This script doesnt support leaf', leaf)
        field, operator, value = leaf
        if operator == '=':
            return f'not {field}' if value is False else field if value is True else f'{field} == {value!r}'
        if operator == '!=':
            return f'not {field}' if value is True else field if value is False else f'{field} != {value!r}'
        if operator == 'in':
            return f'{field} in {value!r}'
        if operator == 'not in':
            return f'{field} not in {value!r}'
        raise ValueError('This script doesnt support operator', operator)

    for file in glob.glob('**/*.xml', recursive=True):
        with open(file) as f:
            content = f.read()
            attrs_list = re.findall(r'attrs="{.*}"', content)
            if attrs_list:
                for attrs in attrs_list:
                    match = re.match(r'''attrs="{'(invisible|readonly)': ?(\[.*\])}"''', attrs)
                    attr = match.groups()[0]
                    domain = literal_eval(match.groups()[1])
                    condition = ' and '.join([leaf_to_python(leaf) for leaf in domain])
                    replace = f'{attr}="{condition}"'
                    content = content.replace(attrs, replace)
                with open(file, 'w') as fw:
                    fw.write(content)
2024-01-30 10:50:58 +01:00
Xavier-Do
e83db83533 [REL] runbot: adapt for 17.0 2024-01-30 10:50:58 +01:00