The pdf417gen python library is needed for the l10n_cl_edi. As this module
is not mandatory, the library was not added in the requirements.txt
file.
In order to test the feature on the runbot, this commit adds the library
in the Docker container.
Related work:
odoo/odoo#54995odoo/enterprise#12080odoo/enterprise@8290bfaf
Runbot initial architechture was working for a single odoo repo, and was
adapted to build enterprise. Addition of upgrade repo and test began
to make result less intuitive revealing more weakness of the system.
Adding to the oddities of duplicate detection and branch matching,
there was some room for improvement in the runbot models.
This (small) commit introduce the runbot v5.0, designed for a closer
match of odoo's development flows, and hopefully improving devs
experience and making runbot configuration more flexible.
**Remotes:** remote intoduction helps to detect duplicate between odoo and
odoo-dev repos: a commit is now on a repo, a repo having multiple remote.
If a hash is in odoo-dev, we consider that it is the same in odoo.
Note: github seems to manage commit kind of the same way. It is possible
to send a status on a commit on odoo when the commit only exists in
odoo-dev.
This change also allows to remove some repo duplicate configuration
between a repo and his dev corresponding repo.
(modules, server files, manifests, ...)
**Trigger:** before v5.0, only one build per repo was created, making it
difficult to tweak what test to execute in what case. The example use
case was for upgrade. We want to test upgrade to master when pushing on
odoo. But we also want to test upgrade the same way when pushing on
upgrade. We introduce a build that should be ran on pushing on either
repo when each repo already have specific tests.
The trigger allows to specify a build to create with a specific config.
The trigger is executed when any repo of the trigger repo is pushed.
The trigger can define depedencies: only build enterprise when pushing
enterprise, but enterprise needs odoo. Test upgrade to master when pushing
either odoo or upgrade.
Trigger will also allows to extract some build like cla that where
executed on both enterprise and odoo, and hidden in a subbuild.
**Bundle:** Cross repo branches/pr branches matching was hidden in build
creation and can be confusing. A build can be detected as a duplicate
of a pr, but not always if naming is wrong or traget is invalid/changes.
This was mainly because of how a community ref will be found. This was
making ci on pr undeterministic if duplicate matching fails. This was
also creating two build, with one pointing to the other when duplicate
detection was working, but the visual result can be confusing.
Associtaions of remotes and bundles fix this by adding all pr and
related branches from all repo in a bundle. First of all this helps to
visualise what the runbot consider has branch matching and that should
be considered as part of the same task, giving a place where to warn
devs of some possible inconsistencies. Associate whith repo/remote, we
can consider branches in the same repo in a bundle as expected to have
the same head. Only one build is created since trigger considers repo,
not remotes.
**Batch:** A batch is a group of build, a batch on a bundle can be
compared to a build on a branch in previous version. When a branch
is pushed, the corresponding bundle creates a new batch, and wait for
new commit. Once no new update are detected in the batch for 60 seconds,
All the trigger are executed if elligible. The created build are added
to the batch in a batch_slot. It is also possible that an corresponding
build exists (duplicate) and is added to the slot instead of creating a
new build.
Co-authored-by d-fence <moc@odoo.com>
73f720a55c refactored the runbot tests,
and amongst other things created a single patch point for the "mock
root" as a testcase attribute.
One of the tests was missed during that refactoring, likely because
it's skipped by default.
The ocrmypdf suite of tools are needed by task #2238654.
Altough this package will be optional for Odoo users, it has to be
usable by dev's in order to test and/or fix the feature.
Docker container names are derived from the dest and step name. The dest
is itself derived from the branch name.
In some rare cases, it happens that a character not allowed by Docker
appears in the container name computed by the runbot.
With this commit, a sanitize_container_name function is used to remove
unallowed characters at the container utility level.
The postgresql-client in the Dockerfile is the one provided by the
Debian package. When the postgresql server on the host has a higher
version than the client, some builds may fail (for example, dumping a
database with the pg_dump).
With this commit, the postgresql-client 12 from the postgresql repo is
used in the Dockerfile.
Runbot can send status multiple time for the same hash:
- if transaction fails in scheduler and is retried
- if multiple subbuild are failing
Leading to multiple issues:
- when github receive more than one failure status, mergebot will
be notified multiple times and send multiple mail (for forward ports mainly)
- github will answer `422 Unprocessable Entity for url...` after
1000 status.
This fix proposes to limit number of status:
- By avoiding to send status for orphan build (parent status will never change)
- By storing last send status to avoid to notify multiple time
- By sending status post commit to avoid to contact github in case of failure.
This will also slightly reduce transaction time by removing an http request.
Sometimes, it happens that a `git fetch` fails with an error code 128
for example. When this happens, the runbot host is immediately disabled.
During investigations of such cases, we found that simply retrying the
fetch command works.
With this commit, the fetch command is tried 5 times with an increasing
delay before deciding to disable the runbot host.
Since we store the target_branch_name, filtering out pull head names
that contains `patch-` is not necessary anymore.
This commit is one first step towards a clean refactoring.
When a build is done, various numerical informations could be extracted
from log files. e.g.: global query count or tests query count ...
The extraction regular expression could be hard-coded in a custom step
but there is no place holder where to store the retrieved information.
In order to compare results, we need to store it.
With this commit, a new model `runbot.build.stat` is used to store
key/values pair linked to a build/config_step. That way, extracted
values can be stored.
Also, another `runbot.build.stat.regex` is used to store regular
expressions that can be used to grep log files and extract values.
The regular expression must contain a named group like this:
`(?P<value>.+)`
The text catched by this group MUST be castable into a float.
Optionally, another named group can be used in the regular expresion
like this:
`(?P<key>.+)`
This `key` group will then be used to augment the key name in the
database.
Example:
Consider a log line like this one:
`odoo.addons.website_blog.tests.test_ui tested in 10.35s`
A regular expression like this, named `test_duration`:
`odoo.addons.(?P<key>.+) tested in (?P<value>\d+\.\d+)s`
Should store the following key:value:
`{
'key': 'test_duration.website_blog.tests.test_ui',
'value': 10.35
}`
A `generic` boolean field is present on the build.stat.regex object,
meaning that when no regex are linked to a make_stats config step, then,
all the generic regex will be applied.
A wizard is added to help the creation the regular expressions, allowing
to test if they work against a user provided example.
A _make_stats method is added to the ConfigStep model which is called
during the _schedule of a build, just before calling the next step.
The regex search is only apllied in steps that have the `make_stats`
boolean field set to true. Also, the build branch have to be flagged
`make_stats` too or the build can have a key `make_stats` in its
config_data field.
The `make_stats` field on the branch is a compute stored field.
That way, sticky branches are automaticaly set `make_stats' true.
Finally, an SQL view is used to facilitate the stats visualisation.
When coverage is computed, a post command is used to generate the HTML
report. In order to use the coverage result locally the HTML report is
not enough.
With this commit, an XML report is also generated. It's a single xml
file, downloadable from the build result web page.
The _post_install_command method is renamed into its plural form because
it was useless to return only one command.
From times to times, a git repo gets corrupted, making builds fail in
chain.
With this commit, the host on which the git fetch fails will be reserved
if not more than half the hosts are reserved.
Before intensive python steps, every build exept create build should have logs.
This is not true since now many python steps are creating build, leading
to 404 links to inexisting logs file.
Checking That a step is a docker build looks like a good heuristic since since
most of the time the log file will be the one created by docker. In all case we
can access other logs files in browsing /logs.
A future improvement would be to listdir logs to create all links.
With this commit, a kind of markdown is allowed in log messages and
build.description.
Following patterns are supported:
**strong**
~~striketrough~~
__underline__
`code`
[link](target)
@icon-font-awesome-class
When a build.config is copied, the config steps are not copied.
The steps have to be explicitly ordered otherwise it leads to a
traceback when trying to copy a 'run' step which have to be the last
one.
When creating a subbuild from a custom python job/cron, some data can
be given through config_data or extra_params to change another step behaviour.
An example is to give a specific -i to an install job in order to
install different modules from the parent build. Anyway since we
are using the same install step in this case, all databases will
have the same name, and the name may not be correct in regard to
the database content.
This commit allows to give a config_param on build giving the default
db_name to use in an install step.
If a branch triggers an hidden build on a another one (sticky ususally),
the last build of the branch will be considered to be this one and
trigger a new build. The same problem can occur when cloning a subbuild
to slighlty change a parameter instead of making an exact rebuild
since the build type may still be normal in this case.
This commit will simply ignore subbuild in this case.
When a new commit is found, a rebuild is forced on sticky branches
builds in repositories that depends on the new commit repository.
If one of the repo is protected by groups, the main page gives access
errors for public users and finally leads to a CPU overload.
With this commit, the feature is removed as it will be re-implemented in
custom config steps.
On non sticky branches, when a new build is found while another is
already testing, the older build is killed. This happens during when the
main runbot instance is discovering new commits and create new builds.
As a result, concurrent updates may occur while the builders access the
concerned build.
With this commit, this garbage collecting procedure occurs during the
scheduler loop that runs on runbot builder hosts.
Also, the logic changed in a way that the kill is requested only if the
host needs room to handle pending builds.
When a build age reaches the gc_days parameter, its database is dropped
and its directory is removed.
With this commit, two fields are added in order to keep some builds
longer that the defined gc_days.
The gc_delay field on the build allows to add a delay (in number of
days) that is added to its gc_days to compute the gc_date.
The gc_date field is the date when the cleaning will occur.
Also, a test is added and the RunbotCase test class is improved to allow
the stop of a patcher.
With this commit, it will be possible to play with different chrome
versions without impacting all runbot instances.
Chrome issues summary:
* 71: innerText linefeed issue
* 74: random chrome crash
* 80: V8 pointer compression exceed Odoo memory limits
Since cb05f2b9d8, when creating a database, the template1 is used, this
allows to customize the template and install some needed Postgress
extensions.
Unfortunately, it's also a source of build failures. For example, if the
pg_activity util is used on a runbot host, database creation may fail
with a message like this:
source database "template1" is being accessed by other users
It's because pg_activity needs a database and uses template1.
With this commit, template1 is still used by default but can be changed
with a system parameter. That way, a custom template can be created on
runbot hosts and used when creating DB in builds.
When a PR branch target is changed on Github, the change is not applied
in the runbot DB.
With this commit, the Github hook payload is taken into account to
detect such a change and the branch infos are recomputed accordingly.
Also, a button is now available on the branch form in order to manually
recompute those changes.
