Before this commit, the build errors page was neither sortable neither
filterable.
This commit adds a way to filter by:
- all
- unassigned
- seen more than once
It also allows to sort by:
- last seen
- nb seen
Typical need is to sort by nb seen and filter out the only seen once to
be able to figure which one is supposed to be checked in priority.
In some rare cases, a docker container has a status of exited, removed
or in removal and the `end-`file has been written right after the code
checked the existence of the file.
To mitigate the issue, this commit checks the `end-` file existence
after the container status have been checked. That way, if the file
exists we can be pretty confident that the build really ended.
When exporting a commit, it will be useful to freeze the modification
time of the exported files. The idea behind that is to pre-generate
bundles at the install time of the Odoo instances, that way when running
the post install tests, the bundles does not have to be generated for
each test.
Because the login link redirected to `/`, when logging in the user would have
to re-navigate to their previous page unless they'd remembered this issue and
kept the original page around.
Fix this because it's annoying and dumb: we know our URL when rendering
templates, so we can redirect back there after login.
Consideration: this could also be done on logout, however it seems likely that
in that case the original page is "privileged" and when coming back we'd just
get an access error. So don't do it for now.
In bd4cf76b7 a new argument `--progress-bar off` was added to the pip
command line. Unfortunately, it appears that this argument is not
available in some pip versions, typically the one from Ubuntu Bionic
(9.0.1) [0]. The argument appeared in version 10.0.0 [1].
Also, in the DockerFile, we allow the user to use `sudo` for
`/usr/bin/pip3` and the main reason for that was to avoid having `pip`
to re-install the packages that are pre-installed by the distribution
package manager.
That said, it appears that since pip 10.0.0 when installing packages
locally for the user, pip now properly detects distribution installed
packages.
So, as the solution to fix the progress bar issue is to upgrade pip to a
newer version, we can take advantage of it to get rid of the `sudo`.
Besides the fact that `--user` is the default on Debian based
distributions [2] , we enforce it in case another distribution is used
in the Dockerfile.
[0] https://packages.ubuntu.com/bionic-updates/python3-pip
[1] https://pip.pypa.io/en/stable/news/#id744
[2] https://wiki.debian.org/Python#Deviations_from_upstream
In some conditions, the google-chrome crashes and a core dump is written
in the bind mounted directory.
With this commit, the core dumps are disabled in the containers.
While using the docker cli as a subprocess was KISS and convenient,
the python Docker SDK is mature, easy to use, available as a Debian
package and much more powerful.
It will permit to monitor the containers memory consumption and will
help to spot memory leaks.
On the actual runbot deployments, the `git gc` command is handled by a
unix cron. From time to time, some repositories get corrupted and we
suspect that some concurrent action may be involved as stated in
documentation [0].
For those reasons, with this commit, the `git gc` will be run by the
runbot clients themselves in order to avoid concurrent operations.
By default, the first gc will occur a few minutes after the start of the
client and the next gc are scheduled a two hours and a few minutes later.
Also, this commit ensures that the git config is written regularly in
case of change.
[0] https://git-scm.com/docs/git-gc
Regarding the number of refs in odoo repo (arround 18 million at this
time), the parsing of the date was significant when filtering old refs.
Using unix time allows a direct comparaison without parsing the date,
and improved performance, going from ~7 seconds to ~1.3 seconds.
The status are currently sent by leader when build are created and by
workers on post_commit.
If the leader fails during the preparing of a batch (while creating
builds) the transaction is rollbacked and the status are send again.
The number of status to send makes it quite slow, making the transaction
longuer, and the retry even more expensive. This leads to preparing time
being quite important, sometimes ten minutes after many retries.
This commit proposes to send status in another dedicated transaction.
Since status are sent in batch, we can also try to use a unique session,
and uniquify commit+context status.
This allows to remove the postcommit logic
A further improvement would be to wait before sending status in order to
skip the pending status if the build is verry fast.
An option is also added on the remote to skip the status: if the remote
is a fork, sending the satus on the main remote should be enough.
Initial model was a (build_id, key, value) where key is in fact a two
part information: `category.key` (where key is usually a module)
This means that for each module, we will have one entry per
modules*category.
We have between 200 and 400 modules per build * 4 keys -> around 1000
entries per build.
The hudge amount of total entries lead to a fast overflow of the table
sequence + this create important indexes.
Also, most of the time, the js will manage the display of stats meaning
that python will transform
(build_id, category.key, value)
into
(build_id, {key: value}) for one category
A new model makes use of a json field to store values for different
modules in a json dict, on entry per category*build. (4 entry per build)
The table will be renamed and migrate later
The initial pull_info_failures feature was introduced before the separate
process for the leader. Since this is here for a rare case, this
wasn't tested since then.
Testing the new infrastructure without token, there was many
pull_info_failure, leading to an error on cleanup.
When rendering templates for git configuration and nginx configuration,
the templates are rendered as Markup, while a byte-like object was
expected for the saving.
With this commit, the Markup is casted into str and the files are saved
as text files.
The `utils.escape` utility was deprecated in werkzeug 2.0.0, so it can
be replaced with our `html_escape` which is itself `markupsafe.escape`.
Also, with this change, the double quotes are now escaped in `"`
instead of `"`, so we fix the test too.
Some non tested code was making running build available without
an nginx config (using port and local address instead of using a proper
dns/nginx config).
Removing this part to reduce complexity.
