Commit Graph

1154 Commits

Author SHA1 Message Date
Xavier-Do
2e44c51678 [FIX] runbot: limit port search to local host.
With the increasing number of runbot servers (17), the total number of docker
instances can reach more than 3570 only for running build. Starting at 2000,
this covers the posrt 5432 used by postgress and make the build run step fail.

This commit simply limit the port unicity constraint by host.
2019-09-03 11:10:38 +02:00
Christophe Monniez
5297ac0f5f [IMP] runbot: show no_build on branch form 2019-09-03 11:10:38 +02:00
Christophe Monniez
4c75a8e107 [IMP] runbot: add an action to parse logs from build error views 2019-09-03 11:10:38 +02:00
Christophe Monniez
c49e422a25 [IMP] runbot: add an error log model based on a SQL view
With this commit, a new model is introduced to facilitate the tracking
of the build errors.

Its based on an SQL view (Thanks @Xavier-Do), that way, there is no new
table in DB and this view is also useful from the PSQL CLI.

In the UI, the search for errors easier than manipulate the ir_logging
view because the builds informations can be used in search and filters.
2019-09-03 11:10:38 +02:00
Xavier-Do
7382aa6b79 [FIX] runbot: various small fixes and imps 2019-09-03 11:10:38 +02:00
Christophe Monniez
f94c8d7390 [FIX] runbot: use odoo repo for google-chrome
The since google chrome 74, a random bug makes it crash at startup,
making the odoo tests crash.

With this commit, an odoo custom deb repo is used on nightly with a
known working chrome version.
2019-09-03 11:10:38 +02:00
Xavier-Do
a91b08897f [IMP] runbot: improve archive failure logging 2019-09-03 11:10:38 +02:00
Christophe Monniez
57a32ee937 [IMP] runbot: log error instead of info when wake up fails 2019-09-03 11:10:38 +02:00
Christophe Monniez
8d199b4c16 [FIX] runbot: catch exceptions on wake-up
When a build is wake-up and something goes wrong during the
_run_odoo_run method, the "fetch and build" cron is broken and the
concerned runbot host stops working.

With this commit, the exception is catched and the build goes back to
the "done" state whith a log.
2019-09-03 11:10:38 +02:00
Christophe Monniez
0da30a9f60 [IMP] runbot: add build_error model
With this commit, a new RunbotBuilError model is added in order to
classify and manage errors that appears during runbot builds.  This is
an helper to find undeterministic bugs in Odoo builds.  Build logs can
be parsed on demand, during the parsing, the logs are cleaned with some
regexes stored on the RunbotErrorRegex model.  A hash is computed on the
cleaned log, if a build error already exists with the same fingerprint,
the build is appended on the build error.

Errors can also be manually linked together with a parent/children
relation in case of a related error log. e.g. the error message is
different in two different branches but the bug is the same.

Also, a new build_url field is added to the runbot_build in order to
access the build web page from the backend.
2019-09-03 11:10:38 +02:00
Xavier-Do
02d2cc4528 [WIP] runbot: monitoring tools
Add a new model runbot.host to keep info and configuration about
hosts (worker servers), like number of worker, reserved or not,
ping times (last start loop, successful iteration, end loop, ...)
and also last errors, number of testing per host, psql connection
count, ...

A new monitoring frontend page is created, similar to glances
but with additionnal information like hosts states and
last_monitored builds (for nightly)

Later this model will be used for runbot_build host instead of char.

Host are automaticaly created when running _scheduler.
2019-09-03 11:10:38 +02:00
Xavier Morel
ef24adad88 [FIX] runbot_merge: cancel button on staging
* split action_cancel (UI button) from cancel (internal): since the
  xhr mapping is weird, if there are available args the mapper thinks
  it should pass the call context as reason which is unexpected
* make cancel a no-op when called on already inactive stagings
* make cancel work when called on multiple statgings
* make computing the active staging work properly in an
  active_test=False context (e.g. when it's interacted with from the
  form view because that comes from the list view which is
  active_test=False, probably so we can see not just the stagings but
  recursively see deactivated batches in deactivated stagings)
* don't show the cancel button on inactive stagings
2019-08-27 12:28:53 +02:00
Xavier Morel
b1b959d472 [FIX] runbot_merge: properly handle retarget from deactivated branch
A deactivated branch is generally treated as unmanaged which is mostly
correct except for the case of retargeting an existing PR.

When a branch is deactivated the corresponding PRs are not removed, so
it's possible to have live PRs associated with ~unmanamaged
branches. When retargeting those PRs to active branches, the mergebot
would assume there was no existing PR and would create a duplicate,
then either get completely lost (before
a84595ea04) or blow up (after the same).

Properly search amongst deactivated branches for retargeting sources
so we update the relevant PR instead of trying to create duplicates.

Fixes #169
2019-08-27 11:27:02 +02:00
Xavier Morel
e40e814b90 [IMP] runbot_merge: show heads on stagings
Stagings have a "statuses" field which was shown but useless (as it's
a binary), they also have a "heads" field which only provides a
mapping of repository names to commits.

This change provides the staging heads as a commits m2m.

Fixes #178
2019-08-26 17:22:21 +02:00
Xavier Morel
1981bd68e6 [IMP] alter ngrok startup to allow concurrent runs
Running multiple ngrok concurrently is only allowed from pro and up
(OOTB and without shenanigans) is only allowed from Pro and up. However
multiple tunnels through a single ngrok is allowed

-> when tunneling through ngrok, start the process without any tunnel,
use the API to create then remove the local tunnel, and shut down ngrok
IIF there's no tunnel left.

There's plenty of race conditions but given how slow tests are when they
involve github that's probably not an issue.
2019-08-26 13:53:37 +02:00
Xavier Morel
28bcc6b5d7 [IMP] runbot_merge: refactor some bits
* extract method to create a PR object from a github result (from the
  PR endpoint)
* move some of the remote's fixtures to a global conftest (so they can
  be reused in the forwardbot)
2019-08-26 13:53:37 +02:00
xmo-odoo
02d85ad523
[FIX] runbot_merge: less restrictive commands matching
Fixes #167 ignores casing when matching bot name
Fixes #168 ignores leading whitespace when matching commands lines
2019-08-26 13:41:33 +02:00
xmo-odoo
8b74e79da9
[FIX] runbot_merge: FF error logging message
In case of error while fast-forwarding a staging to its source, we'd
log the target to which we couldn't FF. Sadly this relied on a
`repo_name` variable which (likely since the introduction of the
"safety dance" fast forwarding) can not actually be set in case of
failure.

So stash the relevant bit (the repo name) inside the FF error exception
and use that to compose our logging message instead of a variable which
can only be None.
2019-08-26 13:41:11 +02:00
Xavier Morel
222f591deb [IMP] runbot_merge: ACL & PR name
* add missing ACL for PR feedback object
* configure name_get for PRs (which don't have a name), fixes some
  layout issues & stuff
2019-08-26 13:22:29 +02:00
Xavier Morel
0bfb018e49 [IMP] runbot_merge: table for staging history
That way all staging labels (timestamps) have the same width, and PRs
/ batches being wrapped don't look like weird-named stagings.
2019-08-26 11:56:42 +02:00
xmo-odoo
c4b7604999
[ADD] runbot_merge: staging history per branch
Closes #175
2019-08-21 14:15:10 +02:00
xmo-odoo
a84595ea04
[FIX] runbot_merge: indexing of PR objects
Github constrains a single issue (/PR) number per repository, having
different targets does not allow two PRs to share a number.

Doesn't fix but should mitigate #169 slightly.
2019-08-21 11:21:06 +02:00
xmo-odoo
429257d013
[FIX] runbot_merge: resync tags on stage change
Before this change mergebot assumes github's tags are in sync with its
"previous" state, but because tags update was highly non-atomic (one
call per removal plus one for additions) and state can further change
between a failure and an update retry (especially as the labels endpoint
fails *a lot*), it's possible for set tags (in github) to be completely
desync'd from the mergebot state, leading to very misleading on-pr
indications.

This first fetches the current tagstate from github (to not lose non-
mergebot tags) then (hopefully atomically) resets all tags tags based on
the current mergebot state. This should avoid desyncs, and eventually
resync PRs (if they change state).

Fixes #170
2019-08-21 11:17:04 +02:00
Xavier-Do
2a18ef4195 [FIX] runbot: missing indirect
In some case _force can return an empty recordset,
if the corresponding branch is in no_build mode in other
repo may be an explanation here.

This commit avoid to stuck the fetch and build loop in this case.
2019-08-19 12:13:55 +02:00
mreficent
9b174ab706 [FIX] runbot: typo (closet -> closest) 2019-08-14 11:43:03 +02:00
Christophe Simonis
059957282e [IMP] runbot: use babel helper to format timedelta
It allow minimal granularity to minutes and rollover thresholds.
2019-08-13 17:56:52 +02:00
Christophe Monniez
857821e41a [IMP] runbot: use the odoo screenshots args
Since odoo/0dc4a63e2 the screenshots arguments can be used to specify
where to save sceenshots.
With this commit, the argument is used if available.
2019-08-13 14:46:07 +02:00
Xavier-Do
13ca8b98cf [IMP] runbot: only update hook on push event
Hook can represent label changes, closed pr, ....
We only want to fetch is some push or synchronize are sent.

TODO We also may want to catch retarget later in order to update branch.
2019-08-13 14:33:19 +02:00
Xavier-Do
d68d0ffa81 [FIX] runbot: fix tracking visibility
Tracking was inspired from master state of the art. This commit addapt
everything to 11.0 tracking in ordr to make it work.
2019-08-13 14:32:18 +02:00
Xavier-Do
a374c2cba3 [FIX] runbot: fix indirect mechanism
indirect state was writen on parent leading to unconsistent info.
indirect was using last build regardless of build_type. Now, indirect
will only use normal build to avoid red-chain after a sticky rebuild.
2019-08-13 14:30:41 +02:00
Xavier-Do
626f454d8b [IMP] runbot: commit message dep param.
A prototype of feature was added some times a go.
No really tested, this commit improves parmater format
and makes dependency closest_branch_id not required
since a repo/sha is all we need.
2019-08-13 14:30:30 +02:00
Xavier-Do
46a7362d18 [FIX] runbot: tempfix wait for docker mechanism
Docker can take some time to be considered as running after docker_run. This
issue can appear when we speedup sheduling loop. To avoid that when can add an
time condition to consider if a docker is running, but we want to avoid to wait
to much since some jobs are fast.

This solution check if a job is a docker run before waiting, and will also
update job_start after a checkout since this can take some time if
a git fetch is performed.
2019-08-13 14:19:51 +02:00
Xavier-Do
5bbfb06ce1 [IMP] runbot: keep result on kill if result is failure
When a build is killed, result will be set to manually killed,
removing the 'error' or 'warn' result.

This commit removes this behaviour in order to keep error result
in this case.
2019-08-13 14:15:26 +02:00
Xavier-Do
2aed7e1a48 [IMP] runbot: imp build result display
If children are killed, they will all look the same in the parent view
making difficult to find the failed one in staging branches.

This commit displays result rather than status in priority if build
is in failure.
2019-08-13 14:12:31 +02:00
Xavier-Do
694f9e6e05 [IMP] runbot: only gc db after job end
If a user really wants to keep a database up for a long time, he has the possibility
to wake it up multiple times.

Using last job end as reference will allow to keep a database alive longer.
2019-08-13 14:10:21 +02:00
Xavier-Do
fb637194f5 [FIX] runbot: use parent global_state for gitthub status
The main motivation of this commit is to be able to notify github status only
when all children are done.

Until today, children where only used for dev branches and nightly. The needs to
use this system for staging need to enforce github_status behaviour.
Before this commit, a parent won't send github status since he will only
create childrens. And childrens are not awared of other children state,
so sending a succes may be wrong if another one failed.

Asking the parent to make the github_status looks the easiest solution:
-If top parent config will have update_github_state False, we also want to take that into account.
-If a child wants to contact github for failfast, parent will be in global_state error too and will send
message immediatly.
-If a child want to contact github for succsess, we actually want to wait for last child, parent will
be in waiting global_state and notify nothing (or pending).
Only last child will be able to notiffy success since global_state will be running or done at this step.
Orphan builds wont have any impact on result in with this scenario.
2019-08-13 14:03:14 +02:00
Xavier-Do
2e5b1cb240 [FIX] runbot: fix typo
kill -> _kill
2019-08-13 11:10:21 +02:00
Xavier-Do
0efd3fde1d [FIX] runbot: less noisy log
Some data are logged at each loop turn even if nothing interresting was done:
- ... builds [] where allocated to runbot
- reload nginx

That kind of info was interresting for debug but now this noise makes
logs heavier and more difficult to read.
Reload ngnix will be done only if file changed and this this will avoid
a log at each loop turn.

We also display difference between existing sources and source that should
be there instead of complete lists.
2019-08-13 10:58:00 +02:00
Xavier-Do
75dc7bd605 [IMP] runbot: clean sources once they are not used
Sources can be easily exported if needed since they are in the bare repository
most of the time. To avoid using to much space, this commit will garbage collect
all sources at the beginning of a long_running cron.

Only real side effect of this is that it will be impossible to wake up
a build that was force pushed since source cannot be fetched anymore.
We may imagine that we could keep sources of recent build, maybe for
48 hours, but keeping build specific data (logs, database)
is more interresting.
2019-08-13 10:35:19 +02:00
Xavier-Do
22fe792e9f [IMP] runbot: add wake_up button where connect button usually stands
This commit add a wake up button in place of connect button when build is
not running and may be wake up.

Connect button will also be visible immediatly when local_state is 'running'
since we don't need to wait sub build to finish.
2019-08-13 10:33:37 +02:00
Xavier Morel
9c9b312f8a [ADD] runbot_merge: staging dependencies 2019-08-09 14:31:21 +02:00
xmo-odoo
cfc7478fcf
[FIX] runbot_merge: PR splits should be updated on PR state change
On a PR being updated, closed or unreviewed, if it is part of an
active staging that staging would get cancelled (yay). However, if the
PR was part of a pending *split*, then the split would *not* get
cancelled / updated (to remove the PR from it), and the PR could go on
to get staged as if everything were right in the world which is an
issue.

It doesn't look like it actually happened (at least I got no echo of
it), but it almost did at least once.

fixes #160
2019-07-31 09:20:02 +02:00
xmo-odoo
6cb58a322d
[IMP] runbot_merge: send feedback when approving PR which failed CI
Also add test for it & feedback of an approved PR failing CI, and fix
corner case with it (might not send a warning immediately on CI failure
depending on status requirement ordering).

Fixes #158
2019-07-31 09:19:50 +02:00
xmo-odoo
85ac2e5d5e
[IMP] runbot_merge: map PR commits to integrated commits
* when rebasing, store a map of rebased to source, that way it'll be
  possible to link cherry-picked forward ports to the originally
  integrated commit rather than just the one from the PR (which was
  likely not itself integrated as the straight merge mode is somewhat
  rare: as of 5600 PRs merged so far only 100 were straight merged)
* while at it, store the "merge head" of the PR (whether squashed,
  merged or rebased) and put *that* in the commit message

fixes #161
2019-07-31 09:19:39 +02:00
xmo-odoo
955b97a023
[IMP] runbot_merge: p=1 > split
Allows merging a fix for e.g. a common false positive during a split
but without cancelling a staging which might just pass (you never
know).
2019-07-31 09:19:28 +02:00
Christophe Monniez
dcf3297bff [FIX] runbot: forbid wake-up on dead builds
When a build is completely dead, with directory and db deleted, the
wake-up system fails.

With this commit, a wake-up is not allowed on such dead builds.
2019-07-19 13:56:19 +02:00
Christophe Monniez
c3e23532be [IMP] runbot: allow nginx access to tests dir
In order to stores other things than logs, that could be accessible by
end users, for example screenshots and screencasts, a "tests" directory
is allowed thruough the nginx template in the builds directories.

Also, the "with" context manager is used to open the nginx configuration
to ensure that the file descriptor is released during long running crons.
2019-07-18 14:39:38 +02:00
Xavier-Do
287efc7989 [FIX] runbot: make result for python code too 2019-07-17 15:51:20 +02:00
Xavier-Do
090b84c86c [FIX] runbot: partial fix for build time with children 2019-07-17 15:51:20 +02:00
Xavier-Do
99fcd9638a [IMP] runbot: take care or extras
If a param is in extra, dont add it automaticaly.
This will allow to remove custom 'test from extras' step.
2019-07-17 15:51:18 +02:00