runbot/runbot/common.py

205 lines
4.8 KiB
Python
Raw Normal View History

# -*- coding: utf-8 -*-
import contextlib
import itertools
import logging
import psycopg2
import re
import requests
import socket
import time
[IMP] runbot: runbot 5.0 Runbot initial architechture was working for a single odoo repo, and was adapted to build enterprise. Addition of upgrade repo and test began to make result less intuitive revealing more weakness of the system. Adding to the oddities of duplicate detection and branch matching, there was some room for improvement in the runbot models. This (small) commit introduce the runbot v5.0, designed for a closer match of odoo's development flows, and hopefully improving devs experience and making runbot configuration more flexible. **Remotes:** remote intoduction helps to detect duplicate between odoo and odoo-dev repos: a commit is now on a repo, a repo having multiple remote. If a hash is in odoo-dev, we consider that it is the same in odoo. Note: github seems to manage commit kind of the same way. It is possible to send a status on a commit on odoo when the commit only exists in odoo-dev. This change also allows to remove some repo duplicate configuration between a repo and his dev corresponding repo. (modules, server files, manifests, ...) **Trigger:** before v5.0, only one build per repo was created, making it difficult to tweak what test to execute in what case. The example use case was for upgrade. We want to test upgrade to master when pushing on odoo. But we also want to test upgrade the same way when pushing on upgrade. We introduce a build that should be ran on pushing on either repo when each repo already have specific tests. The trigger allows to specify a build to create with a specific config. The trigger is executed when any repo of the trigger repo is pushed. The trigger can define depedencies: only build enterprise when pushing enterprise, but enterprise needs odoo. Test upgrade to master when pushing either odoo or upgrade. Trigger will also allows to extract some build like cla that where executed on both enterprise and odoo, and hidden in a subbuild. **Bundle:** Cross repo branches/pr branches matching was hidden in build creation and can be confusing. A build can be detected as a duplicate of a pr, but not always if naming is wrong or traget is invalid/changes. This was mainly because of how a community ref will be found. This was making ci on pr undeterministic if duplicate matching fails. This was also creating two build, with one pointing to the other when duplicate detection was working, but the visual result can be confusing. Associtaions of remotes and bundles fix this by adding all pr and related branches from all repo in a bundle. First of all this helps to visualise what the runbot consider has branch matching and that should be considered as part of the same task, giving a place where to warn devs of some possible inconsistencies. Associate whith repo/remote, we can consider branches in the same repo in a bundle as expected to have the same head. Only one build is created since trigger considers repo, not remotes. **Batch:** A batch is a group of build, a batch on a bundle can be compared to a build on a branch in previous version. When a branch is pushed, the corresponding bundle creates a new batch, and wait for new commit. Once no new update are detected in the batch for 60 seconds, All the trigger are executed if elligible. The created build are added to the batch in a batch_slot. It is also possible that an corresponding build exists (duplicate) and is added to the slot instead of creating a new build. Co-authored-by d-fence <moc@odoo.com>
2020-06-03 21:17:42 +07:00
import os
from collections import OrderedDict
from datetime import timedelta
from babel.dates import format_timedelta
2022-06-07 20:56:26 +07:00
from markupsafe import Markup
from odoo.tools.misc import DEFAULT_SERVER_DATETIME_FORMAT, html_escape, file_open
_logger = logging.getLogger(__name__)
[IMP] runbot: runbot 5.0 Runbot initial architechture was working for a single odoo repo, and was adapted to build enterprise. Addition of upgrade repo and test began to make result less intuitive revealing more weakness of the system. Adding to the oddities of duplicate detection and branch matching, there was some room for improvement in the runbot models. This (small) commit introduce the runbot v5.0, designed for a closer match of odoo's development flows, and hopefully improving devs experience and making runbot configuration more flexible. **Remotes:** remote intoduction helps to detect duplicate between odoo and odoo-dev repos: a commit is now on a repo, a repo having multiple remote. If a hash is in odoo-dev, we consider that it is the same in odoo. Note: github seems to manage commit kind of the same way. It is possible to send a status on a commit on odoo when the commit only exists in odoo-dev. This change also allows to remove some repo duplicate configuration between a repo and his dev corresponding repo. (modules, server files, manifests, ...) **Trigger:** before v5.0, only one build per repo was created, making it difficult to tweak what test to execute in what case. The example use case was for upgrade. We want to test upgrade to master when pushing on odoo. But we also want to test upgrade the same way when pushing on upgrade. We introduce a build that should be ran on pushing on either repo when each repo already have specific tests. The trigger allows to specify a build to create with a specific config. The trigger is executed when any repo of the trigger repo is pushed. The trigger can define depedencies: only build enterprise when pushing enterprise, but enterprise needs odoo. Test upgrade to master when pushing either odoo or upgrade. Trigger will also allows to extract some build like cla that where executed on both enterprise and odoo, and hidden in a subbuild. **Bundle:** Cross repo branches/pr branches matching was hidden in build creation and can be confusing. A build can be detected as a duplicate of a pr, but not always if naming is wrong or traget is invalid/changes. This was mainly because of how a community ref will be found. This was making ci on pr undeterministic if duplicate matching fails. This was also creating two build, with one pointing to the other when duplicate detection was working, but the visual result can be confusing. Associtaions of remotes and bundles fix this by adding all pr and related branches from all repo in a bundle. First of all this helps to visualise what the runbot consider has branch matching and that should be considered as part of the same task, giving a place where to warn devs of some possible inconsistencies. Associate whith repo/remote, we can consider branches in the same repo in a bundle as expected to have the same head. Only one build is created since trigger considers repo, not remotes. **Batch:** A batch is a group of build, a batch on a bundle can be compared to a build on a branch in previous version. When a branch is pushed, the corresponding bundle creates a new batch, and wait for new commit. Once no new update are detected in the batch for 60 seconds, All the trigger are executed if elligible. The created build are added to the batch in a batch_slot. It is also possible that an corresponding build exists (duplicate) and is added to the slot instead of creating a new build. Co-authored-by d-fence <moc@odoo.com>
2020-06-03 21:17:42 +07:00
dest_reg = re.compile(r'^\d{5,}-.+$')
[IMP] runbot: runbot 5.0 Runbot initial architechture was working for a single odoo repo, and was adapted to build enterprise. Addition of upgrade repo and test began to make result less intuitive revealing more weakness of the system. Adding to the oddities of duplicate detection and branch matching, there was some room for improvement in the runbot models. This (small) commit introduce the runbot v5.0, designed for a closer match of odoo's development flows, and hopefully improving devs experience and making runbot configuration more flexible. **Remotes:** remote intoduction helps to detect duplicate between odoo and odoo-dev repos: a commit is now on a repo, a repo having multiple remote. If a hash is in odoo-dev, we consider that it is the same in odoo. Note: github seems to manage commit kind of the same way. It is possible to send a status on a commit on odoo when the commit only exists in odoo-dev. This change also allows to remove some repo duplicate configuration between a repo and his dev corresponding repo. (modules, server files, manifests, ...) **Trigger:** before v5.0, only one build per repo was created, making it difficult to tweak what test to execute in what case. The example use case was for upgrade. We want to test upgrade to master when pushing on odoo. But we also want to test upgrade the same way when pushing on upgrade. We introduce a build that should be ran on pushing on either repo when each repo already have specific tests. The trigger allows to specify a build to create with a specific config. The trigger is executed when any repo of the trigger repo is pushed. The trigger can define depedencies: only build enterprise when pushing enterprise, but enterprise needs odoo. Test upgrade to master when pushing either odoo or upgrade. Trigger will also allows to extract some build like cla that where executed on both enterprise and odoo, and hidden in a subbuild. **Bundle:** Cross repo branches/pr branches matching was hidden in build creation and can be confusing. A build can be detected as a duplicate of a pr, but not always if naming is wrong or traget is invalid/changes. This was mainly because of how a community ref will be found. This was making ci on pr undeterministic if duplicate matching fails. This was also creating two build, with one pointing to the other when duplicate detection was working, but the visual result can be confusing. Associtaions of remotes and bundles fix this by adding all pr and related branches from all repo in a bundle. First of all this helps to visualise what the runbot consider has branch matching and that should be considered as part of the same task, giving a place where to warn devs of some possible inconsistencies. Associate whith repo/remote, we can consider branches in the same repo in a bundle as expected to have the same head. Only one build is created since trigger considers repo, not remotes. **Batch:** A batch is a group of build, a batch on a bundle can be compared to a build on a branch in previous version. When a branch is pushed, the corresponding bundle creates a new batch, and wait for new commit. Once no new update are detected in the batch for 60 seconds, All the trigger are executed if elligible. The created build are added to the batch in a batch_slot. It is also possible that an corresponding build exists (duplicate) and is added to the slot instead of creating a new build. Co-authored-by d-fence <moc@odoo.com>
2020-06-03 21:17:42 +07:00
class RunbotException(Exception):
pass
def fqdn():
return socket.gethostname()
def time2str(t):
return time.strftime(DEFAULT_SERVER_DATETIME_FORMAT, t)
def dt2time(datetime):
"""Convert datetime to time"""
2020-01-06 14:50:57 +07:00
return time.mktime(datetime.timetuple())
def now():
return time.strftime(DEFAULT_SERVER_DATETIME_FORMAT)
2022-08-23 19:53:27 +07:00
def findall(filename, pattern):
return set(re.findall(pattern, file_open(filename).read()))
2022-08-23 19:53:27 +07:00
def grep(filename, string):
if os.path.isfile(filename):
return find(filename, string) != -1
return False
def find(filename, string):
return file_open(filename).read().find(string)
def uniq_list(l):
return OrderedDict.fromkeys(l).keys()
def flatten(list_of_lists):
return list(itertools.chain.from_iterable(list_of_lists))
def rfind(filename, pattern):
"""Determine in something in filename matches the pattern"""
if os.path.isfile(filename):
regexp = re.compile(pattern, re.M)
with file_open(filename, 'r') as f:
if regexp.findall(f.read()):
return True
return False
[IMP] runbot: runbot 5.0 Runbot initial architechture was working for a single odoo repo, and was adapted to build enterprise. Addition of upgrade repo and test began to make result less intuitive revealing more weakness of the system. Adding to the oddities of duplicate detection and branch matching, there was some room for improvement in the runbot models. This (small) commit introduce the runbot v5.0, designed for a closer match of odoo's development flows, and hopefully improving devs experience and making runbot configuration more flexible. **Remotes:** remote intoduction helps to detect duplicate between odoo and odoo-dev repos: a commit is now on a repo, a repo having multiple remote. If a hash is in odoo-dev, we consider that it is the same in odoo. Note: github seems to manage commit kind of the same way. It is possible to send a status on a commit on odoo when the commit only exists in odoo-dev. This change also allows to remove some repo duplicate configuration between a repo and his dev corresponding repo. (modules, server files, manifests, ...) **Trigger:** before v5.0, only one build per repo was created, making it difficult to tweak what test to execute in what case. The example use case was for upgrade. We want to test upgrade to master when pushing on odoo. But we also want to test upgrade the same way when pushing on upgrade. We introduce a build that should be ran on pushing on either repo when each repo already have specific tests. The trigger allows to specify a build to create with a specific config. The trigger is executed when any repo of the trigger repo is pushed. The trigger can define depedencies: only build enterprise when pushing enterprise, but enterprise needs odoo. Test upgrade to master when pushing either odoo or upgrade. Trigger will also allows to extract some build like cla that where executed on both enterprise and odoo, and hidden in a subbuild. **Bundle:** Cross repo branches/pr branches matching was hidden in build creation and can be confusing. A build can be detected as a duplicate of a pr, but not always if naming is wrong or traget is invalid/changes. This was mainly because of how a community ref will be found. This was making ci on pr undeterministic if duplicate matching fails. This was also creating two build, with one pointing to the other when duplicate detection was working, but the visual result can be confusing. Associtaions of remotes and bundles fix this by adding all pr and related branches from all repo in a bundle. First of all this helps to visualise what the runbot consider has branch matching and that should be considered as part of the same task, giving a place where to warn devs of some possible inconsistencies. Associate whith repo/remote, we can consider branches in the same repo in a bundle as expected to have the same head. Only one build is created since trigger considers repo, not remotes. **Batch:** A batch is a group of build, a batch on a bundle can be compared to a build on a branch in previous version. When a branch is pushed, the corresponding bundle creates a new batch, and wait for new commit. Once no new update are detected in the batch for 60 seconds, All the trigger are executed if elligible. The created build are added to the batch in a batch_slot. It is also possible that an corresponding build exists (duplicate) and is added to the slot instead of creating a new build. Co-authored-by d-fence <moc@odoo.com>
2020-06-03 21:17:42 +07:00
def time_delta(time):
if isinstance(time, timedelta):
return time
return timedelta(seconds=-time)
def s2human(time):
"""Convert a time in second into an human readable string"""
return format_timedelta(
[IMP] runbot: runbot 5.0 Runbot initial architechture was working for a single odoo repo, and was adapted to build enterprise. Addition of upgrade repo and test began to make result less intuitive revealing more weakness of the system. Adding to the oddities of duplicate detection and branch matching, there was some room for improvement in the runbot models. This (small) commit introduce the runbot v5.0, designed for a closer match of odoo's development flows, and hopefully improving devs experience and making runbot configuration more flexible. **Remotes:** remote intoduction helps to detect duplicate between odoo and odoo-dev repos: a commit is now on a repo, a repo having multiple remote. If a hash is in odoo-dev, we consider that it is the same in odoo. Note: github seems to manage commit kind of the same way. It is possible to send a status on a commit on odoo when the commit only exists in odoo-dev. This change also allows to remove some repo duplicate configuration between a repo and his dev corresponding repo. (modules, server files, manifests, ...) **Trigger:** before v5.0, only one build per repo was created, making it difficult to tweak what test to execute in what case. The example use case was for upgrade. We want to test upgrade to master when pushing on odoo. But we also want to test upgrade the same way when pushing on upgrade. We introduce a build that should be ran on pushing on either repo when each repo already have specific tests. The trigger allows to specify a build to create with a specific config. The trigger is executed when any repo of the trigger repo is pushed. The trigger can define depedencies: only build enterprise when pushing enterprise, but enterprise needs odoo. Test upgrade to master when pushing either odoo or upgrade. Trigger will also allows to extract some build like cla that where executed on both enterprise and odoo, and hidden in a subbuild. **Bundle:** Cross repo branches/pr branches matching was hidden in build creation and can be confusing. A build can be detected as a duplicate of a pr, but not always if naming is wrong or traget is invalid/changes. This was mainly because of how a community ref will be found. This was making ci on pr undeterministic if duplicate matching fails. This was also creating two build, with one pointing to the other when duplicate detection was working, but the visual result can be confusing. Associtaions of remotes and bundles fix this by adding all pr and related branches from all repo in a bundle. First of all this helps to visualise what the runbot consider has branch matching and that should be considered as part of the same task, giving a place where to warn devs of some possible inconsistencies. Associate whith repo/remote, we can consider branches in the same repo in a bundle as expected to have the same head. Only one build is created since trigger considers repo, not remotes. **Batch:** A batch is a group of build, a batch on a bundle can be compared to a build on a branch in previous version. When a branch is pushed, the corresponding bundle creates a new batch, and wait for new commit. Once no new update are detected in the batch for 60 seconds, All the trigger are executed if elligible. The created build are added to the batch in a batch_slot. It is also possible that an corresponding build exists (duplicate) and is added to the slot instead of creating a new build. Co-authored-by d-fence <moc@odoo.com>
2020-06-03 21:17:42 +07:00
time_delta(time),
format="narrow",
threshold=2.1,
)
[IMP] runbot: runbot 5.0 Runbot initial architechture was working for a single odoo repo, and was adapted to build enterprise. Addition of upgrade repo and test began to make result less intuitive revealing more weakness of the system. Adding to the oddities of duplicate detection and branch matching, there was some room for improvement in the runbot models. This (small) commit introduce the runbot v5.0, designed for a closer match of odoo's development flows, and hopefully improving devs experience and making runbot configuration more flexible. **Remotes:** remote intoduction helps to detect duplicate between odoo and odoo-dev repos: a commit is now on a repo, a repo having multiple remote. If a hash is in odoo-dev, we consider that it is the same in odoo. Note: github seems to manage commit kind of the same way. It is possible to send a status on a commit on odoo when the commit only exists in odoo-dev. This change also allows to remove some repo duplicate configuration between a repo and his dev corresponding repo. (modules, server files, manifests, ...) **Trigger:** before v5.0, only one build per repo was created, making it difficult to tweak what test to execute in what case. The example use case was for upgrade. We want to test upgrade to master when pushing on odoo. But we also want to test upgrade the same way when pushing on upgrade. We introduce a build that should be ran on pushing on either repo when each repo already have specific tests. The trigger allows to specify a build to create with a specific config. The trigger is executed when any repo of the trigger repo is pushed. The trigger can define depedencies: only build enterprise when pushing enterprise, but enterprise needs odoo. Test upgrade to master when pushing either odoo or upgrade. Trigger will also allows to extract some build like cla that where executed on both enterprise and odoo, and hidden in a subbuild. **Bundle:** Cross repo branches/pr branches matching was hidden in build creation and can be confusing. A build can be detected as a duplicate of a pr, but not always if naming is wrong or traget is invalid/changes. This was mainly because of how a community ref will be found. This was making ci on pr undeterministic if duplicate matching fails. This was also creating two build, with one pointing to the other when duplicate detection was working, but the visual result can be confusing. Associtaions of remotes and bundles fix this by adding all pr and related branches from all repo in a bundle. First of all this helps to visualise what the runbot consider has branch matching and that should be considered as part of the same task, giving a place where to warn devs of some possible inconsistencies. Associate whith repo/remote, we can consider branches in the same repo in a bundle as expected to have the same head. Only one build is created since trigger considers repo, not remotes. **Batch:** A batch is a group of build, a batch on a bundle can be compared to a build on a branch in previous version. When a branch is pushed, the corresponding bundle creates a new batch, and wait for new commit. Once no new update are detected in the batch for 60 seconds, All the trigger are executed if elligible. The created build are added to the batch in a batch_slot. It is also possible that an corresponding build exists (duplicate) and is added to the slot instead of creating a new build. Co-authored-by d-fence <moc@odoo.com>
2020-06-03 21:17:42 +07:00
def s2human_long(time):
return format_timedelta(
time_delta(time),
threshold=2.1,
add_direction=True, locale='en'
)
@contextlib.contextmanager
def local_pgadmin_cursor():
cnx = None
try:
cnx = psycopg2.connect("dbname=postgres")
cnx.autocommit = True # required for admin commands
yield cnx.cursor()
finally:
if cnx:
cnx.close()
@contextlib.contextmanager
def local_pg_cursor(db_name):
cnx = None
try:
cnx = psycopg2.connect(f"dbname={db_name}")
yield cnx.cursor()
finally:
if cnx:
cnx.commit()
cnx.close()
def list_local_dbs(additionnal_conditions=None):
additionnal_condition_str = ''
if additionnal_conditions:
additionnal_condition_str = 'AND (%s)' % ' OR '.join(additionnal_conditions)
with local_pgadmin_cursor() as local_cr:
local_cr.execute("""
SELECT datname
FROM pg_database
WHERE pg_get_userbyid(datdba) = current_user
%s
""" % additionnal_condition_str)
return [d[0] for d in local_cr.fetchall()]
def pseudo_markdown(text):
text = html_escape(text)
# first, extract code blocs:
codes = []
def code_remove(match):
codes.append(match.group(1))
return f'<code>{len(codes)-1}</code>'
patterns = {
r'`(.+?)`': code_remove,
r'\*\*(.+?)\*\*': '<strong>\\g<1></strong>',
r'~~(.+?)~~': '<del>\\g<1></del>', # it's not official markdown but who cares
r'__(.+?)__': '<ins>\\g<1></ins>', # same here, maybe we should change the method name
r'\r?\n': '<br/>',
}
for p, b in patterns.items():
text = re.sub(p, b, text, flags=re.DOTALL)
# icons
re_icon = re.compile(r'@icon-([a-z0-9-]+)')
text = re_icon.sub('<i class="fa fa-\\g<1>"></i>', text)
# links
re_links = re.compile(r'\[(.+?)\]\((.+?)\)')
text = re_links.sub('<a href="\\g<2>">\\g<1></a>', text)
def code_replace(match):
return f'<code>{codes[int(match.group(1))]}</code>'
2022-06-07 20:56:26 +07:00
text = Markup(re.sub(r'<code>(\d+)</code>', code_replace, text, flags=re.DOTALL))
return text
def make_github_session(token):
session = requests.Session()
if token:
session.auth = (token, 'x-oauth-basic')
session.headers.update({'Accept': 'application/vnd.github.she-hulk-preview+json'})
return session
def sanitize(name):
for i in ['@', ':', '/', '\\', '..']:
name = name.replace(i, '_')
return name
class ReProxy():
@classmethod
def match(cls, *args, **kwrags):
return re.match(*args, **kwrags)
@classmethod
def search(cls, *args, **kwrags):
return re.search(*args, **kwrags)
@classmethod
def compile(cls, *args, **kwrags):
return re.compile(*args, **kwrags)
@classmethod
def findall(cls, *args, **kwrags):
return re.findall(*args, **kwrags)
VERBOSE = re.VERBOSE
MULTILINE = re.MULTILINE