runbot/runbot_builder/dbmover.py
Xavier-Do 45721cdf6c [IMP] runbot: runbot 5.0
Runbot initial architechture was working for a single odoo repo, and was
adapted to build enterprise. Addition of upgrade repo and test began
to make result less intuitive revealing more weakness of the system.

Adding to the oddities of duplicate detection and branch matching,
there was some room for improvement in the runbot models.

This (small) commit introduce the runbot v5.0, designed for a closer
match of odoo's development flows, and hopefully improving devs
experience and making runbot configuration more flexible.

**Remotes:** remote intoduction helps to detect duplicate between odoo and
odoo-dev repos: a commit is now on a repo, a repo having multiple remote.
If a hash is in odoo-dev, we consider that it is the same in odoo.
Note: github seems to manage commit kind of the same way. It is possible
to send a status on a commit on odoo when the commit only exists in
odoo-dev.
This change also allows to remove some repo duplicate configuration
between a repo and his dev corresponding repo.
(modules, server files, manifests, ...)

**Trigger:** before v5.0, only one build per repo was created, making it
difficult to tweak what test to execute in what case. The example use
case was for upgrade. We want to test upgrade to master when pushing on
odoo. But we also want to test upgrade the same way when pushing on
upgrade. We introduce a build that should be ran on pushing on either
repo when each repo already have specific tests.
The trigger allows to specify a build to create with a specific config.
The trigger is executed when any repo of the trigger repo is pushed.
The trigger can define depedencies: only build enterprise when pushing
enterprise, but enterprise needs odoo. Test upgrade to master when pushing
either odoo or upgrade.
Trigger will also allows to extract some build like cla that where
executed on both enterprise and odoo, and hidden in a subbuild.

**Bundle:** Cross repo branches/pr branches matching was hidden in build
creation and can be confusing. A build can be detected as a duplicate
of a pr, but not always if naming is wrong or traget is invalid/changes.
This was mainly because of how a community ref will be found. This was
making ci on pr undeterministic if duplicate matching fails. This was
also creating two build, with one pointing to the other when duplicate
detection was working, but the visual result can be confusing.
Associtaions of remotes and bundles fix this by adding all pr and
related branches from all repo in a bundle. First of all this helps to
visualise what the runbot consider has branch matching and that should
be considered as part of the same task, giving a place where to warn
devs of some possible inconsistencies. Associate whith repo/remote, we
can consider branches in the same repo in a bundle as expected to have
the same head. Only one build is created since trigger considers repo,
not remotes.

**Batch:** A batch is a group of build, a batch on a bundle can be
compared to a build on a branch in previous version. When a branch
is pushed, the corresponding bundle creates a new batch, and wait for
new commit. Once no new update are detected in the batch for 60 seconds,
All the trigger are executed if elligible. The created build are added
to the batch in a batch_slot. It is also possible that an corresponding
build exists (duplicate) and is added to the slot instead of creating a
new build.

Co-authored-by d-fence <moc@odoo.com>
2020-09-10 13:44:38 +02:00

172 lines
6.2 KiB
Python
Executable File

#!/usr/bin/python3
import argparse
import contextlib
import logging
import psycopg2
import os
import re
import shutil
import sys
from collections import defaultdict
from logging.handlers import WatchedFileHandler
LOG_FORMAT = '%(asctime)s %(levelname)s %(name)s: %(message)s'
logging.basicConfig(level=logging.INFO, format=LOG_FORMAT)
logging.getLogger('odoo.addons.runbot').setLevel(logging.DEBUG)
logging.addLevelName(25, "!NFO")
_logger = logging.getLogger(__name__)
DBRE = r'^(?P<build_id>\d+)-.+-[0-9a-f]{6}-?(?P<db_suffix>.*)$'
@contextlib.contextmanager
def local_pgadmin_cursor():
cnx = None
try:
cnx = psycopg2.connect("dbname=postgres")
cnx.autocommit = True # required for admin commands
yield cnx.cursor()
finally:
if cnx:
cnx.close()
def list_local_dbs():
with local_pgadmin_cursor() as local_cr:
local_cr.execute("""
SELECT datname
FROM pg_database
WHERE pg_get_userbyid(datdba) = current_user
""")
return [d[0] for d in local_cr.fetchall()]
def _local_pg_rename_db(dbname, new_db_name):
with local_pgadmin_cursor() as local_cr:
pid_col = 'pid' if local_cr.connection.server_version >= 90200 else 'procpid'
query = 'SELECT pg_terminate_backend({}) FROM pg_stat_activity WHERE datname=%s'.format(pid_col)
local_cr.execute(query, [dbname])
local_cr.execute("ALTER DATABASE \"%s\" RENAME TO \"%s\";" % (dbname, new_db_name))
class RunbotClient():
def __init__(self, env):
self.env = env
def rename_build_dirs(self, args):
builds_root = os.path.join(self.env['runbot.runbot']._root(), 'build')
builds_backup_root = os.path.join(self.env['runbot.runbot']._root(), 'build-backup')
if not args.dry_run:
try:
_logger.info('Backup build dir in "%s"', builds_backup_root)
shutil.copytree(builds_root, builds_backup_root, copy_function=os.link)
except FileExistsError:
_logger.info('Backup path "%s" already exists, skipping', builds_backup_root)
build_dirs = {}
leftovers = []
for dir_name in os.listdir(builds_root):
match = re.match(DBRE, dir_name)
if match and match['db_suffix'] == '':
build_dirs[match['build_id']] = dir_name
else:
leftovers.append(dir_name)
for build in self.env['runbot.build'].search([('id', 'in', list(build_dirs.keys()))]):
origin_dir = build_dirs[str(build.id)]
origin_path = os.path.join(builds_root, origin_dir)
if origin_dir == build.dest:
_logger.info('Skip moving %s, already moved', build.dest)
continue
_logger.info('Moving "%s" --> "%s"', origin_dir, build.dest)
if args.dry_run:
continue
dest_path = os.path.join(builds_root, build.dest)
os.rename(origin_path, dest_path)
for leftover in leftovers:
_logger.info("leftover: %s", leftover)
def rename_databases(self, args):
total_db = 0
db_names = defaultdict(dict)
leftovers = []
for local_db_name in list_local_dbs():
match = re.match(DBRE, local_db_name)
if match and match['db_suffix'] != '':
db_names[match['build_id']][match['db_suffix']] = local_db_name
else:
leftovers.append(local_db_name)
total_db += 1
nb_matching = 0
ids = [int(i) for i in db_names.keys()]
builds = self.env['runbot.build'].search([('id', 'in', ids)])
for build in builds:
for suffix in db_names[str(build.id)].keys():
origin_name = db_names[str(build.id)][suffix]
dest_name = "%s-%s" % (build.dest, suffix)
nb_matching += 1
_logger.info('Renaming database "%s" --> "%s"', origin_name, dest_name)
if args.dry_run:
continue
_local_pg_rename_db(origin_name, dest_name)
_logger.info("Found %s databases", total_db)
_logger.info("Found %s matching databases", nb_matching)
_logger.info("Leftovers: %s", len(leftovers))
_logger.info("Builds not found : %s", len(set(ids) - set(builds.ids)))
def run():
# parse args
parser = argparse.ArgumentParser()
parser.add_argument('--odoo-path', help='Odoo sources path')
parser.add_argument('--db_host', default='127.0.0.1')
parser.add_argument('--db_port', default='5432')
parser.add_argument('--db_user')
parser.add_argument('--db_password')
parser.add_argument('-d', '--database', default='runbot_upgrade', help='name of runbot db')
parser.add_argument('--logfile', default=False)
parser.add_argument('-n', '--dry-run', action='store_true')
args = parser.parse_args()
if args.logfile:
dirname = os.path.dirname(args.logfile)
if dirname and not os.path.isdir(dirname):
os.makedirs(dirname)
handler = WatchedFileHandler(args.logfile)
formatter = logging.Formatter(LOG_FORMAT)
handler.setFormatter(formatter)
_logger.parent.handlers.clear()
_logger.parent.addHandler(handler)
# configure odoo
sys.path.append(args.odoo_path)
import odoo
_logger.info("Starting upgrade move script using database %s", args.database)
odoo.tools.config['db_host'] = args.db_host
odoo.tools.config['db_port'] = args.db_port
odoo.tools.config['db_user'] = args.db_user
odoo.tools.config['db_password'] = args.db_password
addon_path = os.path.abspath(os.path.join(os.path.dirname(os.path.realpath(__file__)), '..'))
config_addons_path = odoo.tools.config['addons_path']
odoo.tools.config['addons_path'] = ','.join([config_addons_path, addon_path])
# create environment
registry = odoo.registry(args.database)
with odoo.api.Environment.manage():
with registry.cursor() as cr:
env = odoo.api.Environment(cr, odoo.SUPERUSER_ID, {})
runbot_client = RunbotClient(env)
runbot_client.rename_build_dirs(args)
runbot_client.rename_databases(args)
if __name__ == '__main__':
run()
_logger.info("All done")