From 1b6024e0b686f80c611bfc5a91aa1b0965c4dba5 Mon Sep 17 00:00:00 2001 From: Augusto Perez Date: Fri, 5 Jan 2024 12:59:42 +0100 Subject: [PATCH] [ADD] Upgrade documentation: Upgrade scripts and Util package closes odoo/documentation#7546 X-original-commit: 16a209c7500aeeb7c0cbc1c7d0f2438f39d2fb04 Signed-off-by: Antoine Vandevenne (anv) --- .../developer/howtos/upgrade_custom_db.rst | 93 ++++-- content/developer/reference.rst | 2 + .../developer/reference/upgrade_scripts.rst | 101 +++++++ content/developer/reference/upgrade_utils.rst | 279 ++++++++++++++++++ 4 files changed, 456 insertions(+), 19 deletions(-) create mode 100644 content/developer/reference/upgrade_scripts.rst create mode 100644 content/developer/reference/upgrade_utils.rst diff --git a/content/developer/howtos/upgrade_custom_db.rst b/content/developer/howtos/upgrade_custom_db.rst index a8122a747..276c2a588 100644 --- a/content/developer/howtos/upgrade_custom_db.rst +++ b/content/developer/howtos/upgrade_custom_db.rst @@ -33,7 +33,6 @@ These are the steps to follow to upgrade customized databases: #. :ref:`Test extensively and do a rehearsal `. #. :ref:`Upgrade the production database `. - .. _upgrade_custom/stop_developments: Step 1: Stop the developments @@ -47,14 +46,13 @@ Needless to say, bug fixing is exempt from this recommendation. Once you have stopped development, it is a good practice to assess the developments made and compare them with the features introduced between your current version and the version you are targeting. Challenge the developments as much as possible and find functional workarounds. Removing redundancy -between your developments and the standard version of Odoo will lead to an eased -upgrade process and reduce technical debt. +between your developments and the standard version of Odoo will lead to an eased upgrade process +and reduce technical debt. .. note:: You can find information on the changes between versions in the `Release Notes `_. - .. _upgrade_custom/request_upgrade: Step 2: Request an upgraded database @@ -71,7 +69,6 @@ properly. If that's not the case, and the upgrade request fails, request the ass the `support page `_ by selecting the option related to testing the upgrade. - .. _upgrade_custom/empty_database: Step 3: Empty database @@ -85,7 +82,7 @@ features, and guarantees that they will not cause any issues when upgrading the Making the custom modules work in an empty database also helps avoid changes and wrong configurations that might be present in the production database (like studio customization, customized website pages, email templates or translations). They are not intrinsically related to -the custom modules and that can raise unwanted issues in this stage of the upgraded process. +the custom modules and that can raise unwanted issues in this stage of the upgrade process. To make custom modules work on an empty database we advise to follow these steps: @@ -190,24 +187,84 @@ To make sure the custom code is working flawlessly in the new version, follow th Migrate the data ---------------- -During the upgrade of the custom modules, you might have to use migration scripts to reflect changes -from the source code to their corresponding data. +During the upgrade of the custom modules, you might have to use +:doc:`upgrade scripts <../reference/upgrade_scripts>` to reflect changes from the source code +to their corresponding data. Together with the upgrade scripts, you can also make use of the +:doc:`../reference/upgrade_utils` and its helper functions. - Any technical data that was renamed during the upgrade of the custom code (models, fields, - external identifiers) should be renamed using migration scripts to avoid data loss during the - module upgrade. + external identifiers) should be renamed using upgrade scripts to avoid data loss during the + module upgrade. See also: :meth:`rename_field`, :meth:`rename_model`, :meth:`rename_xmlid`. - Data from standard models removed from the source code of the newer Odoo version and from the database during the standard upgrade process might need to be recovered from the old model table if it is still present. -Migration scripts can also be used to: + .. example:: + Custom fields for model ``sale.subscription`` are not automatically migrated from Odoo 15 to + Odoo 16 (when the model was merged into ``sale.order``). In this case, a SQL query can be + executed on an upgrade script to move the data from one table to the other. Take into account + that all columns/fields must already exist, so consider doing this in a ``post-`` script (See + :ref:`upgrade-scripts/phases`). + + .. spoiler:: + + .. code-block:: python + + def migrate(cr, version): + cr.execute( + """ + UPDATE sale_order so + SET custom_field = ss.custom_field + FROM sale_subscription ss + WHERE ss.new_sale_order_id = so.id + """ + ) + + Check the documentation for more information on :doc:`../reference/upgrade_scripts`. + +Upgrade scripts can also be used to: - Ease the processing time of an upgrade. For example, to store the value of computed stored fields on models with an excessive number of records by using SQL queries. -- Recompute fields in case the computation of their value has changed. -- Uninstall unwanted custom modules. +- Recompute fields in case the computation of their value has changed. See also + :meth:`recompute_fields`. +- Uninstall unwanted custom modules. See also :meth:`remove_module`. - Correct faulty data or wrong configurations. +Running and testing upgrade scripts +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. tabs:: + + .. group-tab:: Odoo Online + + As the instalation of custom modules containing Python files is not allowed on Odoo Online + databases, it is not possible to run upgrade scripts on this platform. + + .. group-tab:: Odoo.sh + + As explained on the `Odoo.sh` tab of :ref:`upgrade/request-test-database`, Odoo.sh is + integrated with the upgrade platform. + + Once the upgrade of a staging branch is on "Update on commit" mode, each time a commit is + pushed on the branch, the upgraded backup is restored and all the custom modules are updated. + This update includes the execution of the upgrade scripts. + + When upgrading the production database, the execution of the upgrade scripts is also part of + the update of the custom modules done by the platform when the upgraded database is restored. + + .. group-tab:: On-premise + + Once you receive the upgraded dump of the database from the `Upgrade platform + `_, deploy the database and update all the custom modules by + invoking the command :doc:`odoo-bin ` in the shell. + To update the custom modules, use the option: `-u , + --update `. + + .. important:: + As mentioned in the :doc:`CLI documentation `, the command used + to call the CLI depends on how you installed Odoo. + .. _upgrade_custom/upgraded_database/test_custom: Test the custom modules @@ -221,13 +278,12 @@ Things to pay attention to: - Views not working: During the upgrade, if a view causes issues because of its content, it gets disabled. You can find the information on disabled views on the :ref:`Upgrade report - `. This view needs to be activated again. To achieve this, we recommend - the use of migration scripts. + `. This view needs to be activated again (or removed if not useful anymore). + To achieve this, we recommend the use of upgrade scripts. - :doc:`Module data <../tutorials/define_module_data>` not updated: Custom records that have the ``noupdate`` flag are not updated when upgrading the module in the new database. For the custom - data that needs to be updated due to changes in the new version, we recommend to use migration - scripts to do so. - + data that needs to be updated due to changes in the new version, we recommend to use upgrade + scripts to do so. See also: :meth:`update_record_from_xml`. .. _upgrade_custom/testing_rehearsal: @@ -247,7 +303,6 @@ In addition to that, make a full rehearsal of the upgrade process the day before production database to avoid undesired behavior during the upgrade and to detect any issue that might have occurred with the migrated data. - .. _upgrade_custom/production: Step 6: Production upgrade diff --git a/content/developer/reference.rst b/content/developer/reference.rst index 96ab9eb52..48c56b2d8 100644 --- a/content/developer/reference.rst +++ b/content/developer/reference.rst @@ -12,5 +12,7 @@ Reference reference/user_interface reference/standard_modules reference/cli + reference/upgrade_scripts + reference/upgrade_utils reference/external_api reference/extract_api diff --git a/content/developer/reference/upgrade_scripts.rst b/content/developer/reference/upgrade_scripts.rst new file mode 100644 index 000000000..3dc6c3251 --- /dev/null +++ b/content/developer/reference/upgrade_scripts.rst @@ -0,0 +1,101 @@ +=============== +Upgrade scripts +=============== + +An upgrade script is a Python file containing a function called :meth:`migrate`, which the upgrade +process invokes during the update of a module. + +.. method:: migrate(cr, version) + + :param cr: current database cursor + :type cr: :class:`~odoo.sql_db.Cursor` + :param str version: installed version of the module + +Typically, this function executes one or multiple SQL queries and can also access Odoo's ORM, as +well as the :doc:`./upgrade_utils`. + +Writing upgrade scripts +======================= + +Upgrade scripts follow a specific tree structure with a naming convention which determines when they +are executed. + +The structure of an upgrade script path is :file:`$module/migrations/$version/{pre,post,end}-*.py`, +where `$module` is the module for which the script will run, `$version` is the full version of the +module (including Odoo's major version and the module's minor version) and `{pre|post|end}-*.py` is +the file that needs to be executed. The file's name will determine the :ref:`phase +` and order in which it is executed for that module and version. + +.. note:: + Upgrade scripts are only executed when the module is being updated. Therefore, the + module's minor version set in the `$version` directory needs to be higher than the module's + installed version and equal or lower to the updated version of the module. + +.. example:: + + Directory structure of an upgrade script for a custom module named `awesome_partner` upgraded + to version `2.0` on Odoo 17. + + .. code-block:: text + + awesome_partner/ + |-- migrations/ + | |-- 17.0.2.0/ + | | |-- pre-exclamation.py + + Two upgrade scripts examples with the content of the :file:`pre-exclamation.py`, file adding + "!" at the end of partners' names: + + .. code-block:: python + + import logging + + _logger = logging.getLogger(__name__) + + + def migrate(cr, version): + cr.execute("UPDATE res_partner SET name = name || '!'") + _logger.info("Updated %s partners", cr.rowcount) + + .. code-block:: python + + import logging + from odoo.upgrade import util + + _logger = logging.getLogger(__name__) + + + def migrate(cr, version): + env = util.env(cr) + + partners = env["res.partner"].search([]) + for partner in partners: + partner.name += "!" + + _logger.info("Updated %s partners", len(partners)) + + Note that in the second example, the script takes advantage of the :doc:`./upgrade_utils` to + access the ORM. Check the documentation to find out more about this library. + +.. _upgrade-scripts/phases: + +Phases of upgrade scripts +========================= + +The upgrade process consists of three phases for each version of each module: + + #. The pre-phase, before the module is loaded. + #. The post-phase, after the module and its dependencies are loaded and updated. + #. The end-phase, after all modules have been loaded and updated for that version. + +Upgrade scripts are grouped according to the first part of their filenames into the corresponding +phase. Within each phase, the files are executed according to their lexical order. + +.. admonition:: Execution order of example scripts for one module in one version + + #. :file:`pre-10-do_something.py` + #. :file:`pre-20-something_else.py` + #. :file:`post-do_something.py` + #. :file:`post-something.py` + #. :file:`end-01-migrate.py` + #. :file:`end-migrate.py` diff --git a/content/developer/reference/upgrade_utils.rst b/content/developer/reference/upgrade_utils.rst new file mode 100644 index 000000000..7c5da204f --- /dev/null +++ b/content/developer/reference/upgrade_utils.rst @@ -0,0 +1,279 @@ +============= +Upgrade utils +============= + +`Upgrade utils `_ is a library that contains helper functions +to facilitate the writing of upgrade scripts. This library, used by Odoo for the upgrade scripts of +standard modules, provides reliability and helps speed up the upgrade process: + +- The helper functions help make sure the data is consistent in the database. +- It takes care of indirect references of the updated records. +- Allows calling functions and avoid writing code, saving time and reducing development risks. +- Helpers allow to focus on what is important for the upgrade and not think of details. + +Installation +============ + +Clone the `Upgrade utils repository `_ locally and start +``odoo`` with the ``src`` directory prepended to the ``--upgrade-path`` option. + +.. code-block:: console + + $ ./odoo-bin --upgrade-path=/path/to/upgrade-util/src,/path/to/other/upgrade/script/directory [...] + +On platforms where you do not manage Odoo yourself, you can install this library via `pip`: + +.. code-block:: console + + $ python3 -m pip install git+https://github.com/odoo/upgrade-util@master + +On `Odoo.sh `_ it is recommended to add it to the :file:`requirements.txt` of +the custom repository. For this, add the following line inside the file:: + + odoo_upgrade @ git+https://github.com/odoo/upgrade-util@master + +Using Upgrade utils +=================== + +Once installed, the following packages are available for the upgrade scripts: + +- :mod:`odoo.upgrade.util`: the helper itself. +- :mod:`odoo.upgrade.testing`: base TestCase classes. + +To use it in upgrade scripts, simply import it: + +.. code-block:: python + + from odoo.upgrade import util + + + def migrate(cr, version): + # Rest of the script + +Now, the helper functions are available to be called through ``util``. + +Util functions +============== + +Upgrade utils provides many useful functions to ease the upgrade process. Here, we describe some +of the most useful ones. Refer to the `util folder +`_ for the comprehensive declaration of +helper functions. + +.. note:: + + All util functions receive :attr:`cr` as a parameter. This refers to the database cursor. Pass + the one received as a parameter in :meth:`migrate`. + +Fields +------ + +.. `[source] `_ +.. method:: remove_field(cr, model, fieldname, cascade=False, drop_column=True, skip_inherit=()) + + Remove a field and its references from the database. + + :param str model: model name of the field to remove + :param str fieldname: name of the field to remove + :param bool cascade: whether the field's column and inheritance are removed in ``CASCADE`` + :param bool drop_column: whether the field's column is dropped + :param list(str) or str skip_inherit: list of models whose field's inheritance is skipped. + Use ``"*"`` to skip all inheritances + +.. `[source] `_ +.. method:: rename_field(cr, model, old, new, update_references=True, domain_adapter=None, skip_inherit=()) + + Rename a field and its references from ``old`` to ``new``. + + :param str model: model name of the field to rename + :param str old: current name od the field to rename + :param str new: new name od the field to rename + :param bool update_references: whether all references of field ``old`` to ``new`` are replaced + in: ``ir_filters``, ``ir_exports_line``, ``ir_act_server``, ``mail_alias``, + ``ir_ui_view_custom (dashboard)``, ``domains (using "domain_adapter")``, ``related fields`` + :param function domain_adapter: function that takes three arguments and returns a domain that + substitutes the original leaf: ``(leaf: Tuple[str,str,Any], in_or: bool, negated: bool)`` -> + ``List[Union[str,Tuple[str,str,Any]]]`` + :param list(str) or str skip_inherit: list of models whose field's inheritance is skipped. + Use ``"*"`` to skip all inheritances + +.. `[source] `_ +.. method:: move_field_to_module(cr, model, fieldname, old_module, new_module, skip_inherit=()) + + Move a field's reference in ``ir_model_data`` table from ``old_module`` to ``new_module``. + + :param str model: model name of the field to move + :param str fieldname: name of the field to move + :param str old_module: current module name of the field to move + :param str new_module: new module name of the field to move + :param list(str) or str skip_inherit: list of models whose field's inheritance is skipped. + Use ``"*"`` to skip all inheritances + +Models +------ + +.. `[source] `_ +.. method:: remove_model(cr, model, drop_table=True, ignore_m2m=()) + + Remove a model and its references from the database. + + :param str model: name of the model to remove + :param bool drop_table: whether the model's table is dropped + :param list(str) or str ignore_m2m: list of m2m tables ignored to remove. Use ``"*"`` to ignore + all m2m tables + +.. `[source] `_ +.. method:: rename_model(cr, old, new, rename_table=True) + + Rename a model and its references from ``old`` to ``new``. + + :param str old: current name of the model to rename + :param str new: new name of the model to rename + :param bool rename_table: whether the model's table is renamed + +.. `[source] `_ +.. method:: merge_model(cr, source, target, drop_table=True, fields_mapping=None, ignore_m2m=()) + + Merge the references from ``source`` model into ``target`` model and removes ``source`` model and + its references. By default, only the fields with the same name in both models are mapped. + + .. warning:: + This function does not move the records from ``source`` model to ``target`` model. + + :param str source: name of the source model of the merge + :param str target: name of the destination model of the merge + :param bool drop_table: whether the source model's table is dropped + :param dict fields_mapping: Dictionary ``{"source_model_field_1": "target_model_field_1", ...}`` + mapping fields with different names on both models + :param list(str) or str ignore_m2m: list of m2m tables ignored to remove from source model. + +Modules +------- + +.. `[source] `_ +.. method:: remove_module(cr, module) + + Uninstall and remove a module and its references from the database. + + :param str module: name of the module to remove + +.. `[source] `_ +.. method:: rename_module(cr, old, new) + + Rename a module and its references from ``old`` to ``new``. + + :param str old: current name of the module to rename + :param str new: new name of the module to rename + +.. `[source] `_ +.. method:: merge_module(cr, old, into, update_dependers=True) + + Move all references of module ``old`` into module ``into``. + + :param str old: name of the source module of the merge + :param str into: ame of the destination module of the merge + :param bool update_dependers: whether the dependencies of modules that depends on ``old`` are + updated + +ORM +--- + +.. `[source] `_ +.. method:: env(cr) + + Create a new environment from the cursor. + + .. warning:: + This function does NOT empty the cache maintained on the cursor for superuser with an empty + environment. A call to `invalidate_cache` will most probably be necessary every time you + directly modify something in database. + + :return: The new environment + :rtype: :class:`~odoo.api.Environment` + +.. `[source] `_ +.. method:: recompute_fields(cr, model, fields, ids=None, logger=_logger, chunk_size=256, strategy="auto") + + Recompute field values. Possible strategies to process the recomputation: + + - ``flush``: Flush the recomputation when it's finished + - ``commit``: Commit the recomputation when it's finished + - ``auto``: The function chooses the best alternative based on the number of records to recompute + and the fields traceability + + :param str model: model name of the field(s) to recompute + :param list(str) fields: list of field names to recompute + :param list(int) ids: list of record IDs to recompute + :param logger: Logger used to print the progress of the function + :type logger: :class:`logging.Logger` + :param int chunk_size: size of the chunk used to split the records for better processing + :param str strategy: strategy used to process the recomputation + +Records +------- + +.. `[source] `_ +.. method:: ref(cr, xmlid) + + Return the id corresponding to the given :term:`xml_id `. + + :param str xml_id: Record xml_id, under the format ```` + :return: Found record id, if any + :rtype: int or `None` + +.. `[source] `_ +.. method:: remove_record(cr, name) + + Remove a record and its references corresponding to the given + :term:`xml_id `. + + :param str name: record xml_id, under the format ```` + +.. `[source] `_ +.. method:: rename_xmlid(cr, old, new, noupdate=None, on_collision="fail") + + Rename the :term:`external Identifier` of a record. Possible actions to take if the external + Identifier already exists on the database: + + - ``fail``: raise ``MigrationError`` and prevent renaming + - ``merge``: renames the external Identifier and removes the old one + + :param str old: current xml_id of the record, under the format ```` + :param str new: new xml_id of the record, under the format ```` + :param bool noupdate: value to set on the ir_model_data record ``noupdate`` field + :param str on_collision: action to take if the xml_id already exists + +.. `[source] `_ +.. method:: ensure_xmlid_match_record(cr, xmlid, model, values) + + Match a record with an xmlid by creating or updating the external identifier. + + This function is useful when migrating in-database records into a custom module, to create the + record's xmlid before the module is updated and avoid duplication. + + :param str xmlid: record xml_id, under the format ```` + :param str model: model name of the record + :param dict values: Dictionary ``{"fieldname_1": "value_1", ...}`` mapping fields and values to + search for the record to update. For example: + + .. code-block:: python + + values = {"id": 123} + values = {"name": "INV/2024/0001", company_id: 1} + + :return: the :term:`xml_id ` of the record. + :rtype: str + +.. `[source] `_ +.. method:: update_record_from_xml(cr, xmlid, reset_write_metadata=True, force_create=True, from_module=None, reset_translations=()) + + Update a record based on its definition in the :doc:`/developer/reference/backend/data`. + + Useful to update ``noupdate`` records, in order to reset them for the upgraded version. + + :param str xmlid: record xml_id, under the format ```` + :param bool reset_write_metadata: whether the metadata before the record update is kept + :param bool force_create: whether the record is created if it does not exist + :param str from_module: name of the module from which to update the record. Useful when the + record is rewritten in another module. + :param set of str reset_translations: set of field names whose translations get reset