NextERP/documentation
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[ADD] Upgrade documentation: Upgrade scripts and Util package

Browse Source 
closes odoo/documentation#7330

Signed-off-by: Antoine Vandevenne (anv) <anv@odoo.com>
This commit is contained in:
Augusto Perez 2024-01-05 12:59:42 +01:00
parent a838df1660
commit 16a209c750
4 changed files with 456 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

93
content/developer/howtos/upgrade_custom_db.rst
View File

@ -33,7 +33,6 @@ These are the steps to follow to upgrade customized databases:
#. :ref:`Test extensively and do a rehearsal <upgrade_custom/testing_rehearsal>`.
#. :ref:`Upgrade the production database <upgrade_custom/production>`.
.. _upgrade_custom/stop_developments:
Step 1: Stop the developments
@ -47,14 +46,13 @@ Needless to say, bug fixing is exempt from this recommendation.
Once you have stopped development, it is a good practice to assess the developments made and compare
them with the features introduced between your current version and the version you are targeting.
Challenge the developments as much as possible and find functional workarounds. Removing redundancy
between your developments and the standard version of Odoo will lead to an eased
upgrade process and reduce technical debt.
between your developments and the standard version of Odoo will lead to an eased upgrade process
and reduce technical debt.
.. note::
You can find information on the changes between versions in the `Release Notes
<https:/odoo.com/page/release-notes>`_.
.. _upgrade_custom/request_upgrade:
Step 2: Request an upgraded database
@ -71,7 +69,6 @@ properly. If that's not the case, and the upgrade request fails, request the ass
the `support page <https://odoo.com/help?stage=migration>`_ by selecting the option related to
testing the upgrade.
.. _upgrade_custom/empty_database:
Step 3: Empty database
@ -85,7 +82,7 @@ features, and guarantees that they will not cause any issues when upgrading the
Making the custom modules work in an empty database also helps avoid changes and wrong
configurations that might be present in the production database (like studio customization,
customized website pages, email templates or translations). They are not intrinsically related to
the custom modules and that can raise unwanted issues in this stage of the upgraded process.
the custom modules and that can raise unwanted issues in this stage of the upgrade process.
To make custom modules work on an empty database we advise to follow these steps:
@ -190,24 +187,84 @@ To make sure the custom code is working flawlessly in the new version, follow th
Migrate the data
----------------
During the upgrade of the custom modules, you might have to use migration scripts to reflect changes
from the source code to their corresponding data.
During the upgrade of the custom modules, you might have to use
:doc:`upgrade scripts <../reference/upgrade_scripts>` to reflect changes from the source code
to their corresponding data. Together with the upgrade scripts, you can also make use of the
:doc:`../reference/upgrade_utils` and its helper functions.
- Any technical data that was renamed during the upgrade of the custom code (models, fields,
external identifiers) should be renamed using migration scripts to avoid data loss during the
module upgrade.
external identifiers) should be renamed using upgrade scripts to avoid data loss during the
module upgrade. See also: :meth:`rename_field`, :meth:`rename_model`, :meth:`rename_xmlid`.
- Data from standard models removed from the source code of the newer Odoo version and from the
database during the standard upgrade process might need to be recovered from the old model table
if it is still present.
Migration scripts can also be used to:
.. example::
Custom fields for model ``sale.subscription`` are not automatically migrated from Odoo 15 to
Odoo 16 (when the model was merged into ``sale.order``). In this case, a SQL query can be
executed on an upgrade script to move the data from one table to the other. Take into account
that all columns/fields must already exist, so consider doing this in a ``post-`` script (See
:ref:`upgrade-scripts/phases`).
.. spoiler::
.. code-block:: python
def migrate(cr, version):
cr.execute(
"""
UPDATE sale_order so
SET custom_field = ss.custom_field
FROM sale_subscription ss
WHERE ss.new_sale_order_id = so.id
"""
)
Check the documentation for more information on :doc:`../reference/upgrade_scripts`.
Upgrade scripts can also be used to:
- Ease the processing time of an upgrade. For example, to store the value of computed stored fields
on models with an excessive number of records by using SQL queries.
- Recompute fields in case the computation of their value has changed.
- Uninstall unwanted custom modules.
- Recompute fields in case the computation of their value has changed. See also
:meth:`recompute_fields`.
- Uninstall unwanted custom modules. See also :meth:`remove_module`.
- Correct faulty data or wrong configurations.
Running and testing upgrade scripts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. tabs::
.. group-tab:: Odoo Online
As the instalation of custom modules containing Python files is not allowed on Odoo Online
databases, it is not possible to run upgrade scripts on this platform.
.. group-tab:: Odoo.sh
As explained on the `Odoo.sh` tab of :ref:`upgrade/request-test-database`, Odoo.sh is
integrated with the upgrade platform.
Once the upgrade of a staging branch is on "Update on commit" mode, each time a commit is
pushed on the branch, the upgraded backup is restored and all the custom modules are updated.
This update includes the execution of the upgrade scripts.
When upgrading the production database, the execution of the upgrade scripts is also part of
the update of the custom modules done by the platform when the upgraded database is restored.
.. group-tab:: On-premise
Once you receive the upgraded dump of the database from the `Upgrade platform
<https://upgrade.odoo.com>`_, deploy the database and update all the custom modules by
invoking the command :doc:`odoo-bin </developer/reference/cli>` in the shell.
To update the custom modules, use the option: `-u <modules>,
--update <modules>`.
.. important::
As mentioned in the :doc:`CLI documentation </developer/reference/cli>`, the command used
to call the CLI depends on how you installed Odoo.
.. _upgrade_custom/upgraded_database/test_custom:
Test the custom modules
@ -221,13 +278,12 @@ Things to pay attention to:
- Views not working: During the upgrade, if a view causes issues because of its content, it gets
disabled. You can find the information on disabled views on the :ref:`Upgrade report
<upgrade/upgrade_report>`. This view needs to be activated again. To achieve this, we recommend
the use of migration scripts.
<upgrade/upgrade_report>`. This view needs to be activated again (or removed if not useful anymore).
To achieve this, we recommend the use of upgrade scripts.
- :doc:`Module data <../tutorials/define_module_data>` not updated: Custom records that have the
``noupdate`` flag are not updated when upgrading the module in the new database. For the custom
data that needs to be updated due to changes in the new version, we recommend to use migration
scripts to do so.
data that needs to be updated due to changes in the new version, we recommend to use upgrade
scripts to do so. See also: :meth:`update_record_from_xml`.
.. _upgrade_custom/testing_rehearsal:
@ -247,7 +303,6 @@ In addition to that, make a full rehearsal of the upgrade process the day before
production database to avoid undesired behavior during the upgrade and to detect any issue that
might have occurred with the migrated data.
.. _upgrade_custom/production:
Step 6: Production upgrade

2
content/developer/reference.rst
View File

@ -12,5 +12,7 @@ Reference
reference/user_interface
reference/standard_modules
reference/cli
reference/upgrade_scripts
reference/upgrade_utils
reference/external_api
reference/extract_api

101
content/developer/reference/upgrade_scripts.rst Normal file
View File

@ -0,0 +1,101 @@
===============
Upgrade scripts
===============
An upgrade script is a Python file containing a function called :meth:`migrate`, which the upgrade
process invokes during the update of a module.
.. method:: migrate(cr, version)
:param cr: current database cursor
:type cr: :class:`~odoo.sql_db.Cursor`
:param str version: installed version of the module
Typically, this function executes one or multiple SQL queries and can also access Odoo's ORM, as
well as the :doc:`./upgrade_utils`.
Writing upgrade scripts
=======================
Upgrade scripts follow a specific tree structure with a naming convention which determines when they
are executed.
The structure of an upgrade script path is :file:`$module/migrations/$version/{pre,post,end}-*.py`,
where `$module` is the module for which the script will run, `$version` is the full version of the
module (including Odoo's major version and the module's minor version) and `{pre|post|end}-*.py` is
the file that needs to be executed. The file's name will determine the :ref:`phase
<upgrade-scripts/phases>` and order in which it is executed for that module and version.
.. note::
Upgrade scripts are only executed when the module is being updated. Therefore, the
module's minor version set in the `$version` directory needs to be higher than the module's
installed version and equal or lower to the updated version of the module.
.. example::
Directory structure of an upgrade script for a custom module named `awesome_partner` upgraded
to version `2.0` on Odoo 17.
.. code-block:: text
awesome_partner/
|-- migrations/
| |-- 17.0.2.0/
| | |-- pre-exclamation.py
Two upgrade scripts examples with the content of the :file:`pre-exclamation.py`, file adding
"!" at the end of partners' names:
.. code-block:: python
import logging
_logger = logging.getLogger(__name__)
def migrate(cr, version):
cr.execute("UPDATE res_partner SET name = name || '!'")
_logger.info("Updated %s partners", cr.rowcount)
.. code-block:: python
import logging
from odoo.upgrade import util
_logger = logging.getLogger(__name__)
def migrate(cr, version):
env = util.env(cr)
partners = env["res.partner"].search([])
for partner in partners:
partner.name += "!"
_logger.info("Updated %s partners", len(partners))
Note that in the second example, the script takes advantage of the :doc:`./upgrade_utils` to
access the ORM. Check the documentation to find out more about this library.
.. _upgrade-scripts/phases:
Phases of upgrade scripts
=========================
The upgrade process consists of three phases for each version of each module:
#. The pre-phase, before the module is loaded.
#. The post-phase, after the module and its dependencies are loaded and updated.
#. The end-phase, after all modules have been loaded and updated for that version.
Upgrade scripts are grouped according to the first part of their filenames into the corresponding
phase. Within each phase, the files are executed according to their lexical order.
.. admonition:: Execution order of example scripts for one module in one version
#. :file:`pre-10-do_something.py`
#. :file:`pre-20-something_else.py`
#. :file:`post-do_something.py`
#. :file:`post-something.py`
#. :file:`end-01-migrate.py`
#. :file:`end-migrate.py`

279
content/developer/reference/upgrade_utils.rst Normal file
View File

@ -0,0 +1,279 @@
=============
Upgrade utils
=============
`Upgrade utils <https://github.com/odoo/upgrade-util/>`_ is a library that contains helper functions
to facilitate the writing of upgrade scripts. This library, used by Odoo for the upgrade scripts of
standard modules, provides reliability and helps speed up the upgrade process:
- The helper functions help make sure the data is consistent in the database.
- It takes care of indirect references of the updated records.
- Allows calling functions and avoid writing code, saving time and reducing development risks.
- Helpers allow to focus on what is important for the upgrade and not think of details.
Installation
============
Clone the `Upgrade utils repository <https://github.com/odoo/upgrade-util/>`_ locally and start
``odoo`` with the ``src`` directory prepended to the ``--upgrade-path`` option.
.. code-block:: console
$ ./odoo-bin --upgrade-path=/path/to/upgrade-util/src,/path/to/other/upgrade/script/directory [...]
On platforms where you do not manage Odoo yourself, you can install this library via `pip`:
.. code-block:: console
$ python3 -m pip install git+https://github.com/odoo/upgrade-util@master
On `Odoo.sh <https://www.odoo.sh/>`_ it is recommended to add it to the :file:`requirements.txt` of
the custom repository. For this, add the following line inside the file::
odoo_upgrade @ git+https://github.com/odoo/upgrade-util@master
Using Upgrade utils
===================
Once installed, the following packages are available for the upgrade scripts:
- :mod:`odoo.upgrade.util`: the helper itself.
- :mod:`odoo.upgrade.testing`: base TestCase classes.
To use it in upgrade scripts, simply import it:
.. code-block:: python
from odoo.upgrade import util
def migrate(cr, version):
# Rest of the script
Now, the helper functions are available to be called through ``util``.
Util functions
==============
Upgrade utils provides many useful functions to ease the upgrade process. Here, we describe some
of the most useful ones. Refer to the `util folder
<https://github.com/odoo/upgrade-util/tree/master/src/util>`_ for the comprehensive declaration of
helper functions.
.. note::
All util functions receive :attr:`cr` as a parameter. This refers to the database cursor. Pass
the one received as a parameter in :meth:`migrate`.
Fields
------
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/fields.py#L91>`_
.. method:: remove_field(cr, model, fieldname, cascade=False, drop_column=True, skip_inherit=())
Remove a field and its references from the database.
:param str model: model name of the field to remove
:param str fieldname: name of the field to remove
:param bool cascade: whether the field's column and inheritance are removed in ``CASCADE``
:param bool drop_column: whether the field's column is dropped
:param list(str) or str skip_inherit: list of models whose field's inheritance is skipped.
Use ``"*"`` to skip all inheritances
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/fields.py#L362>`_
.. method:: rename_field(cr, model, old, new, update_references=True, domain_adapter=None, skip_inherit=())
Rename a field and its references from ``old`` to ``new``.
:param str model: model name of the field to rename
:param str old: current name od the field to rename
:param str new: new name od the field to rename
:param bool update_references: whether all references of field ``old`` to ``new`` are replaced
in: ``ir_filters``, ``ir_exports_line``, ``ir_act_server``, ``mail_alias``,
``ir_ui_view_custom (dashboard)``, ``domains (using "domain_adapter")``, ``related fields``
:param function domain_adapter: function that takes three arguments and returns a domain that
substitutes the original leaf: ``(leaf: Tuple[str,str,Any], in_or: bool, negated: bool)`` ->
``List[Union[str,Tuple[str,str,Any]]]``
:param list(str) or str skip_inherit: list of models whose field's inheritance is skipped.
Use ``"*"`` to skip all inheritances
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/fields.py#L337>`_
.. method:: move_field_to_module(cr, model, fieldname, old_module, new_module, skip_inherit=())
Move a field's reference in ``ir_model_data`` table from ``old_module`` to ``new_module``.
:param str model: model name of the field to move
:param str fieldname: name of the field to move
:param str old_module: current module name of the field to move
:param str new_module: new module name of the field to move
:param list(str) or str skip_inherit: list of models whose field's inheritance is skipped.
Use ``"*"`` to skip all inheritances
Models
------
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/models.py#L53>`_
.. method:: remove_model(cr, model, drop_table=True, ignore_m2m=())
Remove a model and its references from the database.
:param str model: name of the model to remove
:param bool drop_table: whether the model's table is dropped
:param list(str) or str ignore_m2m: list of m2m tables ignored to remove. Use ``"*"`` to ignore
all m2m tables
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/models.py#L203>`_
.. method:: rename_model(cr, old, new, rename_table=True)
Rename a model and its references from ``old`` to ``new``.
:param str old: current name of the model to rename
:param str new: new name of the model to rename
:param bool rename_table: whether the model's table is renamed
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/models.py#L323>`_
.. method:: merge_model(cr, source, target, drop_table=True, fields_mapping=None, ignore_m2m=())
Merge the references from ``source`` model into ``target`` model and removes ``source`` model and
its references. By default, only the fields with the same name in both models are mapped.
.. warning::
This function does not move the records from ``source`` model to ``target`` model.
:param str source: name of the source model of the merge
:param str target: name of the destination model of the merge
:param bool drop_table: whether the source model's table is dropped
:param dict fields_mapping: Dictionary ``{"source_model_field_1": "target_model_field_1", ...}``
mapping fields with different names on both models
:param list(str) or str ignore_m2m: list of m2m tables ignored to remove from source model.
Modules
-------
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/modules.py#L218>`_
.. method:: remove_module(cr, module)
Uninstall and remove a module and its references from the database.
:param str module: name of the module to remove
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/modules.py#L263>`_
.. method:: rename_module(cr, old, new)
Rename a module and its references from ``old`` to ``new``.
:param str old: current name of the module to rename
:param str new: new name of the module to rename
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/modules.py#L323>`_
.. method:: merge_module(cr, old, into, update_dependers=True)
Move all references of module ``old`` into module ``into``.
:param str old: name of the source module of the merge
:param str into: ame of the destination module of the merge
:param bool update_dependers: whether the dependencies of modules that depends on ``old`` are
updated
ORM
---
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/orm.py#L43>`_
.. method:: env(cr)
Create a new environment from the cursor.
.. warning::
This function does NOT empty the cache maintained on the cursor for superuser with an empty
environment. A call to `invalidate_cache` will most probably be necessary every time you
directly modify something in database.
:return: The new environment
:rtype: :class:`~odoo.api.Environment`
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/orm.py#L218>`_
.. method:: recompute_fields(cr, model, fields, ids=None, logger=_logger, chunk_size=256, strategy="auto")
Recompute field values. Possible strategies to process the recomputation:
- ``flush``: Flush the recomputation when it's finished
- ``commit``: Commit the recomputation when it's finished
- ``auto``: The function chooses the best alternative based on the number of records to recompute
and the fields traceability
:param str model: model name of the field(s) to recompute
:param list(str) fields: list of field names to recompute
:param list(int) ids: list of record IDs to recompute
:param logger: Logger used to print the progress of the function
:type logger: :class:`logging.Logger`
:param int chunk_size: size of the chunk used to split the records for better processing
:param str strategy: strategy used to process the recomputation
Records
-------
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/records.py#L612>`_
.. method:: ref(cr, xmlid)
Return the id corresponding to the given :term:`xml_id <external identifier>`.
:param str xml_id: Record xml_id, under the format ``<module.id>``
:return: Found record id, if any
:rtype: int or `None`
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/records.py#L281>`_
.. method:: remove_record(cr, name)
Remove a record and its references corresponding to the given
:term:`xml_id <external identifier>`.
:param str name: record xml_id, under the format ``<module.id>``
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/records.py#L548>`_
.. method:: rename_xmlid(cr, old, new, noupdate=None, on_collision="fail")
Rename the :term:`external Identifier` of a record. Possible actions to take if the external
Identifier already exists on the database:
- ``fail``: raise ``MigrationError`` and prevent renaming
- ``merge``: renames the external Identifier and removes the old one
:param str old: current xml_id of the record, under the format ``<module.id>``
:param str new: new xml_id of the record, under the format ``<module.id>``
:param bool noupdate: value to set on the ir_model_data record ``noupdate`` field
:param str on_collision: action to take if the xml_id already exists
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/records.py#L652>`_
.. method:: ensure_xmlid_match_record(cr, xmlid, model, values)
Match a record with an xmlid by creating or updating the external identifier.
This function is useful when migrating in-database records into a custom module, to create the
record's xmlid before the module is updated and avoid duplication.
:param str xmlid: record xml_id, under the format ``<module.id>``
:param str model: model name of the record
:param dict values: Dictionary ``{"fieldname_1": "value_1", ...}`` mapping fields and values to
search for the record to update. For example:
.. code-block:: python
values = {"id": 123}
values = {"name": "INV/2024/0001", company_id: 1}
:return: the :term:`xml_id <external identifier>` of the record.
:rtype: str
.. `[source] <https://github.com/odoo/upgrade-util/blob/master/src/util/records.py#L720>`_
.. method:: update_record_from_xml(cr, xmlid, reset_write_metadata=True, force_create=True, from_module=None, reset_translations=())
Update a record based on its definition in the :doc:`/developer/reference/backend/data`.
Useful to update ``noupdate`` records, in order to reset them for the upgraded version.
:param str xmlid: record xml_id, under the format ``<module.id>``
:param bool reset_write_metadata: whether the metadata before the record update is kept
:param bool force_create: whether the record is created if it does not exist
:param str from_module: name of the module from which to update the record. Useful when the
record is rewritten in another module.
:param set of str reset_translations: set of field names whose translations get reset