[IMP] base: ir.cron documentation

New function and description of how to write CRON jobs and related
best-practices.

task-4472661

content/contributing/development/coding_guidelines.rst

@@ -658,9 +658,15 @@ Never commit the transaction
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The Odoo framework is in charge of providing the transactional context for
-all RPC calls. The principle is that a new database cursor is opened at the
+all RPC calls.
-beginning of each RPC call, and committed when the call has returned, just
+All ``cr.commit()`` calls outside of the server framework from now on must
-before transmitting the answer to the RPC client, approximately like this:
+have an **explicit comment** explaining why they are absolutely necessary, why
+they are indeed correct, and why they do not break the transactions. Otherwise
+they can and will be removed!
+
+The principle is that a new database cursor is opened at the beginning of each
+RPC call, and committed when the call has returned, just before transmitting the
+answer to the RPC client, approximately like this:
 
 .. code-block:: python
 
@@ -671,7 +677,7 @@ before transmitting the answer to the RPC client, approximately like this:
         try:
             res = pool.execute_cr(cr, uid, obj, method, *args, **kw)
             cr.commit() # all good, we commit
-        except Exception:
+        except Exception:  # try to be more specific
             cr.rollback() # error, rollback everything atomically
             raise
         finally:
@@ -682,8 +688,7 @@ If any error occurs during the execution of the RPC call, the transaction is
 rolled back atomically, preserving the state of the system.
 
 Similarly, the system also provides a dedicated transaction during the execution
-of tests suites, so it can be rolled back or not depending on the server
+of tests suites and scheduled actions.
-startup options.
 
 The consequence is that if you manually call ``cr.commit()`` anywhere there is
 a very high chance that you will break the system in various ways, because you
@@ -697,9 +702,9 @@ among others:
    during the transaction)
 
 Here is the very simple rule:
-    You should **NEVER** call ``cr.commit()`` yourself, **UNLESS** you have
+    You should **NEVER** call ``cr.commit()`` or ``cr.rollback()`` yourself,
-    created your own database cursor explicitly! And the situations where you
+    **UNLESS** you have explicitly created your own database cursor!
-    need to do that are exceptional!
+    And the situations in which you need to do this are exceptional!
 
     And by the way if you did create your own cursor, then you need to handle
     error cases and proper rollback, as well as properly close the cursor when
@@ -707,20 +712,64 @@ Here is the very simple rule:
 
 And contrary to popular belief, you do not even need to call ``cr.commit()``
 in the following situations:
+
 - in the ``_auto_init()`` method of an *models.Model* object: this is taken
-care of by the addons initialization method, or by the ORM transaction when
+  care of by the addons initialization method, or by the ORM transaction when
-creating custom models
+  creating custom models
 - in reports: the ``commit()`` is handled by the framework too, so you can
-update the database even from within a report
+  update the database even from within a report
 - within *models.Transient* methods: these methods are called exactly like
-regular *models.Model* ones, within a transaction and with the corresponding
+  regular *models.Model* ones, within a transaction and with the corresponding
-``cr.commit()/rollback()`` at the end
+  ``cr.commit()/rollback()`` at the end
 - etc. (see general rule above if you are in doubt!)
 
-All ``cr.commit()`` calls outside of the server framework from now on must
+Avoid catching exceptions
-have an **explicit comment** explaining why they are absolutely necessary, why
+~~~~~~~~~~~~~~~~~~~~~~~~~
-they are indeed correct, and why they do not break the transactions. Otherwise
+
-they can and will be removed !
+Catch only specific exceptions, and avoid overly broad exception handling.
+Uncaught exceptions will be logged and handled properly by the framework.
+
+You should be specific about the types you catch and handle them
+accordingly, and you should limit the scope of your try-catch block as much
+as possible.
+
+.. code-block:: python
+
+    # BAD CODE
+    try:
+        do_something()
+    except Exception as e:
+        # if we caught a ValidationError, we did not rollback and we left the
+        # ORM in an undefined state
+        _logger.warning(e)
+
+For scheduled actions, you should rollback the changes if you catch errors and
+wish to continue. Scheduled actions run in a separate transaction, so you can
+rollback or commit directly when you signal progress.
+See :ref:`actions <reference/actions/cron>`
+
+If you must handle framework exceptions, you must use **savepoints**
+to isolate your function as much as possible.
+This will flush the computations when entering the block and rollback changes
+properly in case of exceptions.
+
+.. code-block:: python
+
+    try:
+        with self.env.cr.savepoint():
+            do_stuff()
+    except ...:
+        ...
+
+.. warning::
+
+    After you start more than 64 savepoints during a single transaction,
+    PostgreSQL will slow down.
+    In all cases, if the server runs replicas, savepoints have a huge overhead.
+    If you process records and savepoint in a loop, for example when processing
+    records one by one for a batch, limit the size of the batch.
+    If you have more records, maybe the function should become a schedule job
+    or you have to accept the performance penalty.
 
 Use translation method correctly
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

content/developer/reference/backend/actions.rst

@@ -442,34 +442,90 @@ Actions triggered automatically on a predefined frequency.
     Priority of the action when executing multiple actions at the same time
 
 
-Advanced use: Batching
+Writing CRON functions
 ----------------------
 
-When executing a scheduled action, it's recommended to try batching progress in order
+When running a scheduled action, it's recommended that you try to batch the
-to avoid hogging a worker for a long period of time and possibly running into timeout exceptions.
+progress in order to a worker for a long period of time and possibly run into
+timeout exceptions. Therefore you should split the processing so that each call
+make progress on some of the work to be done.
 
-Odoo provides a simple API for scheduled action batching;
+When writing a such a function, you should focus on processing a single batch.
+A batch should process one or many records and should generally take no more
+than *a few seconds*.
 
 .. code-block:: python
 
-      self.env['ir.cron']._notify_progress(done=XX:int, remaining=XX:int)
+    def _cron_do_something(self, *, limit=300):  # limit: allows for tweaking
+        domain = [('state', '=', 'ready')]
+        records = self.search(domain, limit=limit)
+        records.do_something()
+        # notify progression
+        remaining = 0 if len(records) == limit else self.search_count(domain)
+        self.env['ir.cron']._commit_progress(len(records), remaining=remaining)
 
-This method allows the scheduler to know if progress was made and whether there is
+Work is committed by the framework after each batch. The framework will
-still remaining work that must be done.
+call the function as many times as necessary to process the remaining work.
 
-By default, if the API is used, the scheduler tries to process 10 batches in one sitting.
+If called from outside of the cron job, the progress function call will have
-If there are still remaining tasks after those 10 batches, a new cron call will be executed as
+no effect.
+
+There are some cases, where you may want to keep resources between multiple
+batches or want to loop yourself to handle exceptions.
+If this is the case, you must notify the scheduler of the progress of your work
+by calling ``ir.cron._commit_progress`` and checking the result. The progress
+function returns the number of seconds remaining for the call; if it is 0, you
+must return as soon as possible.
+The following is an example of how to commit after each record that is
+processed, while keeping the connection open.
+
+.. code-block:: python
+
+    def _cron_do_something(self):
+        assert self.env.context.get('cron_id'), "Run only inside cron jobs"
+        domain = [('state', '=', 'ready')]
+        records = self.search(domain)
+        self.env['ir.cron']._commit_progress(remaining=len(records))
+
+        with open_some_connection() as conn:
+            for record in records:
+                # You may have other needs; we do some common stuff here:
+                # - lock record (also checks existence)
+                # - prefetch: break prefetch in this case, we process one record
+                # - filtered_domain: record may have changed
+                record = record.try_lock_for_update().filtered_domain(domain)
+                if not record:
+                    continue
+                # Processing the batch here...
+                try
+                    record.do_something(conn)
+                    if not self.env['ir.cron']._commit_progress(1):
+                        break
+                except Exception:
+                    # if you handle exceptions, the default stategy is to
+                    # rollback first the error
+                    self.env.cr.rollback()
+                    _logger.warning(...)
+                    # you may commit some status using _commit_progress
+
+Running CRON functions
+----------------------
+
+You should not call CRON functions directly.
+There are two ways to run functions: immediately (in the current thread, but
+still in a separate transaction) or trigger to start at a specific time or as
 soon as possible.
 
-Advanced use: Triggers
-----------------------
-
-For more complex use cases, Odoo provides a more advanced way to trigger
-scheduled actions directly from business code.
-
 .. code-block:: python
 
-      action_record._trigger(at=XX:date)
+    # run now (still in a separate transaction)
+    cron_record.method_direct_trigger()
+    # trigger
+    when: datetime | None = None  # for now
+    cron_record._trigger(when)
+
+Testing of a CRON function should be done by calling ``method_direct_trigger``
+in the registry test mode.
 
 Security
 --------

content/developer/reference/backend/orm/changelog.rst

@@ -4,6 +4,12 @@
 Changelog
 =========
 
+Odoo Online version 18.3
+========================
+
+- New CRON API for progress and documentation updates described in
+  `#197781 <https://github.com/odoo/odoo/pull/197781>`_.
+
 Odoo Online version 18.2
 ========================