caikit.runtime.model_management.batcher
=======================================

.. py:module:: caikit.runtime.model_management.batcher

.. autoapi-nested-parse::

   The Batcher transparently aggregates individual inference calls into unified
   batches to call the run_batch implementation of the wrapped model.



Attributes
----------

.. autoapisummary::

   caikit.runtime.model_management.batcher.log


Classes
-------

.. autoapisummary::

   caikit.runtime.model_management.batcher.Batcher


Module Contents
---------------

.. py:data:: log

.. py:class:: Batcher(model_name: str, model: caikit.core.ModuleBase, batch_size: int, batch_collect_delay_s: Optional[float] = None)

   .. py:attribute:: __doc__
      :value: Multiline-String

      .. raw:: html

         <details><summary>Show Value</summary>

      .. code-block:: python

         """
         The Batcher transparently aggregates individual inference calls into unified
         batches to call the run_batch implementation of the wrapped model.
         """

      .. raw:: html

         </details>




   .. py:attribute:: _model_name


   .. py:attribute:: _model


   .. py:attribute:: _batch_size


   .. py:attribute:: _batch_collect_delay_s
      :value: None



   .. py:attribute:: _input_q


   .. py:attribute:: _finished_tasks


   .. py:attribute:: _req_num
      :value: 0



   .. py:attribute:: _id_lock


   .. py:attribute:: _ready_event


   .. py:attribute:: _stop_event


   .. py:attribute:: _batch_thread_start_lock


   .. py:attribute:: _batch_thread
      :value: None



   .. py:attribute:: _model_run_defaults


   .. py:method:: __del__()

      Shut down the internal thread



   .. py:method:: run(**kwargs) -> caikit.core.data_model.base.DataBase

      This run function gives a facade to the underlying model's run
      function that is implemented by running batches of individual requests
      through the model's run_batch method.

      NOTE: Only kwargs accepted to simplify batching across inconsistent sets
          of kwargs (and only kwargs are used in the predict servicer)



   .. py:method:: stop()

      Stop this batcher's run thread (cannot be undone)



   .. py:method:: _ensure_batch_thread()

      The run thread will stop itself if there's no work to do, so this
      function is called to ensure that it's up and running



   .. py:method:: _next_req_id()

      Make a unique ID for this request



   .. py:method:: _batch_thread_run()

      This function runs in an independent thread and manages pulling
      requests from the input queue, running the batch, and returning the
      completed results into _finished_tasks.