caikit.runtime.model_management.model_manager ============================================= .. py:module:: caikit.runtime.model_management.model_manager Attributes ---------- .. autoapisummary:: caikit.runtime.model_management.model_manager.log caikit.runtime.model_management.model_manager.error caikit.runtime.model_management.model_manager.MODEL_SIZE_GAUGE caikit.runtime.model_management.model_manager.MODEL_COUNT_GAUGE caikit.runtime.model_management.model_manager.LOAD_MODEL_EXCEPTION_COUNTER caikit.runtime.model_management.model_manager.LOAD_MODEL_DURATION_SUMMARY Classes ------- .. autoapisummary:: caikit.runtime.model_management.model_manager.ModelManager Module Contents --------------- .. py:data:: log .. py:data:: error .. py:data:: MODEL_SIZE_GAUGE .. py:data:: MODEL_COUNT_GAUGE .. py:data:: LOAD_MODEL_EXCEPTION_COUNTER .. py:data:: LOAD_MODEL_DURATION_SUMMARY .. py:class:: ModelManager Model Manager class. The singleton class contains the core implementational details for the Model Runtime (i.e., load/unload functionality, etc). It does not provide the core details for predict calls. .. py:attribute:: __instance :value: None .. py:attribute:: __model_size_gauge_lock .. py:method:: get_instance() -> ModelManager :classmethod: This method returns the instance of Model Manager .. py:attribute:: model_loader :type: caikit.runtime.model_management.model_loader_base.ModelLoaderBase .. py:attribute:: model_sizer :type: caikit.runtime.model_management.model_sizer_base.ModelSizerBase .. py:attribute:: loaded_models :type: Dict[str, caikit.runtime.model_management.loaded_model.LoadedModel] .. py:attribute:: _loaded_models_lock .. py:attribute:: _local_models_dir .. py:attribute:: _lazy_load_local_models .. py:attribute:: _lazy_load_poll_period_seconds .. py:attribute:: _lazy_sync_timer :value: None .. py:attribute:: _enable_lazy_load_poll .. py:attribute:: _lazy_load_write_detection_period_seconds .. py:method:: shut_down() Shut down cache purging .. py:method:: load_model(model_id: str, local_model_path: str, model_type: str, wait: bool = True, retries: Optional[int] = None, finder: Optional[Union[str, caikit.core.model_management.ModelFinderBase]] = None, initializer: Optional[Union[str, caikit.core.model_management.ModelInitializerBase]] = None) -> caikit.runtime.model_management.loaded_model.LoadedModel Load a model using model_path (in Cloud Object Storage) & give it a model ID Args: model_id (str): Model ID string for the model to load. local_model_path (str): Local path to load the model from. model_type (str): Type of the model to load. wait (bool): Wait for the model to finish loading retries (Optional[int]): Number of times to retry on load failure Returns: model (LoadedModel): The LoadedModel instance .. py:method:: sync_local_models(wait: bool = False, load: bool = True) Sync in-memory models with models in the configured local_model_dir New models will be loaded and models previously loaded from local will be unloaded. Args: wait (bool): After starting all loads, wait for them to complete load (bool): Perform loading during sync .. py:method:: unload_model(model_id) -> int Unload a model by ID model. Args: model_id (string): Model ID string for the model to unload. If None, (default) the default model id will be used. Returns: Model_size (int) : Size of the loaded model in bytes .. py:method:: unload_all_models() Unload all loaded models .. py:method:: get_model_size(model_id) -> int Look up size of a model by model ID. Args: model_id (string): Model ID string for the model. Throw Exception if empty, or it is an ID of a model that is not currently loaded. Returns: Model_size (int) : Size of the loaded model in bytes .. py:method:: estimate_model_size(model_id, local_model_path, model_type) -> int Predict size of a model using model ID and path. Args: model_id (string): Model ID string for the model to predict size of. local_model_path (string): Local path to the model. model_type (string): Type of the model Returns: Model_size (int) : Estimated size of the model in bytes. .. py:method:: retrieve_model(model_id: str) -> caikit.core.ModuleBase Retrieve a model from the loaded model map by model ID. Args: model_id (str): Model ID of the model to retrieve Returns: response (caikit.core.module.ModuleBase): A loaded Caikit model .. py:method:: deploy_model(model_id: str, model_files: Dict[str, bytes], **kwargs) -> caikit.runtime.model_management.loaded_model.LoadedModel Given in-memory model files, this will save the model to the local models dir, then load it locally. .. py:method:: undeploy_model(model_id: str) Remove the given model from the loaded model map and delete the artifacts from the local models dir. .. py:method:: _local_models_dir_sync(wait: bool = False, load: bool = True) This function implements the mechanics of synchronizing the local_models_dir and the in-memory loaded_models map. It may raise and therefore errors should be handled by the wrapper function. NOTE: In the case that the local_models_dir becomes unreadable, it will raise StopIteration to indicate that any periodic synchronization should terminate. .. py:method:: _model_write_in_progress(model_dir: str) -> bool Returns true if model_dir is currently being written to. Uses the runtime.lazy_load_write_detection_period_seconds configuration to sleep between consecutive size checks of the directory. Always returns false if runtime.lazy_load_write_detection_period_seconds is zero, negative, or None. .. py:method:: _get_total_disk_size(model_dir: str) -> int :staticmethod: Returns the sum of st_size of all files contained within the directory structure rooted at model_dir. .. py:method:: __report_total_model_size_metric() .. py:method:: __increment_model_count_metric(model_type, model_id) :staticmethod: .. py:method:: __decrement_model_count_metric(model_type, model_id) :staticmethod: .. py:method:: __increment_load_model_exception_count_metric(model_type) :staticmethod: