caikit.runtime.model_management.model_manager
Attributes
Classes
Model Manager class. The singleton class contains the core implementational details |
Module Contents
- caikit.runtime.model_management.model_manager.error
- caikit.runtime.model_management.model_manager.MODEL_SIZE_GAUGE
- caikit.runtime.model_management.model_manager.MODEL_COUNT_GAUGE
- caikit.runtime.model_management.model_manager.LOAD_MODEL_EXCEPTION_COUNTER
- caikit.runtime.model_management.model_manager.LOAD_MODEL_DURATION_SUMMARY
- class caikit.runtime.model_management.model_manager.ModelManager[source]
Model Manager class. The singleton class contains the core implementational details for the Model Runtime (i.e., load/unload functionality, etc). It does not provide the core details for predict calls.
- __instance = None
- __model_size_gauge_lock
- classmethod get_instance() ModelManager[source]
This method returns the instance of Model Manager
- loaded_models: Dict[str, caikit.runtime.model_management.loaded_model.LoadedModel]
- _loaded_models_lock
- _local_models_dir
- _lazy_load_local_models
- _lazy_load_poll_period_seconds
- _lazy_sync_timer = None
- _enable_lazy_load_poll
- _lazy_load_write_detection_period_seconds
- load_model(model_id: str, local_model_path: str, model_type: str, wait: bool = True, retries: int | None = None, finder: str | caikit.core.model_management.ModelFinderBase | None = None, initializer: str | caikit.core.model_management.ModelInitializerBase | None = None) caikit.runtime.model_management.loaded_model.LoadedModel[source]
Load a model using model_path (in Cloud Object Storage) & give it a model ID Args:
model_id (str): Model ID string for the model to load. local_model_path (str): Local path to load the model from. model_type (str): Type of the model to load. wait (bool): Wait for the model to finish loading retries (Optional[int]): Number of times to retry on load failure
- Returns:
model (LoadedModel): The LoadedModel instance
- sync_local_models(wait: bool = False, load: bool = True)[source]
Sync in-memory models with models in the configured local_model_dir
New models will be loaded and models previously loaded from local will be unloaded.
- Args:
wait (bool): After starting all loads, wait for them to complete load (bool): Perform loading during sync
- unload_model(model_id) int[source]
Unload a model by ID model.
- Args:
- model_id (string): Model ID string for the model to unload. If None,
(default) the default model id will be used.
- Returns:
Model_size (int) : Size of the loaded model in bytes
- get_model_size(model_id) int[source]
Look up size of a model by model ID. Args:
- model_id (string): Model ID string for the model. Throw Exception if empty,
or it is an ID of a model that is not currently loaded.
- Returns:
Model_size (int) : Size of the loaded model in bytes
- estimate_model_size(model_id, local_model_path, model_type) int[source]
Predict size of a model using model ID and path. Args:
model_id (string): Model ID string for the model to predict size of. local_model_path (string): Local path to the model. model_type (string): Type of the model
- Returns:
Model_size (int) : Estimated size of the model in bytes.
- retrieve_model(model_id: str) caikit.core.ModuleBase[source]
Retrieve a model from the loaded model map by model ID.
- Args:
model_id (str): Model ID of the model to retrieve
- Returns:
- response (caikit.core.module.ModuleBase):
A loaded Caikit model
- deploy_model(model_id: str, model_files: Dict[str, bytes], **kwargs) caikit.runtime.model_management.loaded_model.LoadedModel[source]
Given in-memory model files, this will save the model to the local models dir, then load it locally.
- undeploy_model(model_id: str)[source]
Remove the given model from the loaded model map and delete the artifacts from the local models dir.
- _local_models_dir_sync(wait: bool = False, load: bool = True)[source]
This function implements the mechanics of synchronizing the local_models_dir and the in-memory loaded_models map. It may raise and therefore errors should be handled by the wrapper function.
- NOTE: In the case that the local_models_dir becomes unreadable, it will
raise StopIteration to indicate that any periodic synchronization should terminate.
- _model_write_in_progress(model_dir: str) bool[source]
Returns true if model_dir is currently being written to. Uses the runtime.lazy_load_write_detection_period_seconds configuration to sleep between consecutive size checks of the directory.
Always returns false if runtime.lazy_load_write_detection_period_seconds is zero, negative, or None.
- static _get_total_disk_size(model_dir: str) int[source]
Returns the sum of st_size of all files contained within the directory structure rooted at model_dir.
- __report_total_model_size_metric()
- static __increment_model_count_metric(model_type, model_id)
- static __decrement_model_count_metric(model_type, model_id)
- static __increment_load_model_exception_count_metric(model_type)