caikit.runtime.client

This module holds common utilities for connecting to caikit.runtime servers from client code

Submodules

Classes

`RemoteModuleConfig`	Helper class to differentiate a local ModuleConfig and a RemoteModuleConfig. The structure
`RemoteModelFinder`	A class can be constructed by a factory if its constructor takes exactly
`RemoteModelInitializer`	A class can be constructed by a factory if its constructor takes exactly
`RemoteModuleBase`	Class to act as the base for remote modules. This class will be subclassed and

Package Contents

class caikit.runtime.client.RemoteModuleConfig(config_dict)[source]

Bases: caikit.core.modules.config.ModuleConfig

Helper class to differentiate a local ModuleConfig and a RemoteModuleConfig. The structure should contain the following fields/structure

connection: caikit.interfaces.common.data_model.remote.ConnectionInfo

protocol: str

model_key: str

task_methods: List[Tuple[Type[caikit.core.task.TaskBase], List[RemoteRPCDescriptor]]]

train_method: RemoteRPCDescriptor

model_path: str

module_id: str

module_name: str

reserved_keys = []

classmethod load_from_module(module_reference: str | Type[caikit.core.modules.base.ModuleBase] | caikit.core.modules.base.ModuleBase, connection_info: caikit.interfaces.common.data_model.remote.ConnectionInfo, protocol: str, model_key: str, model_path: str) → RemoteModuleConfig[source]

Construct a new remote module configuration from an existing local Module

Args:

module_reference: Union[str, Type[ModuleBase]]:
Module_reference should either be the id of the locally loaded module, or a module class

model_path (str):
The path used to load this module

connection_info ConnectionInfo:
The connection information of the remote to use

protocol: str
The protocol to connect with

model_key: str
The model key to use when sending GRPC requests. An example is mm-model-id

Returns:: model_config (RemoteModuleConfig): Instantiated RemoteModuleConfig for model given model_path.

class caikit.runtime.client.RemoteModelFinder(config: aconfig.Config, instance_name: str)[source]

Bases: caikit.core.model_management.model_finder_base.ModelFinderBase

A class can be constructed by a factory if its constructor takes exactly one argument that is an aconfig.Config object and it has a name to identify itself with the factory.

__doc__ = Multiline-String

Show Value

"""
The RemoteModelFinder locates models that are loaded in a remote runtime.

Configuration for RemoteModelFinder lives under the config as follows:

model_management:
    finders:
        <finder name>:
            type: REMOTE
            config:
                connection: ConnectionInfo <Required Connection Information>
                model_key: Optional[str]=MODEL_MESH_MODEL_ID_KEY <Optional setting to override the
                    grpc model name>
                protocol: Optional[str]="grpc" <protocol the remote server is using (grpc or http)>
                min_poll_time: Optional[int]=30 <minimum time before attempting to rediscover
                    models>
                discover_models: Optional[bool]=True <bool to automatically discover remote models
                    via the /info/models endpoint>
                supported_models: Optional[Dict[str, str]]={} <mapping of model names to module_ids
                    that this remote supports. This is automatically populated by discover_models>
                    <model_path>: <module_id>

"""

name = 'REMOTE': This is the name of this constructible type that will be used by the factory to identify this class

_instance_name

_connections: Dict[str, caikit.interfaces.common.data_model.remote.ConnectionInfo]

_connection_template: caikit.interfaces.common.data_model.remote.ConnectionInfo | None = None

_model_key

_protocol

_supported_models: Dict[str, ModuleConnectionInfo]

_discover_models

_min_poll_time

find_model(model_path: str, **__) → caikit.runtime.client.remote_config.RemoteModuleConfig | None[source]: Check if the remote runtime supports the model_path

_discover(model_name: str | None = None) → Dict[str, ModuleConnectionInfo][source]

Helper method to discover models from a remote runtime. This is a separate function to help with subclassing

Returns:

model_map: Dict[str, str]: The map of models to modules

_safe_discover(model_name: str | None = None) → Dict[str, ModuleConnectionInfo][source]

Helper function that lazily discovers models in a thread safe manor. This function also ensures we don’t overload the remote server with discovery requests

Returns:: Dict[str, str]: Result of discover_models

_discover_grpc_models(model_name: str | None) → Dict[str, ModuleConnectionInfo][source]

Helper function to get all the supported models and modules from a remote GRPC runtime

Returns:

support_models: Dict[str, str: Mapping of remote model names to module ids

_discover_http_models(model_name: str | None) → Dict[str, caikit.interfaces.common.data_model.remote.ConnectionInfo][source]

Helper function to get all the supported models and modules from a remote HTTP runtime

Returns:

supported_models:Dict[str, str]: Mapping of remote model names to module_ids

_render_conn_template(model_name: str) → caikit.interfaces.common.data_model.remote.ConnectionInfo | None[source]: Common utility to get the connection for a given model

_get_conn_candidates(model_name: str | None) → List[caikit.interfaces.common.data_model.remote.ConnectionInfo][source]: Common utility to get all connections to try

class caikit.runtime.client.RemoteModelInitializer(config: aconfig.Config, instance_name: str)[source]

Bases: caikit.core.model_management.model_initializer_base.ModelInitializerBase

A class can be constructed by a factory if its constructor takes exactly one argument that is an aconfig.Config object and it has a name to identify itself with the factory.

__doc__ = Multiline-String

Show Value

"""
The RemoteModelInitializer loads a RemoteModuleConfig as an empty Module that
sends all requests to an external runtime server

Configuration for RemoteModelInitializer lives under the config as follows:

model_management:
    initializers:
        <initializer name>:
            type: REMOTE
"""

name = 'REMOTE': This is the name of this constructible type that will be used by the factory to identify this class

_instance_name

_module_class_map

init(model_config: caikit.runtime.client.remote_config.RemoteModuleConfig, **kwargs) → caikit.core.modules.ModuleBase | None[source]: Given a RemoteModuleConfig, initialize a RemoteModule instance

construct_module_class(model_config: caikit.runtime.client.remote_config.RemoteModuleConfig) → Type[caikit.core.modules.ModuleBase][source]

Helper function to construct a ModuleClass. This is a separate function to allow

for easy overloading

Args:

model_config: RemoteModuleConfig: The model config to construct the module from

Returns:

module: Type[ModuleBase]: The constructed module

class caikit.runtime.client.RemoteModuleBase(connection_info: caikit.interfaces.common.data_model.ConnectionInfo, protocol: str, model_key: str, model_name: str)[source]

Bases: caikit.core.modules.ModuleBase

Class to act as the base for remote modules. This class will be subclassed and mutated by construct_remote_module_class to make it have the same functions and parameters as the source module.

_model_name

_connection

_tls

_protocol

_model_key

_channel_lock

_conn_channel: grpc.Channel | requests.Session | None = None

_current_conn_time = None

_max_conn_delta

__del__()[source]: Destructor to ensure channel/session is cleaned up on deletion

classmethod generate_train_function(method: caikit.runtime.client.remote_config.RemoteRPCDescriptor) → Callable[source]: Factory function to construct a train function that will then be set as an attribute

classmethod generate_inference_function(task: Type[caikit.core.task.TaskBase], method: caikit.runtime.client.remote_config.RemoteRPCDescriptor) → Callable[source]: Factory function to construct inference functions that will be set as an attribute.

remote_method_request(method: caikit.runtime.client.remote_config.RemoteRPCDescriptor, service_type: caikit.runtime.names.ServiceType, *args, **kwargs) → Any[source]: Function to run a remote request based on the data stored in RemoteRPCDescriptor

_request_via_http(method: caikit.runtime.client.remote_config.RemoteRPCDescriptor, service_type: caikit.runtime.names.ServiceType, *args, **kwargs) → Any[source]

_request_via_grpc(method: caikit.runtime.client.remote_config.RemoteRPCDescriptor, service_type: caikit.runtime.names.ServiceType, *args, **kwargs) → Any[source]: Helper function to send a grpc request

property _grpc_channel: grpc.Channel: Helper function to construct a GRPC channel with correct credentials and TLS settings.

property _http_session: requests.Session: Helper function to construct a requests Session with with correct credentials and TLS settings.

_get_remote_object(construction_fn: Callable[[None], grpc.Channel | requests.Session]) → grpc.Channel | requests.Session[source]

Helper function to control construction of a grpc channel or http session

Args:: construction_fn (Callable[[None], Union[grpc.Channel, Session]]): _description_
Returns:: Union[grpc.Channel, Session]: _description_

_get_remote_target() → str[source]: Get the current remote target

static _get_streaming_arguments(**kwargs: Dict[str, Any]) → List[str][source]: Helper function to detect which kwargs are streaming

static _rename_union_sequence_types(obj: Any, dm_type: type)[source]

Helper function that renames all references in a dictionary to match the oneOf value of the DataModel and to collapse all Primitive sequences. This is required to match the format of http requests

For example:

{

“union_str”: “test”, “ints”: {

“values”:[1,2,3]

}

Becomes:

{: “union”: “test”, “ints”:[1,2,3]

}