caikit.runtime.client.remote_model_finder

The RemoteModelFinder locates models that are loaded in a remote runtime.

Configuration for RemoteModelFinder lives under the config as follows:

model_management:

finders:

<finder name>:

type: REMOTE config:

connection: ConnectionInfo <Required Connection Information> model_key: Optional[str]=MODEL_MESH_MODEL_ID_KEY <Optional setting to override the

grpc model name>

protocol: Optional[str]=”grpc” <protocol the remote server is using (grpc or http)> min_poll_time: Optional[int]=30 <minimum time before attempting to rediscover

models>

discover_models: Optional[bool]=True <bool to automatically discover remote models
via the /info/models endpoint>

supported_models: Optional[Dict[str, str]]={} <mapping of model names to module_ids
that this remote supports. This is automatically populated by discover_models> <model_path>: <module_id>

Attributes

`log`
`error`

Classes

`ModuleConnectionInfo`
`RemoteModelFinder`	A class can be constructed by a factory if its constructor takes exactly

Module Contents

caikit.runtime.client.remote_model_finder.log[source]

caikit.runtime.client.remote_model_finder.error

class caikit.runtime.client.remote_model_finder.ModuleConnectionInfo[source]

conn: caikit.interfaces.common.data_model.remote.ConnectionInfo

module_id: str

class caikit.runtime.client.remote_model_finder.RemoteModelFinder(config: aconfig.Config, instance_name: str)[source]

Bases: caikit.core.model_management.model_finder_base.ModelFinderBase

A class can be constructed by a factory if its constructor takes exactly one argument that is an aconfig.Config object and it has a name to identify itself with the factory.

__doc__ = Multiline-String

Show Value

"""
The RemoteModelFinder locates models that are loaded in a remote runtime.

Configuration for RemoteModelFinder lives under the config as follows:

model_management:
    finders:
        <finder name>:
            type: REMOTE
            config:
                connection: ConnectionInfo <Required Connection Information>
                model_key: Optional[str]=MODEL_MESH_MODEL_ID_KEY <Optional setting to override the
                    grpc model name>
                protocol: Optional[str]="grpc" <protocol the remote server is using (grpc or http)>
                min_poll_time: Optional[int]=30 <minimum time before attempting to rediscover
                    models>
                discover_models: Optional[bool]=True <bool to automatically discover remote models
                    via the /info/models endpoint>
                supported_models: Optional[Dict[str, str]]={} <mapping of model names to module_ids
                    that this remote supports. This is automatically populated by discover_models>
                    <model_path>: <module_id>

"""

name = 'REMOTE': This is the name of this constructible type that will be used by the factory to identify this class

_instance_name

_connections: Dict[str, caikit.interfaces.common.data_model.remote.ConnectionInfo]

_connection_template: caikit.interfaces.common.data_model.remote.ConnectionInfo | None = None

_model_key

_protocol

_supported_models: Dict[str, ModuleConnectionInfo]

_discover_models

_min_poll_time

find_model(model_path: str, **__) → caikit.runtime.client.remote_config.RemoteModuleConfig | None[source]: Check if the remote runtime supports the model_path

_discover(model_name: str | None = None) → Dict[str, ModuleConnectionInfo][source]

Helper method to discover models from a remote runtime. This is a separate function to help with subclassing

Returns:

model_map: Dict[str, str]: The map of models to modules

_safe_discover(model_name: str | None = None) → Dict[str, ModuleConnectionInfo][source]

Helper function that lazily discovers models in a thread safe manor. This function also ensures we don’t overload the remote server with discovery requests

Returns:: Dict[str, str]: Result of discover_models

_discover_grpc_models(model_name: str | None) → Dict[str, ModuleConnectionInfo][source]

Helper function to get all the supported models and modules from a remote GRPC runtime

Returns:

support_models: Dict[str, str: Mapping of remote model names to module ids

_discover_http_models(model_name: str | None) → Dict[str, caikit.interfaces.common.data_model.remote.ConnectionInfo][source]

Helper function to get all the supported models and modules from a remote HTTP runtime

Returns:

supported_models:Dict[str, str]: Mapping of remote model names to module_ids

_render_conn_template(model_name: str) → caikit.interfaces.common.data_model.remote.ConnectionInfo | None[source]: Common utility to get the connection for a given model

_get_conn_candidates(model_name: str | None) → List[caikit.interfaces.common.data_model.remote.ConnectionInfo][source]: Common utility to get all connections to try