caikit.runtime.servicers.global_predict_servicer
Attributes
Classes
This class contains RPC calls affiliated with the Caikit Runtime that are |
Module Contents
- caikit.runtime.servicers.global_predict_servicer.PREDICT_RPC_COUNTER
- caikit.runtime.servicers.global_predict_servicer.JOB_PREDICT_RPC_COUNTER
- caikit.runtime.servicers.global_predict_servicer.PREDICT_FROM_PROTO_SUMMARY
- caikit.runtime.servicers.global_predict_servicer.PREDICT_CAIKIT_LIBRARY_SUMMARY
- caikit.runtime.servicers.global_predict_servicer.PREDICT_TO_PROTO_SUMMARY
- caikit.runtime.servicers.global_predict_servicer.NON_PRIMITIVE_TYPES
- class caikit.runtime.servicers.global_predict_servicer.GlobalPredictServicer(inference_service: caikit.runtime.service_factory.ServicePackage, interrupter: caikit.runtime.work_management.abortable_context.ThreadInterrupter = None)[source]
This class contains RPC calls affiliated with the Caikit Runtime that are not a part of the Model Runtime proto definition. They will be serviced by mocking the particular RPC (based on the message type), leveraging the CaikitRuntimeServicerMock to return the appropriate mock response for a given request
- INPUT_SIZE_KEY = 'input-length'
- _started_metering = False
- _model_manager = None
- _rpc_meter = None
- _interrupter = None
- _inference_service
- _tracer
- Predict(request: google.protobuf.message.Message | Iterable[google.protobuf.message.Message], context: grpc.ServicerContext, caikit_rpc: caikit.runtime.service_generation.rpcs.TaskPredictRPC, *_, **__) google.protobuf.message.Message | Iterable[google.protobuf.message.Message][source]
Global predict RPC – Mocks the invocation of a Caikit Library module.run() method for a loaded Caikit Library model
- Args:
- request (ProtobufMessage):
A deserialized RPC request message
- context (ServicerContext):
Context object (contains request metadata, etc)
- Returns:
- response (Union[ProtobufMessage, Iterable[ProtobufMessage]]):
A Caikit Library data model response object
- StartPredictionJob(request: google.protobuf.message.Message | Iterable[google.protobuf.message.Message], context: grpc.ServicerContext, caikit_rpc: caikit.runtime.service_generation.rpcs.TaskPredictionJobRPC, *_, **__) caikit.interfaces.runtime.data_model.PredictionJob[source]
StartPredictionJob – Mocks the invocation of a Caikit Core Library ModelManager.start_prediction_job() method using a loaded model
- Args:
- request (ProtobufMessage):
A deserialized RPC request message
- context (ServicerContext):
Context object (contains request metadata, etc)
- caikit_rpc: TaskPredictionJobRPC
The RPC used to
- Returns:
- response (Union[ProtobufMessage, Iterable[ProtobufMessage]]):
A Caikit Library data model response object
- run_prediction_job(request_name: str, model_id: str, prediction_func_name: str = 'run', task: caikit.core.TaskBase | None = None, context: caikit.interfaces.runtime.data_model.RuntimeServerContextType | None = None, context_arg: str | None = None, model: caikit.core.ModuleBase | None = None, **kwargs) caikit.interfaces.runtime.data_model.PredictionJob[source]
Start a prediction job against the given model using the raw arguments to the model’s run function.
- Args:
- request_name (str):
The name of the request message to validate the model’s task
- model_id (str):
The ID of the loaded model
- prediction_func_name (str):
Explicit name of the prediction function to predict
- task (Optional[TaskBase])
The task to use for inference (if multitask model)
- context (Optional[RuntimeServerContextType]):
The context object from the inbound request
- context_arg (Optional[str]):
The arg name to the model inference method where the context should be passed
- model (Optional[ModuleBase]):
Pre-fetched model object
**kwargs: Keyword arguments to pass to the model’s run function
- Returns:
- response (Union[DataBase, Iterable[DataBase]]):
The object (unary) or objects (output stream) produced by the inference request
PredictionJob: _description_
- predict_model(request_name: str, model_id: str, inference_func_name: str = 'run', input_streaming: bool | None = None, output_streaming: bool | None = None, task: caikit.core.TaskBase | None = None, aborter: caikit.runtime.work_management.rpc_aborter.RpcAborter | None = None, context: caikit.interfaces.runtime.data_model.RuntimeServerContextType | None = None, context_arg: str | None = None, model: caikit.core.ModuleBase | None = None, **kwargs) caikit.core.data_model.DataBase | Iterable[caikit.core.data_model.DataBase][source]
Run a prediction against the given model using the raw arguments to the model’s run function.
- Args:
- request_name (str):
The name of the request message to validate the model’s task
- model_id (str):
The ID of the loaded model
- inference_func_name (str):
Explicit name of the inference function to predict (ignored if input_streaming and output_streaming set)
- input_streaming (Optional[bool]):
Use the task function with input streaming
- output_streaming (Optional[bool]):
Use the task function with output streaming
- task (Optional[TaskBase])
The task to use for inference (if multitask model)
- aborter (Optional[RpcAborter]):
If using abortable calls, this is the aborter to use
- context (Optional[RuntimeServerContextType]):
The context object from the inbound request
- context_arg (Optional[str]):
The arg name to the model inference method where the context should be passed
- model (Optional[ModuleBase]):
Pre-fetched model object
**kwargs: Keyword arguments to pass to the model’s run function
- Returns:
- response (Union[DataBase, Iterable[DataBase]]):
The object (unary) or objects (output stream) produced by the inference request
- notify_backends_with_context(model_id: str, context: caikit.interfaces.runtime.data_model.RuntimeServerContextType)[source]
Utility to notify all configured backends of the request context
- _verify_model_task(model: caikit.core.ModuleBase)[source]
Raise if the model is not supported for the task
- _build_caikit_library_request_stream(request_stream: Iterable[google.protobuf.message.Message], module_signature: caikit.core.signature_parsing.CaikitMethodSignature, caikit_rpc: caikit.runtime.service_generation.rpcs.TaskPredictRPC) Dict[str, Any][source]
Builds the kwargs dict to pass to a caikit module. Specifically handles the case of constructing input DataStreams for some parameters which are meant to be streamed in.
See caikit.runtime.build_caikit_library_request_dict