caikit.runtime.servicers.global_predict_servicer ================================================ .. py:module:: caikit.runtime.servicers.global_predict_servicer Attributes ---------- .. autoapisummary:: caikit.runtime.servicers.global_predict_servicer.PREDICT_RPC_COUNTER caikit.runtime.servicers.global_predict_servicer.JOB_PREDICT_RPC_COUNTER caikit.runtime.servicers.global_predict_servicer.PREDICT_FROM_PROTO_SUMMARY caikit.runtime.servicers.global_predict_servicer.PREDICT_CAIKIT_LIBRARY_SUMMARY caikit.runtime.servicers.global_predict_servicer.PREDICT_TO_PROTO_SUMMARY caikit.runtime.servicers.global_predict_servicer.log caikit.runtime.servicers.global_predict_servicer.NON_PRIMITIVE_TYPES Classes ------- .. autoapisummary:: caikit.runtime.servicers.global_predict_servicer.GlobalPredictServicer Module Contents --------------- .. py:data:: PREDICT_RPC_COUNTER .. py:data:: JOB_PREDICT_RPC_COUNTER .. py:data:: PREDICT_FROM_PROTO_SUMMARY .. py:data:: PREDICT_CAIKIT_LIBRARY_SUMMARY .. py:data:: PREDICT_TO_PROTO_SUMMARY .. py:data:: log .. py:data:: NON_PRIMITIVE_TYPES .. py:class:: GlobalPredictServicer(inference_service: caikit.runtime.service_factory.ServicePackage, interrupter: caikit.runtime.work_management.abortable_context.ThreadInterrupter = None) This class contains RPC calls affiliated with the Caikit Runtime that are not a part of the Model Runtime proto definition. They will be serviced by mocking the particular RPC (based on the message type), leveraging the CaikitRuntimeServicerMock to return the appropriate mock response for a given request .. py:attribute:: INPUT_SIZE_KEY :value: 'input-length' .. py:attribute:: _started_metering :value: False .. py:attribute:: _model_manager :value: None .. py:attribute:: _rpc_meter :value: None .. py:attribute:: _interrupter :value: None .. py:attribute:: _inference_service .. py:attribute:: _tracer .. py:method:: Predict(request: Union[google.protobuf.message.Message, Iterable[google.protobuf.message.Message]], context: grpc.ServicerContext, caikit_rpc: caikit.runtime.service_generation.rpcs.TaskPredictRPC, *_, **__) -> Union[google.protobuf.message.Message, Iterable[google.protobuf.message.Message]] Global predict RPC -- Mocks the invocation of a Caikit Library module.run() method for a loaded Caikit Library model Args: request (ProtobufMessage): A deserialized RPC request message context (ServicerContext): Context object (contains request metadata, etc) Returns: response (Union[ProtobufMessage, Iterable[ProtobufMessage]]): A Caikit Library data model response object .. py:method:: StartPredictionJob(request: Union[google.protobuf.message.Message, Iterable[google.protobuf.message.Message]], context: grpc.ServicerContext, caikit_rpc: caikit.runtime.service_generation.rpcs.TaskPredictionJobRPC, *_, **__) -> caikit.interfaces.runtime.data_model.PredictionJob StartPredictionJob -- Mocks the invocation of a Caikit Core Library ModelManager.start_prediction_job() method using a loaded model Args: request (ProtobufMessage): A deserialized RPC request message context (ServicerContext): Context object (contains request metadata, etc) caikit_rpc: TaskPredictionJobRPC The RPC used to Returns: response (Union[ProtobufMessage, Iterable[ProtobufMessage]]): A Caikit Library data model response object .. py:method:: run_prediction_job(request_name: str, model_id: str, prediction_func_name: str = 'run', task: Optional[caikit.core.TaskBase] = None, context: Optional[caikit.interfaces.runtime.data_model.RuntimeServerContextType] = None, context_arg: Optional[str] = None, model: Optional[caikit.core.ModuleBase] = None, **kwargs) -> caikit.interfaces.runtime.data_model.PredictionJob Start a prediction job against the given model using the raw arguments to the model's run function. Args: request_name (str): The name of the request message to validate the model's task model_id (str): The ID of the loaded model prediction_func_name (str): Explicit name of the prediction function to predict task (Optional[TaskBase]) The task to use for inference (if multitask model) context (Optional[RuntimeServerContextType]): The context object from the inbound request context_arg (Optional[str]): The arg name to the model inference method where the context should be passed model (Optional[ModuleBase]): Pre-fetched model object **kwargs: Keyword arguments to pass to the model's run function Returns: response (Union[DataBase, Iterable[DataBase]]): The object (unary) or objects (output stream) produced by the inference request PredictionJob: _description_ .. py:method:: predict_model(request_name: str, model_id: str, inference_func_name: str = 'run', input_streaming: Optional[bool] = None, output_streaming: Optional[bool] = None, task: Optional[caikit.core.TaskBase] = None, aborter: Optional[caikit.runtime.work_management.rpc_aborter.RpcAborter] = None, context: Optional[caikit.interfaces.runtime.data_model.RuntimeServerContextType] = None, context_arg: Optional[str] = None, model: Optional[caikit.core.ModuleBase] = None, **kwargs) -> Union[caikit.core.data_model.DataBase, Iterable[caikit.core.data_model.DataBase]] Run a prediction against the given model using the raw arguments to the model's run function. Args: request_name (str): The name of the request message to validate the model's task model_id (str): The ID of the loaded model inference_func_name (str): Explicit name of the inference function to predict (ignored if input_streaming and output_streaming set) input_streaming (Optional[bool]): Use the task function with input streaming output_streaming (Optional[bool]): Use the task function with output streaming task (Optional[TaskBase]) The task to use for inference (if multitask model) aborter (Optional[RpcAborter]): If using abortable calls, this is the aborter to use context (Optional[RuntimeServerContextType]): The context object from the inbound request context_arg (Optional[str]): The arg name to the model inference method where the context should be passed model (Optional[ModuleBase]): Pre-fetched model object **kwargs: Keyword arguments to pass to the model's run function Returns: response (Union[DataBase, Iterable[DataBase]]): The object (unary) or objects (output stream) produced by the inference request .. py:method:: stop_metering() .. py:method:: notify_backends_with_context(model_id: str, context: caikit.interfaces.runtime.data_model.RuntimeServerContextType) Utility to notify all configured backends of the request context .. py:method:: _handle_predict_exceptions(model_id: str, request_name: str) .. py:method:: _verify_model_task(model: caikit.core.ModuleBase) Raise if the model is not supported for the task .. py:method:: _build_caikit_library_request_stream(request_stream: Iterable[google.protobuf.message.Message], module_signature: caikit.core.signature_parsing.CaikitMethodSignature, caikit_rpc: caikit.runtime.service_generation.rpcs.TaskPredictRPC) -> Dict[str, Any] Builds the kwargs dict to pass to a caikit module. Specifically handles the case of constructing input `DataStreams` for some parameters which are meant to be streamed in. See caikit.runtime.build_caikit_library_request_dict