Production Engineer Dashboard & Workflow

_images/Workflow_Block_Diagram_R22.png

The Production Engineer (PE) takes the models that are optimized and published by the ML Engineer and deploys them for live inference after validating them and comparing them to the existing inference models.

_images/Prod_Eng_Menu.png

Menu Item

Function

Model Catalog

Optimized models, ready for deployment

Model Serving

Staged or deployed inferences

The Model Catalog is the catalog described in the section Publish Model.

Deployment Workflow

After a Model has been published by the ML Eng, the PE can either stage or deploy the model through the icons on the far right side of the screen under “Actions”.

Function

Description

Stage

Deploy locally for testing

Deploy

Deploy for live inference

_images/Prod_Eng_Model_Catalog.png

Staging or deploying the model will create a serving endpoint. This endpoint can be used to:

  • Test the model using inference data to ensure that it meets the goals

  • Compare the results to the existing live inference model

  • Provide the endpoint for live serving if it meets the goals

Selecting “Deploy Model” from the Action buttons on the right will cause a popup to appear where the deployment options can be provided.

_images/Prod_Eng_Model_Catalog_Deploy.png

Field

Value

Name

User-chosen name for the Deployment

Description

Optional user-chosen name to provide more details for the Deployment

CPU/GPU

Type of inference

Minimum Replicas

Minimum number of inference pods that will run in the idle state with no inference requests (https://knative.dev/docs/serving/autoscaling/scale-bounds/)

Maximum Concurrent Requests

Soft target for the number of concurrent requests that a single inference pod can serve for the Model (https://knative.dev/docs/serving/autoscaling/concurrency/)

_images/Prod_Eng_Model_Serving.png

Change Model Deployment

After a Model has been deployed, it is associated with an endpoint URL. The Model associated with that endpoint can be changed from the Deployment screen. Select the Edit Action button to the right of the Model name. This will cause a Popup to appear that allows the Model version and other associated information to be changed for that endpoint.

_images/Data_Scientist_Model_Deploy_Change.png