Compose a pipeline

A pipeline is a machine learning workflow, consisting of one or more steps, to deploy code using Docker containers. By specifying the output of one step as the input of another step, the user can create a full pipeline formed with a computed acyclic graph (DAG).

Info

For the moment, the pipelines are only single-step. We are actively working on the integration of multi-steps for the next versions of the platform.

To deploy ML code in production, you need to go through a pipeline. So you have to go through a single-step pipeline in any case to perform an endpoint or another type of deployment.

Summary:

Compose a mono-step pipeline
Delete a pipeline
Find and get pipeline information

Function name	Method	Return type	Description
create_pipeline	create_pipeline(pipeline_name, step_name)	dict[str, str]	Create a pipeline containing a single step.
get_pipeline	get_pipeline(pipeline_name)	dict	Get a single pipeline if it exists.
delete_pipeline	delete_pipeline(pipeline_name, force_deployments_deletion=False)	dict	Delete a pipeline identified by its name and ID.
list_pipelines	list_pipelines()	list of dict	Get the list of all pipelines.

Create a mono-step pipeline

Before creating a pipeline, the step creation must be finished. You can check this by checking that the step's status are equal to Ready using the get_step() function.

SDK function pipeline

Function definition

pipeline = sdk.create_pipeline(
   pipeline_name="my_pipeline",
   step_name="my_step",
)

Parameters

pipeline_name (str) – Name of the pipeline to create.
step_name (str) – Name of the step to include in the pipeline.

!!! note The step should have the status “Ready” before being used to create the pipeline.

Returns

Created pipeline represented as dict using this keys:

pipeline_name (str): Pipeline name.
created_at (str): Pipeline date of creation.
steps (list[str]): List of step names.
open_inputs (list[dist]): List of all input of step.
input_name (str): Name of the open input.
step_name (str): Name of the step that provides the open input.
data_type (str): Data type of the open input.
description (str): Description of the open input.
default_value (str): Default value of the open input.
is_required (bool): Whether the open input is required or not.
open_outputs (list[dist]): List of all output of step.
output_name (str): Name of the open output.
step_name (str): Name of the step that provides the open output.
data_type (str): Data type of the open output.
description (str): Description of the open output.

Information about pipeline store

Pipeline can have multiple inputs and outputs, or no one. In fact, it’s dependent on the inputs and outputs of step inside. All input and output will be the same as the step inside. So, you have nothing to configure on pipeline creation for input and output.

Get information pipeline

Get information about one pipeline

Function definition

Get all information about one pipeline, referred by its name.

CraftAiSdk.get_pipeline(pipeline_name)

Parameters

pipeline_name (str) – Name of the pipeline to get.

Returns

The pipeline information in a dict (or None if the pipeline does not exist), with the following keys:

pipeline_name (str): Pipeline name.
created_at (str): Pipeline date of creation.
created_by (str): ID of the user who created the deployment.
last_execution_id (str): ID of the last execution of the pipeline.
steps (list[str]): List of step names.
deployments (list[str]): List of deployment names which are associated to the pipeline.
open_inputs (list[dist]): List of all input of step.
input_name (str): Name of the open input.
step_name (str): Name of the step that provides the open input.
data_type (str): Data type of the open input.
description (str): Description of the open input.
default_value (str): Default value of the open input.
is_required (bool): Whether the open input is required or not.
open_outputs (list[dist]): List of all output of step.
output_name (str): Name of the open output.
step_name (str): Name of the step that provides the open output.
data_type (str): Data type of the open output.
description (str): Description of the open output.

Get all pipelines

Function definition

Get the list of all pipelines on your environment.

CraftAiSdk.list_pipelines()

Returns

List of pipelines represented as dict with keys :

"pipeline_name" (str): Name of pipeline
"created_at" (str): Create date of the pipeline.

Delete a pipeline

Function definition

Delete a pipeline from the current environment.

CraftAiSdk.delete_pipeline(pipeline_name, force_deployments_deletion=False)

Parameters

pipeline_name (str) – Name of the pipeline.
force_deployments_deletion (bool, optional) – if True, the associated endpoints will be deleted too. Defaults to False.

Returns

The deleted pipeline and its associated deleted deployments represented as a dict with the following keys:

pipeline (dict): Deleted pipeline represented as dict with the following keys:
name (str): Name of the deleted pipeline.
deployments (list): List of deleted deployments represented as dict with the following keys:
name (str): Name of the deleted deployments.
type (str): Type of the deleted deployments.

Warning

You can't delete a pipeline that is used in a Deployment

Run a pipeline

Now that your pipeline is created you can run it directly from the SDK. Or you can configure a deployment.