Skip to content

Compose a pipeline

A pipeline is a machine learning workflow, consisting of one or more steps, to deploy code using Docker containers. By specifying the output of one step as the input of another step, the user can create a full pipeline formed with a computed acyclic graph (DAG).

Info

For the moment, the pipelines are only single-step. We are actively working on the integration of multi-steps for the next versions of the platform.

To deploy ML code in production, you need to go through a pipeline. So you have to go through a single-step pipeline in any case to perform an endpoint or another type of deployment.

Summary:

  1. Compose a mono-step pipeline
  2. Delete a pipeline
  3. Find and get pipeline information
Function name Method Return type Description
create_pipeline create_pipeline(pipeline_name, step_name) dict[str, str] Create a pipeline containing a single step.
get_pipeline get_pipeline(pipeline_name) dict Get a single pipeline if it exists.
delete_pipeline delete_pipeline(pipeline_name, force_deployments_deletion=False) dict Delete a pipeline identified by its name and ID.
list_pipelines list_pipelines() list of dict Get the list of all pipelines.

Create a mono-step pipeline

Before creating a pipeline, the step creation must be finished. You can check this by checking that the step's status are equal to Ready using the get_step() function.

SDK function pipeline

Function definition

pipeline = sdk.create_pipeline(
   pipeline_name="my_pipeline",
   step_name="my_step",
)

Parameters

  • pipeline_name (str) – Name of the pipeline to create.

  • step_name (str) – Name of the step to include in the pipeline.

!!! note The step should have the status “Ready” before being used to create the pipeline.

Returns

Created pipeline represented as dict using this keys:

  • pipeline_name (str): Pipeline name.
  • created_at (str): Pipeline date of creation.
  • steps (list[str]): List of step names.
  • open_inputs (list[dist]): List of all input of step.
  • input_name (str): Name of the open input.
  • step_name (str): Name of the step that provides the open input.
  • data_type (str): Data type of the open input.
  • description (str): Description of the open input.
  • default_value (str): Default value of the open input.
  • is_required (bool): Whether the open input is required or not.
  • open_outputs (list[dist]): List of all output of step.
  • output_name (str): Name of the open output.
  • step_name (str): Name of the step that provides the open output.
  • data_type (str): Data type of the open output.
  • description (str): Description of the open output.

Information about pipeline store

Pipeline can have multiple inputs and outputs, or no one. In fact, it’s dependent on the inputs and outputs of step inside. All input and output will be the same as the step inside. So, you have nothing to configure on pipeline creation for input and output.

Get information pipeline

Get information about one pipeline

Function definition

Get all information about one pipeline, referred by its name.

CraftAiSdk.get_pipeline(pipeline_name)

Parameters

  • pipeline_name (str) – Name of the pipeline to get.

Returns

The pipeline information in a dict (or None if the pipeline does not exist), with the following keys:

  • pipeline_name (str): Pipeline name.
  • created_at (str): Pipeline date of creation.
  • created_by (str): ID of the user who created the deployment.
  • last_execution_id (str): ID of the last execution of the pipeline.
  • steps (list[str]): List of step names.
  • open_inputs (list[dist]): List of all input of step.
  • input_name (str): Name of the open input.
  • step_name (str): Name of the step that provides the open input.
  • data_type (str): Data type of the open input.
  • description (str): Description of the open input.
  • default_value (str): Default value of the open input.
  • is_required (bool): Whether the open input is required or not.
  • open_outputs (list[dist]): List of all output of step.
  • output_name (str): Name of the open output.
  • step_name (str): Name of the step that provides the open output.
  • data_type (str): Data type of the open output.
  • description (str): Description of the open output.

Get all pipelines

Function definition

Get the list of all pipelines on your environment.

CraftAiSdk.list_pipelines()

Returns

List of pipelines represented as dict with keys :

  • "pipeline_name" (str): Name of pipeline
  • "created_at" (str): Create date of the pipeline.

Delete a pipeline

Function definition

Delete a pipeline from the current environment.

CraftAiSdk.delete_pipeline(pipeline_name, force_deployments_deletion=False)

Parameters

  • pipeline_name (str) – Name of the pipeline.
  • force_deployments_deletion (bool, optional) – if True, the associated endpoints will be deleted too. Defaults to False.

Returns

The deleted pipeline and its associated deleted deployments represented as a dict with the following keys:

  • pipeline (dict): Deleted pipeline represented as dict with the following keys:
  • name (str): Name of the deleted pipeline.
  • deployments (list): List of deleted deployments represented as dict with the following keys:
  • name (str): Name of the deleted deployments.
  • execution_rule (str): Execution rule of the deleted deployments.

Warning

You can't delete a pipeline that is used in a Deployment

Run a pipeline

Now that your pipeline is created you can run it directly from the SDK. Or you can configure a deployment.