Connect your project to Git repository

Usually, the platform is used with local files for creating the source code of pipelines. This approach has several drawbacks:

Lack of versioning: It's difficult to track changes made to the code.
Limited sharing: Sharing code with other team members is complex and inconvenient.
Poor practice for production: Working with local files is not recommended for production environments due to the lack of controls and security measures.

To overcome these limitations, we propose fetching the code directly from a Git repository. This allows easier versioning and code sharing. This practice is strongly encouraged for production projects, but not exclusively.

Here are the different steps needed to have everything ready:

Generate SSH Key
1. on Linux / MacOS
2. on Windows
Add key to repository
Use Git in Platform

Generate SSH Key

Warning

The topic of sharing private SSH Key is a sensitive one. Please do not share your private key and store it where you can find it again if needed and in some place that is not easily found ! Do not put your private key in the data store of the Platform.

First, you need to indicate in your Git repository that there is a secure way for you to communicate and send or receive files. This secure way is an SSH connection that uses a Private/Public SSH key encrypted with RSA Algorithm to identify and secure the connection.

The public key, must be set in the git environment administration settings for the repository (github or gitlab).
The private key, must be sent to the Platform, so it can access the repository.

On Linux / MacOS

Move to a new directory and run the following command, by replacing my-key-filename by a name of your choice.

ssh-keygen -m PEM -t rsa -b 4096 -f my-key-filename -q -N "" -C ""

This will generate two files: a file named my-key-filename which contains the “private key” used for creating a pipeline, and a file named my-key-filename.pub which contains the “public key” used to create the Deploy Key on GitHub.

On Windows

Install OpenSSH : Go in Settings > Apps & features > Optional feature to get list of features and click on add feature to find and install OpenSSH)

Go to the console :

Press the Windows key.
Type cmd.
Under Best Match, right-click Command Prompt.
Click Run as Administrator.
If prompted, click Yes in the ``Do you want to allow this app to make changes to your device?`` pop-up and then type

ssh-keygen

By default, the system will save the keys to C:Users\your_username.ssh\id_rsa. Choose a name for the file and press Enter. You’ll be asked to enter a passphrase.. A passphrase is a second security that locks the key and you need to type it everytime you use it. If you are familiar with this feature, define one, if not, leave it blank to bypass this.

The system will generate the key pair, and display the key fingerprint and a randomart image. Open your file browser and go to C:Users\your_username.ssh. You should see two files. The private key is saved in the id_rsa file and the public key is labeled id_rsa.pub. This is your SSH key pair.

Add key to repository

Now, you have to add the public key to the Git repository settings as a deployment key.

For GitHub :

Head to the homepage of your repository on GitHub.
Go to the Settings page.
Once there, select the tab on the left named Deploy Keys
Select Add deploy key on the Deploy Keys page.
Insert the name you want for your deploy key
Copy/paste the public key (content of *your-key-filename*.pub) in the second text box.
Click on “Add key” (you don’t need to allow write access)

For GitLab :

Head to the homepage of your repository on GitLab.
Click on Settings (left bar) then go to Repository
Click on Expand in the Deploy keys section
Insert the name you want for your deploy key.
Copy/paste the public key (content of *your-key-filename*.pub) in the second text box.
Click on Add key (you don’t need to Grant write permissions to this key)

Use Git in Platform

There are two methods to integrate the SSH key on the platform side:

In the project information
In the pipeline creation command in SDK

In project

If you want to have your whole project depending on a git repository, please fill the information in the Project settings. Once configured, every pipeline created will by default be with the files on you repository.

To do this, go to your project's settings page and enter these 3 parameters:

Repository URL : Enter the SSH URL of your repository. It should start with git@

How to get my repository URL

On GitHub, from your repository, click on the “Code” button and choose SSH.

On GitLab, from your repository, click on the "Clone" button and copy SSH URL.

Deploy key : Enter your Github / GitLab private key.

Warning

Remember to keep the begin and end tags when you copy/paste the key. It should look like this :

Default branch : Enter the Git branch you want as default for this project. If this field is empty, we will use the default Git branch. This can be overwritten in the Environment.

connection-git

Once this project information is saved, you can set up your pipelines so that the platform will use your Git repository by default:

sdk.create_pipeline(
    function_path="src/my-source-code.py",
    function_name="my-function-name", 
    pipeline_name="my-pipeline-name"
)

During pipeline creation

Wether you defined a git connection in the project or not, you can still specify a repository in the pipeline creation function in SDK. This parameter will overwrite the one in Project if it is defined.

For example:

.
├── sdk-platform-script.py
├── my-git-repo/
│   ├── README.md
│   ├── requirements.txt 
│   └── src/
│       └── my-source-code.py
└── keys/
    ├── my-key-filename
    └── my-key-filename.pub

The script sdk_platform_script.py is responsible for creating platform objects using the SDK. The file my_source_code.py contains the Python code that will be executed in the pipeline, and the keys directory contains the previously generated keys.

When creating a pipeline, you can configure your SDK script in this way to create a pipeline from the code in the Git repo:

sdk-platform-script.py

with open('keys/my-key-filename', 'r') as file:
    private_key_value = file.read().rstrip()

sdk.create_pipeline(
    function_path="src/my-source-code.py",
    function_name="my-function-name", 
    pipeline_name="my-pipeline-name",
    container_config = {
        "repository_url": "git@github.com:my-account/my-git-repo.git",
        "repository_deploy_key": private_key_value,
        "repository_branch": "main"
    }
)

This will create a pipeline using the code from my-source-code.py on the repository my-git-repo.