Connect your project to Git repository
Usually, the platform is used with local files for creating the source code of pipelines. This approach has several drawbacks:
- Lack of versioning: It's difficult to track changes made to the code.
- Limited sharing: Sharing code with other team members is complex and inconvenient.
- Poor practice for production: Working with local files is not recommended for production environments due to the lack of controls and security measures.
To overcome these limitations, we propose fetching the code directly from a Git repository. This allows easier versioning and code sharing. This practice is strongly encouraged for production projects, but not exclusively.
Here are the different steps needed to have everything ready:
Generate SSH Key
Warning
The topic of sharing private SSH Key is a sensitive one. Please do not share your private key everywhere and store it where you can find it again if needed and in some place that is not easily found ! Do not put your private key in the datastorage of the Platform.
First, you need to tell Git that there is a secure way for you to communicate and send or receive files. This secure way is an SSH connection that uses a Private/Public SSH key encrypted with RSA Algorithm to identify and secure the connection.
- The public key, must be set in the git environment administration settings for the repository (github or gitlab).
- The private key, must be sent to the Platform, so it can access the repository.
On Linux / MacOS
Move to a new directory and run the following command, by replacing my-key-filename
by a name of your choosing.
This will generate two files: a file named my-key-filename
with the “private key” used for creating a pipeline, and a file named
my-key-filename.pub
with the “public key” used to create the Deploy Key on GitHub.
On Windows
Install OpenSSH : Go in Settings > Apps & features > Optional feature to get list of features and click on add feature
to find and install OpenSSH)
Go to the console :
- Press the Windows key.
- Type cmd.
- Under Best Match, right-click Command Prompt.
- Click Run as Administrator.
- If prompted, click Yes in the ``Do you want to allow this app to make changes to your device?`` pop-up and then type
By default, the system will save the keys to C:Users\your_username.ssh\id_rsa. Choose a name for the file and press Enter. You’ll be asked to enter a passphrase.. A passphrase
is a second security that locks the key and you need to type it everytime you use it. If you are familiar with this feature, define one, if not, leave it blank to bypass this.
The system will generate the key pair, and display the key fingerprint and a randomart image. Open your file browser and go to C:Users\your_username.ssh. You should see two files. The private key is saved in the id_rsa file and the public key is labeled id_rsa.pub. This is your SSH key pair.
Add key to repository
Now, you have to add the public key to the Git repository settings as a deployment key.
For GitHub :
- Head to the homepage of your repository on GitHub.
- Go to the Settings page.
- Once there, select the tab on the left named
Deploy Keys
-
Select
Add deploy key
on the Deploy Keys page. -
Insert the name you want for your deploy key
- Copy/paste the public key (content of
*your-key-filename*.pub
) in the second text box. - Click on “Add key” (you don’t need to allow write access)
For GitLab :
- Head to the homepage of your repository on GitLab.
- Click on
Settings
(left bar) then go toRepository
- Click on
Expand
in theDeploy keys
section - Insert the name you want for your deploy key.
- Copy/paste the public key (content of
*your-key-filename*.pub
) in the second text box. - Click on
Add key
(you don’t need to Grant write permissions to this key)
Use Git in Platform
There are two methods to integrate the SSH key on the platform side:
- In the project information
- In the pipeline creation command in SDK
In project
If you want to have your whole project depending on a git repository, please fill the information in the Project
settings. Once configured, every pipeline created will by default be with the files on you repository.
To do this, go to your project's settings page and enter these 3 parameters:
-
Repository URL
: Enter the SSH URL of your repository. It should start withgit@
How to get my repository URL
On GitHub, from your repository, click on the “Code” button and choose SSH.
On GitLab, from your repository, click on the "Clone" button and copy SSH URL.
-
Deploy key
: Enter your Github / GitLab private key.Warning
Remember to keep the begin and end tags when you copy/paste the key. It should look like this :
-----BEGIN RSA PRIVATE KEY----- MIIJKQIBFSKCAgEAwH/zbeYm3M7elJHIjQTiO2+2QdTOh3ebvZotNQNATJ4UIqVN T9P2xN3Xd/27w8/jv9wmGqHzSVyEo53FfnyDm2zlFvqImRZm3znujA9bbp00itB5 ... Bo1gJMJxYJ4npi+0VULc33Ao6FzOfGxSACoTA/gG/q7LHO68c6Zgz+dI/ekDqG7C Gx52WhCP26GdneD/EhPgcUh41FzDbgO2BBIboNnrJLQzSQboK8JNrsislPr7 -----END RSA PRIVATE KEY-----
-
Default branch
: Enter the Git branch you want as default for this project. If this field is empty, we will use the default Git branch. This can be overwritten in the Environment.
Once this project information is saved, you can set up your pipelines so that the platform will use your Git repository by default:
sdk.create_pipeline(
function_path="src/my-source-code.py",
function_name="my-function-name",
pipeline_name="my-pipeline-name"
)
During pipeline creation
Wether you defined a git connection in the project or not, you can still specify a repository in the pipeline creation function in SDK. This parameter will overwrite the one in Project
if it is defined.
For example:
.
├── sdk-platform-script.py
├── my-git-repo/
│ ├── README.md
│ ├── requirements.txt
│ └── src/
│ └── my-source-code.py
└── keys/
├── my-key-filename
└── my-key-filename.pub
The script sdk_platform_script.py
is responsible for creating platform objects using the SDK. The file my_source_code.py
contains the Python code that will be executed in the pipeline, and the keys
directory contains the previously generated keys.
When creating a pipeline, you can configure your SDK script in this way to create a pipeline from the code in the Git repo:
with open('keys/my-key-filename', 'r') as file:
private_key_value = file.read().rstrip()
sdk.create_pipeline(
function_path="src/my-source-code.py",
function_name="my-function-name",
pipeline_name="my-pipeline-name",
container_config = {
"repository_url": "git@github.com:my-account/my-git-repo.git",
"repository_deploy_key": private_key_value,
"repository_branch": "main"
}
)
This will create a pipeline using the code from my-source-code.py
on the repository my-git-repo
.