CI/CD Part 2: Building and pushing packages

This is the second post in a mini-series on designing Gitlab CI/CD pipelines. In order to build packages and push them to a remote package registry, we use the build and twine packages. build generates a package, and twine pushes this package to a registry (or “index”). twine requires access to authentication usernames, passwords, and a registry URL in order to do so. twine can access these tokens from a .pypirc file – the tokens are generated by the registry, and ensure that the submitting user has permissions to perform a certain action.

Other processes, such as pulling or pushing code from a remote repository, often require additional usernames and passwords. In order to alleviate the need to consistently provide these variables at request time, we can save them in a .netrc file.

These are straightforward to set up locally. But we also need to set these up to ensure a properly functional CI/CD workflow. I’ve put together a basic script, called setup_tokens.sh that does just that:

setup_tokens.sh

#!/bin/bash

# generate .pypirc file
echo "[distutils]
index-servers =
    personal

[personal]
repository = https://gitlab.com/api/v4/projects/$PACKAGE_REGISTRY_ID/packages/pypi
username = $CI_DEPLOY_USER
password = $CI_DEPLOY_PASSWORD" > ~/.pypirc

# generate .netrc file
echo "machine gitlab.com
login gitlab-ci-token
password $CI_JOB_TOKEN" > ~/.netrc

The .pypirc refers to your Project Registry via a previously generated authentication token and password, and allows your to build and upload Python packages to that registry.

The .netrc file enables you to pull private packages from that same registry. In the context of our work, we’ll want to build and push packages to the registry first so that they are available for pulling. For example, in the Pipfile for this template project, we have the following:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${CI_DEPLOY_USER}:${CI_DEPLOY_PASSWORD}@gitlab.com/api/v4/projects/${$PACKAGE_REGISTRY_ID}/packages/pypi/simple"
verify_ssl = true
name = "personal"

We see the same user authentication happening, along with the reference to the Package Registry ID variable. For local installation of your package, and in order to make sure that your Pipfile and Pipfile.lock are in sync, you’ll need to define the following local environment variables:

• CI_DEPLOY_USER: generated user token
• CI_DEPLOY_PASSWORD: generated token password
• PACKAGE_REGISTRY_ID: the ID of the repository that you created that will store your packages

Building a package locally is straightforward, but doing so within a Gitlab CI/CD pipeline is a little more complicated. But, we can imagine adding this type of task to a CI/CD pipeline, and conditioning it on a merge request (or something of that kind).

I’ve called this job build-package and it belongs to a stage also called build-package. We first set up the .pypirc and .netrc files in image running the job, and then install the build and twine libraries in the before_script attribute. Then, using the script attribute, we build our package, and push it to our package registry (we’ve defined the registry in our .pypirc file – here, it’s referred to as the “personal” registry.)

image: python:3.9-slim
variables:
  PACKAGE_REGISTRY_NAME: "personal"

stages:
  - build-package

#### BUILDING PACKAGE AND PUSHING TO GITLAB PACKAGE REGISTRY
build-package:
  stage: build-package
  before_script:
    - chmod +x ./setup_tokens.sh; ./setup_tokens.sh
    - apt-get update
    - apt-get install --yes --no-install-recommends gcc g++ libffi-dev
    - python3 -m pip install build twine
  script:
    - python3 -m build
    - python3 -m twine upload --repository ${PACKAGE_REGISTRY_NAME} dist/* --verbose