Packaging |

CI/CD Part 1: Gitlab Pipelines and Package Registries

Tue, 02 May 2023 02:14:14 -0700

I recently developed a template workflow to help our team adopt a CI/CD-based development strategy. Many of our web applications and tools were based on simple repository structures. With growing datasets and ever-increasing use by outside teams, we found ourselves needing to add new features more frequently to many of these tools and believed that continuous integration and deployment could help us not just develop more quickly, but also more intelligently. Since we use Gitlab to store our code, we decided to use the Gitlab CI/CD tools.

Documentation on much of this process was scattered and/or sparse, so I decided to put what I learned and implemented into a more coherent set of notes. This post covers the anatomy of a .gitlab-ci.yml file, how jobs and stages fit together, how to make a job conditional, and how to authenticate against a package registry so a pipeline can build a Python package and push it there.

The second post covers the other half: writing a multi-stage Dockerfile to build an image from that package, and pushing the image to a container registry.

Setup

I did all of my testing using my personal Gitlab account. To separate things out, I created a new Project called “Package Registry”, as well as a test repository that was used for building a local Python project called “TemplateCI”.

The “Package Registry” Project serves as just that – an all-inclusive location for any software packages your CI/CD pipelines build. You can find the built packages by clicking ${Project Name} > Deploy > Package Registry. Every “Project” in Gitlab has the ability to store packages in its own registry, but I felt it cleaner to store everything in one repo. Similarly, the actual code that I’ll be packaging will be stored in the “TemplateCI” repo.

Authentication: .pypirc and .netrc

In order to build packages and push them to a remote package registry, we use the build and twine packages. build generates a package, and twine pushes this package to a registry (or “index”). twine requires access to authentication usernames, passwords, and a registry URL in order to do so. twine can access these tokens from a .pypirc file – the tokens are generated by the registry, and ensure that the submitting user has permissions to perform a certain action.

Other processes, such as pulling or pushing code from a remote repository, often require additional usernames and passwords. In order to alleviate the need to consistently provide these variables at request time, we can save them in a .netrc file.

These are straightforward to set up locally. But we also need to set these up to ensure a properly functional CI/CD workflow. I’ve put together a basic script, called setup_tokens.sh that does just that:

setup_tokens.sh

#!/bin/bash

# generate .pypirc file
echo "[distutils]
index-servers =
 personal

[personal]
repository = https://gitlab.com/api/v4/projects/$PACKAGE_REGISTRY_ID/packages/pypi
username = $CI_DEPLOY_USER
password = $CI_DEPLOY_PASSWORD" > ~/.pypirc

# generate .netrc file
echo "machine gitlab.com
login gitlab-ci-token
password $CI_JOB_TOKEN" > ~/.netrc

The .pypirc refers to your Project Registry via a previously generated authentication token and password, and allows your to build and upload Python packages to that registry.

The .netrc file enables you to pull private packages from that same registry. In the context of our work, we’ll want to build and push packages to the registry first so that they are available for pulling. For example, in the Pipfile for this template project, we have the following:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${CI_DEPLOY_USER}:${CI_DEPLOY_PASSWORD}@gitlab.com/api/v4/projects/${$PACKAGE_REGISTRY_ID}/packages/pypi/simple"
verify_ssl = true
name = "personal"

We see the same user authentication happening, along with the reference to the Package Registry ID variable. For local installation of your package, and in order to make sure that your Pipfile and Pipfile.lock are in sync, you’ll need to define the following local environment variables:

CI_DEPLOY_USER: generated user token
CI_DEPLOY_PASSWORD: generated token password
PACKAGE_REGISTRY_ID: the ID of the repository that you created that will store your packages

Basic jobs

The basis of a Gitlab CI/CD pipeline is the .gitlab-ci.yml file, which is composed of a set of explicitly-defined and temporally-ordered “stages”. A stage is composed of a set of “jobs”. Jobs are the workhorses of the CI/CD pipeline, and define explicit tasks that a CI/CD pipeline runs. By default, all jobs from one stage run in parallel, unless specified otherwise (using the needs keyword as an attribute of a job induces a temporal directed acyclic graph – jobs can be made “dependent” on the successful completion of other jobs within a stage). In this example, I’ve defined three stages:

run-unit-tests
build-package
build-image

Any job associated with the run-unit-tests stage will run to completion (or failure) PRIOR TO ANY job in the stages build-package and build-image starting. If all jobs in the run-unit-tests stage complete successfully, then the next stage (build-package) will begin. We define individual jobs, and give them a stage attribute. Stages define the rough ordering of jobs. Each job runs in the context of an “image” or environment. We can set a global image and define stages as

# global CI/CD image
image: python:3.9-slim

# stages of this example pipeline
stages:
 - run-unit-tests
 - build-package
 - build-image

 variables:
 LC_ALL: C.UTF-8
 LANG: C.UTF-8

or define the image as an attribute of a job. Gitlab CI/CD by default uses Docker images in which to run jobs. Setting the image is analogous to using the FROM command in a Dockerfile. We’ve also set some global variables, here the LC_ALL and LANG variables.

To “run” Gitlab pipelines for the purpose of CI/CD, we use “runners”, which are build instances installed on a server. Gitlab offers “shared” runners (use of these is free if you use www.gitlab.com, but you need to register a credit card to prevent abuse of Gitlab resources). You can also register your own device(s) to act as a Gitlab runner.

Below are examples of two jobs in the run-unit-tests stage. These two jobs are effectively the same code, apart from the unique unit tests that they run. However, we’ve made the job unit-tests-2 dependent on the output of the job unit-tests-1 (see the needs keyword of unit-tests-2). Both jobs use the global python:3.9-slim image. We can run some “setup” stuff (before_script), run an actual script (script), and run clean up (after_script, not shown) – these delineations (before, during, after) are for organizational purposes, and not due to any explicit functional differences in the delineations.

# setup_tokens.sh is the script from the previous section.


# example job #1
unit-tests-1:
 stage: run-unit-tests
 before_script:
 - chmod +x ./setup_tokens.sh; ./setup_tokens.sh
 - python3 -m pip install pipenv
 - apt-get update
 - apt-get install --yes --no-install-recommends gcc g++ libffi-dev
 - python3 -m pipenv install --deploy --dev
 # run first set of unit tests
 script:
 - python3 -m pipenv run pytest -k 'test_examples1.py'


# example job #2
unit-tests-2:
 stage: run-unit-tests
 before_script:
 - chmod +x ./setup_tokens.sh; ./setup_tokens.sh
 - python3 -m pip install pipenv
 - apt-get update
 - apt-get install --yes --no-install-recommends gcc g++ libffi-dev
 - python3 -m pipenv install --deploy --dev
 # run second set of unit tests
 script:
 - python3 -m pipenv run pytest -k 'test_examples2.py'
 # wait for job 1 to finish
 needs: [unit-tests-1]

Conditional pipeline jobs

The above jobs are relatively simple and will run every time you push a repository to Gitlab. However, sometimes, we might only want to run a job if certain conditions are met. For example, we might only want to build a package from the main branch, or only after a merge request is made. To this end, we can add “rules” to a job that restrict when it is actually run.

Below is a more complicated job example. The overarching goal of this job is to build a Docker image from a local Python project and push the image to the Gitlab Container Registry. There’s a lot going on here, so I’ll break it up into pieces, but here is the whole job:

# Conditional job
# Building a docker image and pushing this container to a container registry
# Link to main image: https://github.com/bentolor/docker-dind-awscli

# Conditions:
# --- merge request events
# --- target branch of merge request is "main"

# job name
build-image-glcr:

 # stage of pipeline
 stage: build-image
 # image that job is based on
 image: docker:20.10.16
 # sub-services of job
 services:
 - docker:20.10.16-dind
 # variables available to the job
 variables:
 DOCKER_TLS_CERTDIR: "/certs"
 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

 # some setup scripts -- here, just making sure docker is available
 before_script:
 - docker info

 # meat of the job -- authentication, building, run, push
 script:
 - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
 - docker build 
 --build-arg CI_DEPLOY_USER=$CI_DEPLOY_USER 
 --build-arg CI_DEPLOY_PASSWORD=$CI_DEPLOY_PASSWORD 
 --build-arg CI_JOB_TOKEN=$CI_JOB_TOKEN
 -t $IMAGE_TAG .
 - docker run $IMAGE_TAG
 - docker push $IMAGE_TAG

 # job conditions
 rules:
 - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"

This job is associated with a new stage called build-image that will run as the last stage of this example CI/CD pipeline. Without going into the specifics of the Dockerfile just yet, this stage builds and pushes a Docker image to a remote repository. We defined the job image as docker:20.10.16 and an additional “service” attribute as docker:20.10.16-dind where “dind” means “Docker-in-Docker”. The Docker-in-Docker feature allows an image to run Docker itself (pretty meta, huh). The idea here is to instantiate a job from a specific Docker container (“image”) which itself has Docker installed (“dind”), which will allow the docker:20.10.16 image to build another Docker container. We’ve also defined some variables:

DOCKER_TLS_CERTDIR: needed to allow the larger scale image to communicate with a service (honestly, I don’t quite understand this and documentation on CI/CD “services” is sparse)
IMAGE_TAG: refers to the container destination and the image “name” – this just makes our lives easier by turning into a variable what would otherwise be a really long string

#### Build Docker image and push to Gitlab Container Registry
build-image-glcr:
 stage: build-image
 image: docker:20.10.16
 services:
 - docker:20.10.16-dind
 variables:
 DOCKER_TLS_CERTDIR: "/certs"
 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

Next up, we have all the script stuff. You’ll probably recognize most of the docker ${command} commands. We first “authenticate” the current job with the Gitlab container registry (apparently there are a variety of ways to “authenticate” with Docker using one set of variables or another – the way below is the way I was able to get working, but there are other solutions). We then build the Docker image, run the image, and push the image to the container registry.

script:
 - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
 - docker build 
 --build-arg CI_DEPLOY_USER=$CI_DEPLOY_USER 
 --build-arg CI_DEPLOY_PASSWORD=$CI_DEPLOY_PASSWORD 
 --build-arg CI_JOB_TOKEN=$CI_JOB_TOKEN
 -t $IMAGE_TAG .
 - docker run $IMAGE_TAG
 - docker push $IMAGE_TAG

You’ll notice a bunch of variables that I didn’t explicitly define anywhere:

CI_REGISTRY_USER: username for project (I think we can also use CI_DEPLOY_USER, though it looks like there might be some things to figure out with branch / variable protections / non-protections)
CI_REGISTRY_PASSWORD: defaults to CI_JOB_TOKEN value (this value is ephemeral e.g. valid only for one job at a time I think? I think we can also use the CI_DEPLOY_PASSWORD for a longer-lived alternative, though it looks like there might be some things to figure out with branch / variable protections / non-protections)
CI_REGISTRY: defaults to https://gitlab.com/${group}/${project-name}/container_registry
CI_DEPLOY_USER: generated user token
CI_DEPLOY_PASSWORD: generated token password
CI_JOB_TOKEN: see documentation here

These are all “predefined” variables, meaning they already exist in the Gitlab CI/CD context as part of having a www.gitlab.com account, without you explicitly defining them. However, I did run into some issues using CI_DEPLOY_USER and CI_DEPLOY_PASSWORD. In addition to having predefined variables provided by Gitlab CI/CD, we can also manually predefine variables for a whole Gitlab Project or for a whole Group. Go to the page for your Project/Group > Settings > CI/CD > Variables > Expand. For example, I defined the CI_DEPLOY_USER and CI_DEPLOY_PASSWORD variables for my Group. These variables are the Group-level authentication tokens and are now made accessible to all Gitlab CI/CD jobs running under this Group. Additionally, although not shown here (because it’s used within the setup_tokens.sh script), we’ve also defined an environment variable called PACKAGE_REGISTRY_ID that tells the CI/CD pipeline where to build and push packages.

And finally, what we’ve been waiting for, conditional pipeline jobs:

rules:
 - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"

As part of this job, I’ve defined a rule that indicates that this job should only be run 1) upon a merge request event (i.e. if you click “Create Merge Request” in the console), and 2) if the name of the target branch of that merge request event is main. For example, if I create a new branch called dev off of main and create a merge request in the Gitlab console, this job will run. However, it’s important to note that, if you push a commit to the dev branch after creating the merge request, this job will still run, even if you haven’t created a new merge request e.g. CI_PIPELINE_SOURCE == “merge_request_event” will always default to TRUE after the first merge request event, as long as the source branch is still actively being developed. Additionally, the job itself for this will run the source branch code, not the target branch code. For Gitlab Premium users, there is an additional criterion called a “merged_result_event”, which would run the target branch (main) code after merging the source branch (dev) into the target branch.

Building and pushing a package

Building a package locally is straightforward, but doing so within a Gitlab CI/CD pipeline is a little more complicated. But, we can imagine adding this type of task to a CI/CD pipeline, and conditioning it on a merge request (or something of that kind).

I’ve called this job build-package and it belongs to a stage also called build-package. We first set up the .pypirc and .netrc files in image running the job, and then install the build and twine libraries in the before_script attribute. Then, using the script attribute, we build our package, and push it to our package registry (we’ve defined the registry in our .pypirc file – here, it’s referred to as the “personal” registry.)

image: python:3.9-slim
variables:
 PACKAGE_REGISTRY_NAME: "personal"

stages:
 - build-package

#### BUILDING PACKAGE AND PUSHING TO GITLAB PACKAGE REGISTRY
build-package:
 stage: build-package
 before_script:
 - chmod +x ./setup_tokens.sh; ./setup_tokens.sh
 - apt-get update
 - apt-get install --yes --no-install-recommends gcc g++ libffi-dev
 - python3 -m pip install build twine
 script:
 - python3 -m build
 - python3 -m twine upload --repository ${PACKAGE_REGISTRY_NAME} dist/* --verbose

Some of the (many) external links I used in this process:

Predefined CI/CD variables
Adding variables to the Gitlab CI/CD context
Setting up a ~/.netrc file with Gitlab credentials
Executing a bash scripting within a Gitlab CI/CD pipeline
Passing variables to Docker image at build time
Building Docker image and pushing to registry
Authenticating container registries
Setting up a ~/.pypirc files with Gitlab credentials

Lab Meeting: pip and the Python Packaging Index

Sun, 07 Jun 2020 10:20:09 -0700

What follows are the contents of part of a lab meeting presentation I gave recently. The topic of the meeting was “Python for Neuroimaging”, where I covered basic software development tools that brain imaging scientists might be interested in.

Creating Python Packages

In this lesson, I’ll show you how to build your own Python package that you can then install locally or upload to the Python Packaging Index (for those of you familiar with R, think CRAN, but for Python).

I’m going to be basing a lot of the material off of this documentation, but will also show a real example using some of my own personal code.

What are packages?

I’m sure most of you are familiar with packages and libraries already, either from Matlab, R, or Python. Packages are basically bundles of various snippets of code, i.e. methods, classes, scripts, tests etc. that are bundled together to perform some function. Generally (hopefully), there is coherence to what these snippets of code do – they should interact together in some way or relate to some overarching computational goal.

Within a package, you can have different groupings of code, where each grouping does some unique or discrete computing. These groupings are called submodules. A common submodule in many packages is an Input / Output (io) module that will read and write data that this package interacts with or produces. Another common submodule is often related to plotting the outputs of your code. And then almost always, there are submodules that perform the brunt of the algorithmic work. So inside modules, you’ll find snippets of code that relate to the goal or concept of the module.

Think of a package as a toolbox with a bunch of drawers, each with a label: wood-working, welding, gardening, flooring, etc. These drawers are submodules. You can tell by their names that they each cover certain topics. Each drawer contains a set of tools: wood-working might contain saw, nail, sandpaper, wood glue, while welding might contain solder, flux, oxygen, glove. These tools are the functions, classes, and scripts that relate to that submodule.

Overall, this toolbox performs some stuff related to construction, homebuilding, repair, and has discrete bundles of code useful for a variety of those tasks.

Directory structure for a Python package

Here we examine the skeleton of a package. All packages follow this basic structure.

pkg_name
|-- __init__.py
|-- LICENSE
|-- pkg_name/
| |-- submodule_a/
| | |-- __init__.py
| | |-- a_1.py
| |-- submodule_b/
| | |-- __init__.py
| | |-- b_1.py
| | |-- b_2.py
|-- README.md
|-- setup.py
|-- test/
| |-- __init__.py

__init__.py is a required file that allows your package to be imported. The only __init__.py file that needs to contain anything is the highest-level file. The others can be empty, but they must exist. Here are the contents of the highest-level __init__.py file:

__all__ = [
 'a_1', 'b_1', 'b_2'
]

from .submodule_a import (a_1)
from .submodule_b import (b_1, b_2)

LICENSE tells other users / individuals in what capacity they are allowed to use your code.

README.md describes how to use your code, and often contains examples. This is a markdown file, but can generally be any type of markup language.

test/ is a directory in which you would want to write unit tests for your code.

setup.py is what allows you to install your package. It’s a set of instructions that get supplied to setuptools package.

from setuptools import setup, find_packages

with open("README.md", "r") as fh:
 long_description = fh.read()

 setup(
 name='pkg_name',
 version='0.1.0',
 author='Kristian M. Eschenburg',
 author_email='keschenb@uw.edu',
 packages=find_packages(),
 scripts=[],
 url="https://github.com/kristianeschenburg/pkg_name",
 license='LICENSE.txt',
 description='An awesome package that does something',
 long_description=long_description,
 install_requires=[
 "numpy"
 "pytest",
 "matplotlib",
 ],
)

Compiling, installing, and uploading your package

1. Register on PyPi

Once we’ve done all this, we’re just about ready to create our Python package and upload it to pypi.org. But first, we need to create an account. For testing purposes, we’ll create a test account here, but the process is the same.

After you create your account, we need to create an API token, that will allow us to upload files to either Test PyPi or PyPi (depending on what we’re doing) – the following steps are the same, regardless.

Under your Test PyPi account, click your username in the top right, go to Account Settings:

Scroll down and click Add API Token:

Follow the instructions there, making sure to select “Entire Account” option under the Scope tab.

DO NOT CLOSE THIS WINDOW WHEN THIS IS COMPLETE

Next, type

cd $HOME
touch .pypirc

and using your favorite text editor, enter the following:

[testpypi]
 repository: https://test.pypi.org/legacy/
 username = __token__
 password = pypi-***

If we were creating a token for PyPi, we’d type:

[pypi]
 repository: https://pypi.org/
 username = __token__
 password = pypi-***

Close your current terminal, and open a new window to refresh your settings. Now, when we go to upload our package to PyPi, we’ll be able to type the commands without needed to supply a username and password directly.

2. Compile your package

First, we need to make sure that a few Python packages are installed. Namely, we need to install pip

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install -U pip

Then we can install the following packages:

pip install --upgrade pip setuptools wheel # for installing Python packages
pip install tqdm # progress bar package
pip install --user --upgrade twine # for publishing to PyPi

Then, we can compile our package:

python setup.py bdist_wheel

which creates the directories dist, build, and pkg_name.egg-info. The *.egg-info file is basically some zipped meta-data about your package, but we’re really only interested in the *.whl file in dist – “wheels” are a “distribution” format, newly designed to replace “eggs”. I won’t go into it here, but eggs were sort an ad hoc solution to packaging Python code – wheels were part of PEP427 i.e. is actually an “enhancement” to the Python language, and the formal way of packaging Python code.

3. Installing your code

We can install our code locally with:

pip install dist/pkg_name-0.0.0-py3-none-any.whl

If you want to install an “editable” version of your package, do this:

pip install -e .

This will allow you to change your *.py files and have these changes take effect immediately when importing your package, without needing to rebuild each time – but this method installs from the egg distribution, and generally produces larger build files, since the build needs to keep track of your actual source code.

4. Upload your code

We can upload our code to PyPi now using the following command:

python3 -m twine upload --repository testpypi dist/*

Now, if you click “Your Projects” under your account name on PyPi, you’ll see that you project has been uploaded.

** I should note that, any time you want to upgrade your code and upload it to PyPi again, you need to remove all files from the dist directory, increment the version number in the setup.py file – i.e. 0.0.0 –> 0.0.1 – rebuild your package with

bash python setup.py bdist_wheel

Example with personal package

I don’t generally upload my code to PyPi (probably scared bugs in the code, and people finding them, and then thinking I’m terrible at software development, and going down a long spiral of self-deprecation, but I digress…) but I do upload it all to GitHub. In either case, here is a walk-through of packaging some software called pysurface that I use for processing mesh-based data – I use it for adjacency matrices, performing Laplacian smoothing on surfaces, sampling points from mesh triangle simplices, plotting on surfaces… Just some stuff that I find myself doing a lot.

Here is the directory containing all my code:

You’ll see 5 different modules: graphs, operations, plotting, spectra, and utilities, and you’ll note that each module directory has a __init__.py file.

Here is my setup.py file:

from os import path
from setuptools import setup, find_packages
import sys

here = path.abspath(path.dirname(__file__))

with open(path.join(here, 'README.rst'), encoding='utf-8') as readme_file:
 readme = readme_file.read()

with open(path.join(here, 'requirements.txt')) as requirements_file:
 # Parse requirements.txt, ignoring any commented-out lines.
 requirements = [line for line in requirements_file.read().splitlines()
 if not line.startswith('#')]


setup(
 name='pysurface',
 version="0.0.4",
 description="Python package for quickly processing surface meshes.",
 long_description=readme,
 author="Kristian Eschenburg",
 author_email='keschenb@uw.edu',
 url='https://github.com/kristianeschenburg/pysurface',
 packages=find_packages(),
 entry_points={
 'console_scripts': [
 # 'some.module:some_function',
 ],
 },
 include_package_data=True,
 package_data={
 'pysurface': [
 # When adding files here, remember to update MANIFEST.in as well,
 # or else they will not be included in the distribution on PyPI!
 # 'path/to/data_file',
 ]
 },
 install_requires=requirements,
 license="BSD (3-clause)",
 classifiers=[
 'Development Status :: 2 - Pre-Alpha',
 'Natural Language :: English',
 'Programming Language :: Python :: 3',
 ],
)

You can see that I’ve run

python setup.py bdist_wheel

based off the dist, build, and pysurface.egg-info directories. dist contains a file called pysurface-0.0.4-py3-none.any.whl, which is the actual distribution that can be used for installation. I’ve uploaded the code to Test PyPi, and this is what we see:

We can then install the package and all of it’s dependencies from TestPypi via

pip install --index-url https://test.pypi.org/simple/ pysurface

Now I can do something like the following in a Python script:

import pysurface
# or
from pysurface import graphs, spectra
# or
from pysurface.spectra import eigenspectrum