Build and Release process

Automatic builds

RASCIL is built automatically via a GitLab CI pipeline, which can be triggered by:

• on schedule

• commit to any branch

• merge/commit to master

• a tag is pushed to the repository

RASCIL uses the SKA CI job templates for python and docs builds. The python-test is replaced by our own.

Currently, we run custom jobs for building, testing, and publishing docker images, as well as to release the RASCIL software (see below).

Additional test jobs:

• dask-test: runs on every commit to test Dask-specific code. This job deploys a new Dask cluster on the Data Processing Cluster in the dp-orca-p namespace.

• docker-test: runs a set of basic tests on all of the docker images built by RASCIL. (See: Dockerfiles for RASCIL)

• gpu-test: tests GPU-based code. Currently disabled.

• cluster-test: it deploys a cluster for working with RASCIL (Dask-based, including a Jupyter Notebook server) and runs a set up tests. This makes sure RASCIL can be deployed and worked with in kubernetes.

Building and storing RASCIL data:

• data job compresses the data uploaded with GitLFS and stores the files for an easy download at https://ska-telescope.gitlab.io/external/rascil-main/rascil_data.tgz . The build runs on every commit to master and the artefacts are stored for up to 6 months.

Scheduled jobs:

• the compile-requirements job runs, whose sole purpose is to regularly update the requirements files with the latest package versions. It also runs the .post stage. (See also: Process automation)

• the clean-up-dask job runs once every hour to clean up dask clusters that were created during the dask-test job.

The above process makes sure that new code is automatically tested at every point of the development process, and that the correct version of the python package and the docker images are published with the appropriate tag and at the appropriate time.

Releasing a new version

The release process:

• Overall based on: https://developer.skao.int/ and in particular https://developer.skao.int/en/latest/tutorial/release-management/automate-release-process.html

• Use semantic versioning: https://semver.org

• Follow the packaging process in: https://packaging.python.org/tutorials/packaging-projects/

The release of a new package happens in two stages:

• a release tag is pushed to the repository (manually by a maintainer)

• the CI pipeline’s relevant stages publish the new package.

Note: while commits are allowed directly to master by maintainers of the repository, this should not be used as an option, but rather update the code via Merge Requests. This is only allowed for releasing a new version of the package.

Steps:

• Ensure that the current master builds on GitLab: https://gitlab.com/ska-telescope/external/rascil-main/-/pipelines

• Decide whether a release is warranted and what semantic version number it should be: https://semver.org

• Check if the documentation has been updated. If not, create a new branch, update the documentation, create a merge request and merge that to master (after approval).

• Check out master and pull the latest version of it.

• Update CHANGELOG.md for the relevant changes in this release, putting newer description at the top.

• Commit the changes (do not push!)

• Bump the version using the Makefile:

  make release-[patch||minor||major]
  

  Note: bumpver needs to be installed. This step automatically commits the new version tag to the repository.

• Review the pipeline build for success

• Create a new virtualenv and try the install by using pip3 install rascil:

  virtualenv test_env
  . test_env/bin/activate
  pip3 install --index-url=https://artefact.skao.int/repository/pypi-all/simple rascil
  python3
  >>> import rascil