Automate the Release Process Using Templates

This guide will walk you through the steps to automate the release process of your software package using GitLab CI/CD templates. The artefact names and versions are following the SKAO naming conventions. You can read more about them here. Note that this requires a SKAO login but generally the artefact names are in the format of ska-<package-name> and the versions are following Semantic Versioning like <major>.<minor>.<patch>.

Prerequisites

You have a GitLab account and a project repository where your software package is hosted.
You have set up the ska-cicd-makefile as a submodule in your project. If not, add it as a submodule to your project using the following command: git submodule add https://gitlab.com/ska-telescope/sdi/ska-cicd-makefile.git .make. If you are working with an existing repository make sure the submodule is checked out using the following command: git submodule update –init

Steps

Include the Release Makefile in Your Repo Makefile

Open your Makefile file and include the release support from the ska-cicd-makefile repository at the top of the file.
```
-include .make/release.mk
```

This will add supporting scripts for changelog generation using git-chglog and GitLab release pages.

Include the Release Template in Your GitLab CI/CD Configuration

Open your .gitlab-ci.yml file and include the release template from the templates-repository project towards the ends of the file (preferably at the end):
```
include:
  - project: 'ska-telescope/templates-repository'
    file : 'gitlab-ci/includes/release.gitlab-ci.yml'
```
This will add changelog generation and release note publishing mechanism (to #artefact-releases slack channel and as a Gitlab Release) support into the project.
Customise the Release Process (Optional)

Developers are strongly encouraged to use the default template to ensure that similar practices are followed in all SKA repositories, but if any departures from standard procedures are required the process can be customised. You can learn about the variables and how to override them by running make help release or make long-help release.
Commit and Push Your Changes

Commit your changes to the .gitlab-ci.yml and Makefile files and push them to your GitLab repository.
Trigger the Release Process

The release process is now automated and will be triggered whenever a new tag is pushed to the repository. You can create and push a new major, minor or patch version using the following below steps.

How to add the CHANGELOG

The changelog is a file that contains a curated, chronologically ordered list of notable changes for each version of a project. It is a way to communicate the significant changes that have been made to the software package. The changelog is automatically generated by the pipeline using the git-chglog tool for the GitLab Releases while allowing customisation. The changelog is generated based on the commit messages in the repository and the CHANGELOG.md file in the repository.

Add the CHANGELOG.md file to repository

Create a new file named CHANGELOG.md in the root of your project repository. This file will contain the changelog for your project.
Create a symlink to the `CHANGELOG.md` file in the `docs/src` folder of your project repository. This will allow the Read the Docs to include the changelog in the documentation.. Make sure to use the correct path for your documentation and CHANGELOG.md file.

If the docs/src folder doesn’t exist, please create it before executing this step.
```
cd docs/src
ln -s ../../CHANGELOG.md docs/CHANGELOG.rst
```
Update your index file in the `docs/src/` folder to include the changelog in the documentation. (if applicable)

Open the docs/index.rst file and add the following line to include the changelog in the documentation:
```
.. toctree::
:maxdepth: 1
:caption: Releases

CHANGELOG.rst
```
Add the required dependencies to build markdown files

recommonmark package is needed to build our markdown files. Note that the below code assumes you are using poetry with docs group to manage documentation dependencies.

poetry add recommonmark --group docs

Update the conf.py file to include recommonmark as an extension and make sure .md files are included as source file suffix.

extensions = [
   ...,
   "recommonmark"
]
source_suffix = ['.rst', '.md']

Commit and push the changes to the repository

Now, everytime a release happens the changelog will be updated with the new release notes that will be pulled from the CHANGELOG.md file. One example can be seen in this repository: CHANGELOG.md with associated RTD documentation here.

For further customisation of the changelog, please refer to the make help release or make long-help release targets in the Pipeline Machinery repository. This enables the combined use of automatic and manual methods for changelog generation.

How to Make a Release

This guide provides practical steps on how to make a patch release using the provided Makefile. For making major or minor version, the equivalent commands should be used.

Important

Note: The steps described here can be run on any branch (including the default (master/main) branch). The steps should work with any branching policy. Examples:

Using a new branch for Jira Release Management (REL) tickets. i.e. rel-xxx-release-v-x-y-z

Using an existing branch. i.e. key-1234-story-work

Using default branch. i.e. main/master

(Optional) Create a JIRA issue and the branch

1st: Create a new issue on the Release Management Jira Project with a summary of your release, and set it to “IN PROGRESS”. This is applicable if your release is integrated with Release Management and a Jira ticket is required.
Check the Current Version

Before making a patch release, you should check the current version of your project. You can do this by running the following command:
```
make show-version
```
This command will display the current version of your project.
Bump the Version

Choose which bump version you want to use:
- bump-major-release
- bump-minor-release
- bump-patch-release
Run for example make bump-patch-release, if for example .release was 1.2.1 it will be moved to 1.2.2. To increment the patch level of your project’s version, you can use the bump-patch-release target. Run the following command:
```
make bump-patch-release
```
This command will increment the patch level of the current version and update the .release file.
Set the Release

To set the version for different kind of artefacts, run make set-release target. This command will update the different versions of artefact types with an interactive prompt for you to follow.

If you have helm charts on your project it will automatically run make helm-set-release which will set all charts to - following the example - version 1.2.2, as well as update the version on the charts’ dependencies

If you have python packages on your project it will automatically run make python-set-release. This will set pyproject.toml to - following the example - version 1.2.2;

The release variable in your docs/conf.py will also be automatically updated according to the version in .release, confirm if this is the correct version for the documentation;

Make any other manual changes that are necessary to bump the version. For example:

Updating your python package’s __version__ attribute;

Updating python tests that check the version;

Manually updating a human-readable CHANGELOG file.

Create a Git Tag

Warning

Note: The creation of git tag marks this point as the release and once it’s pushed (in next step) a release is made. So this step should be performed with correct permissions to push tags. i.e. If the tag is supposed to be pushed to the main/master branch, only project maintainers have the correct permission by default. However, it is possible, instead, to push the tag onto your branch immediately before it is merged. In this case, it is very important that the tag is pushed to the branch only after the MR has been approved and no further commits will be made to it.

After bumping the patch version, you should create a git tag for the new version. By this point you’ll also require a JIRA ticket to link your release. The following target will ask you for the ticket as a prompt. This can be skipped by setting AUTO_RELEASE variable.
```
make create-git-tag
```
This command will create a git tag for the new version.
Push the Git Tag

Finally, you should push the new git tag to your remote repository. You can do this by running the following command:
```
make push-git-tag
```
This command will push the new git tag to your remote repository triggering the release process.

That’s it! You have successfully made a patch release for your project. Your release process is now automated. Whenever a new tag is pushed to the repository, the release process will be triggered, and the release notes will be generated and published automatically.

How to Make a Release Candidate

This guide complements the information described on how to make a release, in the previous section, and describes the steps required to create a release candidate.

Check the Current Version

Before creating a release candidate (RC), you should check the current version of your project. Run the following command:
```
make show-version
```
This command will display the current version of your project.
Decide on the version to be targetted for the RC and bump it

To create a release candidate you should, firstly, bump the version level desired (i.e. major, minor or patch).

You can do this by running the appropriate make target bump command (as described for step 3 of How To Make a Release):
```
make bump-<level>-release
```
This command will bump the version to the level indicated and display the version of the project you are updating to.
Create the RC version

Once the version has been bumped to the desired level, the release candidate version can be created. Run the following command:
```
make bump-rc
```
This command will add the ‘-rc.1’ to the version of the project you previously bumped or, if you run it multiple times with an RC version, it will keep incrementing the ‘-rc.N’. It should be noted that, for a python package, the version format added will be an exception and will follow the pattern ‘rcN’ (creating the version X.Y.ZrcN instead of X.Y.Z-rc.N).
The remaining release steps should be followed according to the How to Make a Release section (i.e. steps 5 and 6)
Promote a Release Candidate to a Release

If the Release Candidate has been successfully tested and you want to promote it to a release, run the following command:
```
make rc-to-release
```
This command will take your existing “X.Y.Z-rc.N” version and promote it to a release version with the format “X.Y.Z”.
```
# Current version: 2.0.1-rc.1

make rc-to-release

# New version: 2.0.1
```
Note that the remaining formal release steps described in the How to Make a Release section - steps 5 and 6 - should be followed after this.

How to Undo a Release Mistake

When using the make targets to bump a release/release candidate version there might be the case that one of the following mistakes is made:

The version was bumped to the wrong level on the local environment (e.g. bumped to a major version instead of a patch version)

The version was bumped to the wrong level and the files with the updated versions have been committed and pushed to the repository

The version was bumped to the wrong level and the release was created and published

For problem 1, having only updated the files locally and not committing any changes, one can simply run the git command to undo the changes:

# Current version: 2.0.1
make bump-major-release
# New version: 3.0.0
git restore .
# Restored version: 2.0.1
It is important to note that this approach should only be used if there are no other uncommitted changes in the repo. If there are, the git restore should be run specifically for the files where the version change needs to be undone.

For problem 2, even though the wrong versions have been pushed to the repository, as long as a tagging pipeline hasn’t been run, the release hasn’t been created yet. This allows the user to undo their changes locally and push the correct version to the repo and - only then - following the usual remaining release steps.

Problem 3 cannot be undone, the process to follow should be to mark the published release as incorrect and to create a new release with the correct versioning.

Release results

After the tagged pipeline finishes, the new release generated with the git-chglog will be appended to the tag in the gitlab project, an example of the release notes can be seen here. And the Jira ticket (preferable one created on the Release Management Jira Project) that is present on the commit message that triggered the tag pipeline will be updated with links to the gitlab release page.