Integration tests

Unlike unit tests, which operate independently of external services, integration tests rely on other software components. Typically, these components are deployed within a Kubernetes cluster and are only accessible from inside there.

The external components that this project depends on are declared in a Helm ‘umbrella’ chart, which you can find in charts/ska-oso-scripting-testing.

Implementation

Integration tests are specified in Behaviour-Driven Development (BDD) form and are implemented using the Pytest-BDD package, a plugin for Pytest that enables some compatibility with Cucumber/Gherkin tests formatted in pseudo-English.

You can find the .feature files that outline the test scenarios in tests/integration/features and the Python code that executes these scenarios in tests/integration/tests.

X-Ray Integration

X-Ray is a feature that allows test results to be presented in JIRA. To add more BDD tests, you need to…

  1. Create a new .feature file and/or test_something.py file.

  2. Add a new “Test” in the JIRA XTP project for each Cucumber Scenario. Note that JIRA “Tests” are analogous to Cucumber Scenarios, not features.

  3. Link the new test to XTP-57212, the “Test Plan” containing the OSO Scripting/TMC Simulator integration tests.

  4. (Optional) Link the new test to XTP-57209, a “Test Set” that groups these tests together.

  5. Use @XTP-#### in your feature file to tag each Scenario with the newly-created XTP test, also tag with @XTP-57212 for the Test Plan and @XTP-57209 for the Test Set.

Running integration tests

The standard procedure for setting up the Kubernetes environment and executing integration tests is to:

  1. Build the project image with make oci-build.

  2. Create the Kubernetes namespace and deploy test dependencies using make k8s-install-chart.

  3. Run the integration tests with make k8s-test.

  4. Tear down the integration test environment with make k8s-uninstall-chart.

This workflow is utilized in the k8s-test stage of the CI/CD pipelines.

Optional: faster testing and debugging of integration tests

IMPORTANT: the following steps requires your machine to be set up following the OSO developer environment instructions available on Confluence.

When BDD tests fail on the CI/CD pipeline, you often want to reproduce that failure locally and run the tests in quick succession after making fixes, without having to go through the steps above. For iterative developing and debugging of BDD tests in an environment similar to the CI/CD environment (i.e., a pod running inside a Kubernetes namespace), use make devpod.

First, run make oci-build to rebuild the project image (repeat this step whenever making changes to pyproject.toml) and launch devpod with make devpod. This command start a bash shell in a new, long-lived pod using the project image, similar to the transient testrunner pod in which the integration tests are run each time make k8s-test is executed.

If wanting direct view of the content of the ODA during the test execution, set the MOUNT_ODA environment variable to true when starting devpod (e.g., make MOUNT_ODA=true devpod). This will create the ./oda directory in the project root (if necessary) that will act as the backing store for the ODA, containing all OSO entities persisted by the ODA.

From this devpod bash shell, BDD tests can be run directly using:

poetry run pytest --forked /app/tests/integration

After completing development, exit the devpod, rebuild the project image to include your latest changes, then run the integration tests following the steps in Running integration tests. Once the tests pass, your changes are ready to be pushed to GitLab.