Run the Test Suite

The Test Suite can be run using the Certsuite tool directly or through a container.

To run the Test Suite direct use:

certsuite run -l <label-filter> -c <certsuite-config> -k <kubeconfig> -o <output-dir> [<flags>]

If the kubeconfig is not provided the value of the KUBECONFIG environment variable will be taken by default.

The CLI output will show the following information:

• Details of the Certsuite and claim file versions, the test case filter used and the location of the output files.
• The results for each test case grouped into test suites (the most recent log line is shown live as each test executes).
• Table with the number of test cases that have passed/failed or been skipped per test suite.
• The log lines produced by each test case that has failed.

Once the test run has completed, the test results can be visualized by opening the results.html website in a web browser and loading the claim.json file.

For more information on how to analyze the results see Test Output.

Building the Certsuite tool executable

The Certsuite binary can be built as follows:

make build-certsuite-tool

Test labels

The test cases cases have several labels to allow for different types of groupings when selecting which to run. These are the following:

• The name of the test case
• The name of the test suite
• The category of the test case (common, telco, faredge, extended)

These labels can be combined with some operators to create label filters that match any condition. For example:

• The label filter “observability,access-control” will match the test suites observability and access-control.
• The label filter “operator && !operator-crd-versioning” will match the operator test suite without the operator_crd_versioning test case.
• To select all the test cases the all label filter can be used.

To view which test cases match a label or label filter:

certsuite info -t <label-filter> --list

See the test catalog for all test labels.

Disable intrusive tests

To skip intrusive tests which may disrupt cluster operations, issue the following:

certsuite run --intrusive=false

The intrusive test cases are:

• lifecycle-deployment-scaling
• lifecycle-statefulset-scaling
• lifecycle-crd-scaling
• lifecycle-pod-recreation

Likewise, to enable intrusive tests, set the following:

certsuite run --intrusive=true

Intrusive tests are enabled by default.

Flag reference

The certsuite run command organizes its flags into groups. To see the complete list use the -h, --help flag.

Common flags

• -c, --config-file: Path to the certsuite_config.yml file.

• -l, --label-filter: Label expression to filter test cases. Can be a test suite or list or test suites, such as "observability,access-control" or a more complex expression with logical operators such as "access-control && !access-control-sys-admin-capability".

Note

If -l is not specified, the Test Suite will run in ‘diagnostic’ mode. In this mode, no test case will run: it will only get information from the cluster (PUTs, CRDs, nodes info, etc…) to save it in the claim file. This can be used to make sure the configuration was properly set and the autodiscovery found the right pods/crds…

• -o, --output-dir: Path of the local directory where test results (claim.json), the execution logs (certsuite.log), and the results artifacts file (results.tar.gz) will be available from after the container exits.

• -k, --kubeconfig: Path to the Kubeconfig file of the target cluster.

• --timeout: Time allowed for the test suite execution to complete (e.g. --timeout 30m or --timeout 1h30m). Defaults to 24h.

• --log-level: Sets the log level. Defaults to debug.

• --intrusive: Run intrusive tests that may disrupt the test environment. Enabled by default. Set to --intrusive=false to skip intrusive tests.

Test behavior flags

• --allow-non-running: Include non-Running pods during the autodiscovery phase. Disabled by default; enable this if your workloads include pods in CrashLoopBackOff or other non-running states that still need testing.

• --server-mode: Run the certsuite in web server mode.

Output & artifact flags

• --omit-artifacts-zip-file: Prevents the creation of a zip file with the result artifacts.

• --include-web-files: Save the HTML results viewer files in the configured output folder alongside the claim.json and log files.

• --create-xml-junit-file: Generate a JUnit XML file with the test results, useful for CI/CD integration with systems that consume JUnit reports.

• --sanitize-claim: Sanitize the claim.json file by removing sensitive data before sending it to the collector. Only relevant when --enable-data-collection is enabled.

Probe daemonset flags

• --certsuite-probe-image: Override the default certsuite probe daemonset image. Defaults to quay.io/redhat-best-practices-for-k8s/certsuite-probe:v0.0.42 (debugTag in version.json).

• --daemonset-cpu-req, --daemonset-cpu-lim: Set the CPU request and limit for the probe daemonset container. Both default to 100m.

• --daemonset-mem-req, --daemonset-mem-lim: Set the memory request and limit for the probe daemonset container. Both default to 100M.

• --cleanup-probe: Controls whether the probe daemonset and its namespace are deleted at the end of the test run. By default (true), the probe daemonset is cleaned up after tests complete. Set to --cleanup-probe=false to keep the probe daemonset running on the cluster for debugging or repeated test runs.

certsuite run --cleanup-probe=false

When running in container mode, add the flag to the certsuite command:

docker run --rm --network host \
  -v <path-to-local-dir>/config:/usr/certsuite/config:Z \
  -v <path-to-local-dir>/results:/usr/certsuite/results:Z \
  quay.io/redhat-best-practices-for-k8s/certsuite:latest \
  certsuite run \
  --kubeconfig=/usr/certsuite/config/kubeconfig \
  --config-file=/usr/certsuite/config/certsuite_config.yml \
  --output-dir=/usr/certsuite/results \
  --label-filter=all \
  --cleanup-probe=false

• --require-probe: Abort the test run if the probe daemonset fails to deploy. Disabled by default.

Preflight flags

• --preflight-dockerconfig: Path to the Dockerconfig file to be used by the Preflight test suite.

• --allow-preflight-insecure: Allow insecure connections in the Preflight test suite.

• --offline-db: Path to an offline DB to check the certification status of container images, operators and helm charts. Defaults to the DB included in the test container image.

Note

See the OCT tool for more information on how to create this DB.

Red Hat Connect flags

• --enable-data-collection: Allow sending test results to an external data collector.

• --connect-api-key: API key for uploading results to the Red Hat Connect portal.

• --connect-project-id: Project ID for uploading results to the Red Hat Connect portal.

• --connect-api-base-url: Base URL for the Red Hat Connect API.

• --connect-api-proxy-url, --connect-api-proxy-port: Proxy URL and port for the Red Hat Connect API, for environments that require HTTP proxies.

Using the container image

The only prerequisite for running the Test Suite in container mode is having Docker or Podman installed.

Pull the test image

The test image is available at this repository and can be pulled using:

docker pull quay.io/redhat-best-practices-for-k8s/certsuite:<image-tag>

The image tag can be latest to select the latest release, unstable to fetch the image built with the latest commit in the repository, or any existing version tag from the releases page.

Launch the Test Suite

The Test Suite requires 3 files that must be provided to the test container:

• The Kubeconfig for the target cluster.
• The Dockerconfig of the local Docker installation (only for the Preflight test suite).
• The certsuite_config.yml.

To reduce the number of shared volumes with the test container in the example below those files are copied into a folder called “config”. Also, another folder to contain the output files called “results” has been created. The files saved in the output directory after the test run are:

• A claim.json file with the test results.
• A certsuite.log file with the execution logs.
• A .tar.gz file with the above two files and an additional results.html file to visualize the results in a website.

docker run --rm --network host \
  -v <path-to-local-dir>/config:/usr/certsuite/config:Z \
  -v <path-to-local-dir>/results:/usr/certsuite/results:Z \
  quay.io/redhat-best-practices-for-k8s/certsuite:latest \
  certsuite run \
  --kubeconfig=/usr/certsuite/config/kubeconfig \
  --preflight-dockerconfig=/usr/certsuite/config/dockerconfig \
  --config-file=/usr/certsuite/config/certsuite_config.yml \
  --output-dir=/usr/certsuite/results \
  --label-filter=all