Ephemeral Clusters in Konflux CI

Konflux pipelines can leverage the OpenShift CI facilities to either provision or claim an ephemeral cluster and use it to run tests.

Who is this documentation meant for?

The user facing documentation is available at konflux-ci.dev/docs/testing/integration/third-parties/openshift-ci.
This is intended as an high level overview for an OpenShift CI developer or for a user who is curious to understand how this feature works under the hood.

Lifecycle of an Ephemeral Cluster

The design document gives an high level overview on the mechanism.

The steps needed for provisioning an ephemeral cluster can be summarized as follow:

1. A Konflux pipeline runs the provision-ephemeral-cluster task.
2. The task creates a TestPlatformCluster object.
3. The Crossplane component acts upon the object and creates an EphemeralCluster object in app.ci.
4. The ephemeral cluster reconciler watches the EphemeralCluster object within the ephemeral-cluster namespace and spawns a ProwJob.
5. The ProwJob runs ci-operator that, in turns, use one of the available workflows to provision an ephemeral cluster.
6. ci-operator eventually succeeds in provisioning the cluster, then waits until it receives a “signal” from the reconciler to start the deprovisioning procedures.
7. The ephemeral cluster reconciler continuously polls the ci-op-xxxx namespace until it finds a secret that holds the kubeconfig for the ephemeral cluster. Once it shows up the cluster is ready.
8. The ephemeral cluster reconciler copies the kubeconfig into the .status.kubeconfig stanza of the EphemeralCluster object.
9. The crossplane reports the kubeconfig from the EphemeralCluster to a secret into the Konflux pipeline tenant namespace.
10. The provision-ephemeral-cluster task reads the kubeconfig from the secret and writes it into its results.
11. The Konflux pipeline uses the kubeconfig to run a test.
12. Upon test completion, the Konflux pipeline runs the deprovision-ephemeral-cluster task to start the deprovisioning procedures.
13. The deprovision-ephemeral-cluster task sets the .spec.tearDownCluster stanza to true.
14. The crossplane task sets the stanza to true but on the EphemeralCluster object in app.ci.
15. The ephemeral cluster reconciler signals ci-operator.
16. ci-operator starts the deprovisioning procedures. The ephemeral cluster is, eventually, destroyed.

Konflux Crossplane

XRDs and Composition for Test Platform are defined as follow:

• https://github.com/konflux-ci/crossplane-control-plane/config/xtestplatformcluster.

Deployments on various Konflux enviroments are defined as follow:

• https://github.com/redhat-appstudio/infra-deployments/components/crossplane-control-plane
• https://github.com/redhat-appstudio/infra-deployments/components/crossplane-config

IMPORTANT: When any change is introduced in Crossplane, make sure to bump the deployments above.

Crossplane doesn’t write an EphemeralCluster object directly in app.ci but it rather leverages the provider-kubernetes to achieve that.
The provider creates an EphemeralCluster proxy object within the Konflux cluster, keeping it in sync with the same EphemeralCluster object on app.ci within the ephemeral-cluster namespace.

+- Konflux -------------------------+
|                                   |
|        TestPlatformCluster        |
|               |                   |
|               v                   |
|        EphemeralCluster (proxy)   |
|               |                   |
+- app.ci ------|-------------------+
|               v                   |
|        EphemeralCluster           |
|                                   |
+-----------------------------------+

How Crossplane communicates to app.ci

Crossplane uses a kubeconfig for the app.ci cluster defined at https://github.com/redhat-appstudio/infra-deployments/components/crossplane-control-plane/staging/testplatform-provider-config.yaml#L16-L35.

The kubeconfig is synchronized from Vault. Follow this guide to rotate the kubeconfig token.

XRD and CRD

The TestPlatformCluster is an XRD (this is not entirely true, it’s a Claim but the concept it’s strictly tied to an XRD) resource (see docs.crossplane.io/latest/composition/composite-resource-definitions) whereas the EphemeralCluster is a regular CRD.
The EphemeralCluster CRD .spec definition is embedded into TestPlatformCluster, this means that when the CRD definition changes the XRD must be updated as well (see xtestplatformcluster/README.md).