Adding a New Secret to CI
Jobs execute as
Pods; those that need access to sensitive information will have access to it through mounted Kubernetes
Secrets. Secret data is managed self-service by the owners
of the data.
Add A New Secret
In order to add a new secret to our system, you will first need to create a secret collection. Secret collections are managed at selfservice.vault.ci.openshift.org. Just head there, log in, create a new one and ideally also add your teammates as members. Important: Secret collection names are globally unique in our system.
InfoUsers must have logged in to the DPTP Vault system at least once before they are listed as potential members.
The secrets themselves are managed in our Vault instance at vault.ci.openshift.org.
You need to use the OIDC auth to log in there. After logging in, click on
selfservice and you should see your secret collection.
To create a new secret, simply click
Create secret. Put your data into it and include the special
secretsync key value pairs listed below. These key value pairs will ensure that the new secret is propagated into the build clusters:
As an advanced feature, it is also possible to limit the clusters to which the secret should be synced. This is not needed in most cases and will result in failures if used for secrets that are used by jobs. This also works by using a special key in vault:
Use A Secret In A Job Step
The most common case is to use secrets in a step of a job. In this case, we
require the user to mirror secrets to
test-credentials namespace. The pod which runs the step can access the secrets
defined in the
credentials stanza of the step definition. See the documentation
The propagation of secret contents is scheduled immediately after they are added or modified and should be completed within 30m.
Protecting Secrets from Leaking
Unfortunately, secrets can often leak indirectly in various ways. Commonly, a setup step of a CI job uses a secret to configure a resource in the cluster, and then later another step collects that resource when capturing artifacts for the CI job. Logs and artifacts in OpenShift CI are publicly accessible, so when secrets are included in artifacts, they leak and must be rotated. To mitigate this risk, the Prow component that processes and stores all artifacts and logs contains a feature that automatically censors all secrets it can detect before uploading them to storage. Although this feature is relatively powerful (it detects and censors the content of artifacts that are tar or gzip archives, has built-in support for some compound secret formats like pull secrets and INI files, censors base64-encoded forms of the secret strings, etc.), it still needs to know what secret strings to search for.
The censoring process takes advantage of the fact that the secret value needs to be provided to the
the test code. In OpenShift CI, all secrets are provided to CI jobs via populating a namespace with
and therefore the CI job cannot use (and thus, leak) anything that is not present in one of the
in the namespace. The censoring code scans all artifacts for all values of all
Secret resources in the namespace where
Pod runs and removes all matches it finds.
Therefore, this censoring can only protect a secret from leaking if the secret is present in Vault in a “direct” form.
It may be convenient to store the secrets in Vault in a better consumable form, such as in a shell script that gets
sourced by the test code and populates multiple environmental variables at once. This approach is risky because it makes
that whole shell script the censored secret: it will only get stripped if you happen to
cat it in full by mistake.
However, if the content of one of the environment variables is the actual password that should not leak, the CI has no
chance of knowing that. If that password ends up in a resource in the cluster, and that resource will get collected
as an artifact, the password would leak.
In this example, the password is a direct value of a key stored in Vault. When synced to the CI clusters, the password
will be stored in the
password key of a
Secret resource and hence will be censored in all artifacts and logs
collected from the CI run.
In the example above, the password is an arbitrary substring of the value stored in Vault. When synced to the CI
secrets.sh key content will match the one in Vault. The actual secret string
s3cr3t is unknown to Prow
and therefore will not be censored from any artifacts or logs. Note that this practice is not limited to the shell
script form; the same applies for storing secrets inside JSON snippets and other formats.
With many secrets passed to CI jobs, it can become inconvenient to pass and consume them in individual keys. It can therefore be acceptable to pass them for consumption by CI jobs in the “convenient” compound form if, at the same time, the actual secret value is present directly in a separate key. Structuring your secrets this way makes them convenient to consume while still being protected by the censoring mechanism. Of course, this assumes you never forget to manage the secret in two places instead of one.
Use A Secret In Non-Step jobs
WarningThis section is used only for the jobs that had existed before Test Step Registry was introduced and have not yet been converted to multistage tests with steps. It is strongly suggested to use steps for any new jobs.
For non-step jobs, we have to use
ci as the targeting namespace in the secret mirroring configuration.
- For a job which is generated from
ci-operatorconfiguration and does not use steps, we can mount the secrets via
secretsstanza in the
- For a job which does not even use
ci-operatorat all, i.e. handcrafted jobs, the following example shows how to use secrets in a job definition. As stated there, creating handcrafted jobs is discouraged.