Onboarding a New Component for Testing and Merge Automation
This document overviews the workflow for onboarding new public component repositories to the Openshift CI. Private repositories are not supported at this time, but will be in the future
Granting Robots Privileges and Installing the GitHub App
In order to add labels, move PRs and issues into milestones, merge PRs, etc, the robots will need write access to your repository.
If your repository is in the OpenShift organization and it was created by Dev Productivity team, it probably has Team OpenShift Robots added by default as part of the repo creation process. To check if your repo has the robots added, please ask in slack in #forum-dp-platform or open a ticket. If your repo does not have robots, then open a new ticket in their JIRA project with the details.
If your component repository is not in this organization:
openshift-merge-robotinto your organization or add them as collaborators for the repository.
- Contact a CI administrator to accept the invitation. Otherwise, This will happen automatically after some time.
- By default, we enable branch-protection for
all prow-controlled repos. This requires
openshift-merge-robotto be an admin of the repo. We can disable it in prow’s config.yaml
- If a repository is enrolled in centralized branch management and no write permissions is granted to
openshift-merge-robot, ensure that the
tide/merge-blockerlabel exists on the repository. Otherwise, the periodic-openshift-release-merge-blockers job would fail. See how to create a label at Github’s documentation.
Additionally, all repositories need the Openshift CI GitHub App installed. We plan to eventually replace the bot accounts entirely with that app, but that work is not yet done.
Prow is the k8s-native upstream CI system, source code hosted in the kubernetes/test-infra repository. Prow interacts with GitHub to provide the automation UX that developers use on their pull requests, as well as orchestrating test workloads for those pull requests.
Bootstrapping Configuration for a new Repository
From the root of the openshift/release repository, run the following target and use the interactive tool to bootstrap a configuration for your repository:
This should fully configure your repository, so the changes that it produces are ready to be submitted in a pull
request. The resulting yaml file called
will be found in the
Plugins implement many portions of CI system functionality. These must be configured before they can be used. They can be enabled as-needed and configured for your repository independently of other repositories in the organization. After your repository is configured to deliver webhooks to Prow, those webhooks will be ignored until plugins are configured to consume and react to them. A live list of plugins and their descriptions is hosted on our website, please consult that list while reading the following section for more detail.
After initializing your repository with the
make new-repo target, you can create a new Plugin configuration for your repository with the following target:
This will place a new
_pluginconfig.yaml file in the
This file is used to configure the specific plugins for your repository.
Default plugin configuration is stored in
_plugins.yaml in the
openshift/release repository. Plugins are enabled for a repository or
organization under the
plugins key. Plugin-specific configuration is under keys like
owners. The set of a
repository’s enabled plugins is the union of plugins configured for the repository’s organization (found at the
plugins.yaml["plugins"]["$org"] key) and the repository itself (found in the
Most individual plugins can be configured to change their behavior; only some plugins allow for granular configuration
at a repository level, many only expose global configuration options for all repositories that Prow monitors. If you
think you need to configure an individual plugin, consult a CI administrator. While we work on a better solution,
documentation for these options lives in the
type Configuration struct
Repositories Under Existing Organizations
If you are onboarding a repository in an organization for which plugins are already configured, you will only need to
enable plugins that you do not already inherit from the organization by adding those plugins under a new
plugins.yaml["plugins"]["$org/$repo"] key. The one plugin you may want to configure at a repository-scoped level is:
|enables the |
Review the list of plugins enabled for your owning organization and the live plugin catalog when choosing which plugins you want on your repo.
If your repository does not have
OWNERS files, you will not be able to opt into the
/approve process or automatic pull
request review assignment.
OWNERS file format and interaction details can be found
upstream. You will also
need to add bugzilla component information to your
OWNERS file. Format can be found
Repositories Under New Organizations
If you are onboarding a component not in any organization we already have configured, consider copying the openshift
organization’s plugin configuration for your organization under a new
plugins.yaml["plugins"]["$org"] key, or/ setting
your repo’s configuration without adding any organization-wide configuration by adding the full list of plugins you
require under a new
Prow provides the following test trigger types:
|Push to a PR||A single PR merged into the branch it is targeting||Testing commits within a PR before they are merged|
|Push/merge to a branch||User specified set of branches||Integration tests after a PR is merged|
|User-specified set of branches||Scheduled test runs|
Configuration for your repository’s tests live in YAML files in the
openshift/release repository. Jobs are stored in
many files, sharded by branch and job type, found under:
The org and repo name redundancy is because of a requirement that the basename of your YAML file is unique under the
ci-operator/jobs tree. More detailed Prow job configuration documentation lives
Generating Prow jobs from ci-operator configuration files
The Test Platform team created a tool to generate Prow
job configuration files out of ci-operator configuration files. The generator has knowledge of the naming and directory structure conventions in
openshift/release repository. Provided you have put the
configuration file to
ci-operator/config/$org/$repo directory in it (as described by Containerized Tests
section), you can generate the needed Prow files by running this command from the root of the
This will create all necessary files under
ci-operator/jobs/$org/$repo, creating a good default set of Prow jobs.
Setting up team ownership of ci-operator and Prow config files
While the initial PR to
openshift/release will need to be reviewed and
approved by root approvers, once the component config is in
place, it should be owned by the component team. To achieve this, an
OWNERS file mirroring what exists upstream should
be placed into both
Enabling Automatic Merges
periodically searches for pull requests that fit merge criteria (for instance, presence of a
label and absence of the
do-not-merge/hold label) and merges them.
Tide furthermore requires not only that all required
tests in the Prow configuration succeed and all posted statuses on the GitHub pull request are green but also that the
tests tested the latest commit in the pull request on top of the latest commit in the branch that the pull request is
targeting before a pull request is considered for merging.
If you want to configure automatic merges for your component repository, you can either write a new merge criterion for
the repository or add your repository to one of the existing configurations. Tide’s configuration is documented
upstream. The existing Tide
configuration for our CI system lives in
config.yaml in the
openshift/release repository under the
config.yaml["tide"]["queries"] key. If
your repository does not have
OWNERS files, or if you have not chosen to opt into the
/approve process, it is
suggested that you add your repository to the merge criterion that requires only
OWNERS_ALIASES define the list. It is a concept of Prow workflow. Those files also define who
could get selected as reviewers of PRs in that repo. See Prow’s
on this topic.
Github’s users who are the repo’s collaborators. It is a concept of github. Contributors should follow this mojo to become a collaborator for repositories in openshift org. See Prow’s doc on this topic.
CI Operator Configuration
CI-Operator is a second-level orchestrator which translates Prow’s testing requests into OpenShift-native test
workloads. Think translating “run integration tests on my PR” into “trigger an OpenShift
Build to create a container
with test artifacts and a
Pod to run the integration test using that container image”.
While the Prow configuration describes when to run a test, the CI Operator configuration describes the test’s content.
ci-operator configuration reference document for
information on specific fields.
Adding a containerized test is as simple as adding an entry to the
tests array in the CI Operator configuration file
and a Prow job configuration that runs the test with
--target=$target. Consult the documentation
for more details on how to configure containerized tests and test them locally.
We recommend breaking up your tests into logical sections when adding tests here. More granular test reporting will allow for higher parallelism during test execution and more efficient re-testing if one suite fails.
Containerized tests are configured first-class in CI Operator configuration files in the
openshift/release repository, sharded by branch, at:
ci-operator/config/$org/$repo/$org-$repo-$branch.yaml. The org and repo name redundancy is due to
a requirement that all filenames be unique under
Tests that require a running Openshift cluster should use one of the provided step registry workflows, more details here.
Image Publishing and Mirroring
When container images are declared as release artifacts for a repository in the CI Operator configuration file under the
images list (like
[images] target is available for CI Operator execution that will simply build all
release images. In order for the container images built from your repository to be published, the
Prow job generator will configure a Prow postsubmit job that uses the CI Operator
Information on how to publish images to an external registry can be found in a separate document..
Product builds and becoming part of an OpenShift release image
Some images become part of the OpenShift release image because they are core to the platform. To be part of the release
image, there are additional requirements (beyond those described in the previous section). All product teams that will
ship an image to product must ensure their image is built in OSBS at least once and published back to the nightly test
jobs BEFORE you reference them from another component (via the image-references file), or before you set the image label
io.openshift.release.operator to get automatically included.
- Ensure you have successfully published your image to the CI integration stream
- Follow the ART instructions to have them build your image
- On the dist-git part of the process it is critical that you ensure your component/image names match as described in the bulleted criteria
- Ensure a single successful build is run (sync with ART to confirm)
- Open the PR to add your new image to another component (your operator, usually)
- Once the PR is merged, verify that the nightly builds continue to pass (usually 2-3 hours after your PR merges) and that you didn’t break the OCP CI test
Removing or renaming an image from an OpenShift release image
Sometimes we rename or remove images from a given release. To rename an image you must generally perform the following:
- Open but do not merge PRs to
openshift/releaseto change the name you create
- to components that reference that image (in
ocp-build-datato change the name that ART will publish as
- Once all PRs are reviewed and the new name is approved
- merge the
- If this breaks, revert the
openshift/releasePR and delete the
- merge the
- Merge the
ocp-build-dataPR and wait until a build succeeds and it is published to the
- Merge the component PRs
- Verify that all components build and a new CI release image is built and succeeds
- Locally test that removing the component succeeds. Run
oc adm release new --from-image-stream 4.2 -n ocp --exclude=OLD_NAME
- Get someone from ART or a release architect to delete the image stream tag for the old component name:
oc delete istag -n ocp 4.$VERSION:OLD_NAME
- This should trigger a new CI release image job - ensure it passes