Creating a Pipeline

Prow does not support pipelines. Hence, when submitting a new pull request, all tests are executed, including resource-intensive tests that can lead to increased infrastructure costs. To address this limitation, OpenShift CI introduces an abstraction layer utilizing GitHub’s branch protection rules. This enables the use of both manual (triggered by a user) and automatic (triggered by a bot) two-stage pipelines.

Marking CI job as second stage job

To enable two-stage pipelines, two actions are required. First, a specific branch protection setting must be activated within the Prow configuration of the repository (example) or organization.

1
2
3
4
5
6

branch-protection:
  orgs:
    openshift:
      repos:
        ci-tools:
          require_manually_triggered_jobs: true # enable pipelines on repo level

Secondly, a job that is intended to be promoted to the second stage must be appropriately marked. This involves explicitly setting the always_run: false property. Additionally, the job cannot be marked as optional: true and should not have skip_if_only_changed or run_if_changed configured (for run_if_changed, there is an alternative if one is using Automatic Pipelines).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

- always_run: false # job has to be explicitly marked
  as: e2e
  steps:
    test:
    - as: e2e
      commands:
      ...
      from: test-bin
      resources:
        requests:
          cpu: 100m
          memory: 200Mi

Upon submitting a PR with changes and merging, it may take up to six hours for the changes to be applied. This is because the Test Platform executes a branch protection periodic job every 6h which updates the GitHub configuration.

Manual Pipelines

Users can enable manual pipeline support and have complete control over when the second stage is executed. Manual pipelines are recommended for users who:

• Are comfortable manually executing the second stage of the pipeline.
• Want to wait for labels and then trigger the second stage of testing, such as trigger manually after receiving an LGTM and approve label.

Manual pipelines are currently a default solution, so if all tests are properly marked, they should be enabled. To trigger the second stage of the pipeline, user should create a comment on PR /test remaining-required. This will run all tests that have been promoted to the second stage.

Known issues

• Tide displays the message In merge pool if all required labels are applied, the first stage is complete, and no /test remaining-required comment is made. While this may suggest that the pull request is ready to be merged, this is not the case. The pull request will not be merged until all tests, including those in the second stage, have passed. TP is currently developing a fix for Tide to address this misleading message.

Automatic Pipelines

Users can opt-in to use automatic pipelines, which eliminates the need to track job execution. Our bot account will automatically comment /test remaining-required on pull requests when the following criteria are met:

• The PR is not a draft.
• All required jobs necessary for merging are green (optional jobs are ignored by the controller).

The controller responsible for triggering the second stage maintains a 24-hour memory of job contexts. Consequently, it will not comment on PRs that have been abandoned. If new commits are added or existing commits are modified and the first stage is triggered again, the controller resets the clock and reconsiders the PR.

Enabling automatic pipelines at the repository or organization level

To enable automatic pipelines, a PR must be submitted to modify the openshift/release config, adding the desired repository. Alternatively, it is also possible to add an entire organization by submitting a PR with the organization name, and without specifying any repositories.

run_if_changed alternative

There exists the ability to utilize functionality similar to run_if_changed when employing an automated pipelines workflow. For example, the provided test configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

- always_run: false
  as: e2e-oo
  run_if_changed: pkg/steps/(index|bundle).*.go
  steps:
    test:
    - as: e2e
      commands:
      ...
      from: test-bin
      resources:
        requests:
          cpu: 200m
          memory: 200Mi

May be repurposed to leverage a specialized pipeline_run_if_changed field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

- always_run: false
  as: e2e-oo
  pipeline_run_if_changed: pkg/steps/(index|bundle).*.go
  steps:
    test:
    - as: e2e
      commands:
      ...
      from: test-bin
      resources:
        requests:
          cpu: 200m
          memory: 200Mi

Previously mentioned rules must be consistently applied to the job. This includes explicitly setting the always_run: false property. Additionally, the job should not be marked as optional: true and must avoid using the skip_if_only_changed or run_if_changed settings. While establishing pipeline_run_if_changed, it does not exempt labeling a GitHub context as required. Consequently, some contexts may be labeled as overridden if they are not conditionally required. The pipeline controller will issue a comment listing the contexts to either trigger or override them. Example comment:

[GitHub Comment]
/test remaining-required

Scheduling tests matching the `pipeline_run_if_changed` parameter:
/test ci/prow/e2e

Overriding unmatched contexts:
/override ci/prow/e2e-oo

Overridden contexts will be considered as passed, and once all the remaining contexts have successfully passed, the pull request will be scheduled for merging.

Troubleshooting

TP has made every effort to cover all known corner cases and ensure that the bot comments on PRs. However, in rare instances, including situations when the GitHub webhook is unavailable due to factors such as partial GitHub outages, the bot may not post the comment. In such cases, users should comment on the PR themselves /test remaining-required to activate second stage. If the issue persists or is repeatable in certain situations, please submit a bug report using our bot in the #forum-ocp-testplaftorm Slack channel.