Payload Testing

TRT needs the capability to run a selected subset of release qualification jobs on selected pull requests in all repositories that contribute to OCP, before they are merged. The PullRequestPayloadQualificationRun CRD and the /payload command is provided for this purpose.

Usage

Payload Command

Any collaborator of the GitHub OpenShift organization can issue the command on a pull request to a branch of a repository of the organization that contributes to OpenShift official images:

/payload <ocp_version> <ci|nightly> <informing|blocking>

For example, if /payload 4.10 nightly informing is issued on a PR, the robot will reply the list of the triggered jobs:

The jobs triggered by the command are determined by OpenShift Release Controllers. The linked page from payload-tests portal at the bottom of the comment shows the status of the payload testing and the details of those jobs.

The images from multiple PRs can also be included in the payload tested by using the /payload-with-prs command, such as

/payload-with-prs <ocp_version> <ci|nightly> <informing|blocking> <org/repo#number> [<org/repo#number …]

A particular job or set of jobs can be triggered by /payload-job, such as

/payload-job <periodic_ci_openshift_release_some_job> <periodic_ci_openshift_release_another_job>

The payload tested can also include the images from multiple PRs by using /payload-job-with-prs, such as

/payload-job-with-prs <periodic_ci_openshift_release_some_job> <org/repo#number> [<org/repo#number …]

A job can be executed more than once by a single /payload-aggregate command, e.g,

/payload-aggregate <periodic_ci_openshift_release_some_job> <aggregated_count>

It is also possible to use the aggregation logic with additional PRs included in the payload with the /payload-aggregate-with-prs command, e.g,

/payload-aggregate-with-prs <periodic_ci_openshift_release_some_job> <aggregated_count> <org/repo#number> [<org/repo#number …]

Abort all Payload Jobs

It is possible to quickly abort all running payload jobs for a specific PR. Simply comment /payload-abort on the PR to do so.

Manually Submitting a PRPQR

It is also possible to manually create a PullRequestPayloadQualificationRun instance without using the payload command. This allows for additional options to be supplied that are currently not possible via the command. The following is an example of a full PRPQR CR that can be applied to the app.ci cluster to trigger a payload job:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

apiVersion: ci.openshift.io/v1
kind: PullRequestPayloadQualificationRun
metadata:
  name: manually-submitted-prpqr
  namespace: ci
spec:
  initial: registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2024-02-01-213342
  payload:
    base: registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2024-02-06-120750
    tags:
      - name: "machine-os-content"
        tag: "4.13-2024-02-04-192545"
  jobs:
    releaseControllerConfig:
      ocp: ''
      release: ''
      specifier: ''
    releaseJobSpec:
    - ciOperatorConfig:
        branch: master
        org: openshift
        repo: release
        variant: ci-4.15
      test: e2e-aws-sdn-upgrade
  pullRequests:
  - baseRef: master
    baseSHA: 270de19d62fc7275f22de22a7eca270bd77dd05d
    org: openshift
    pr:
      author: developer
      number: 1575
      sha: c9817bfb09b48bc84ef20a1cf5a01cac36c2687d
      title: 'A Pull Request'
    repo: kubernetes

Supplying Multiple PRs from Component Repositories

The ci-operator can assemble a release payload by building and using images from multiple PRs in OCP component repos. In order to do this, refs for each PR must be provided in the PRPQR spec:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

...
spec:
  jobs:
    ...
  pullRequests:
    - baseRef: master
      baseSHA: 270de19d62fc7275f22de22a7eca270bd77dd05d
      org: openshift
      pr:
        author: developer
        number: 1575
        sha: c9817bfb09b48bc84ef20a1cf5a01cac36c2687d
        title: 'A Pull Request'
      repo: kubernetes
    - baseRef: master
      baseSHA: dcf812295b06c9463cb7c8d8126a337334049234
      org: openshift
      pr:
        author: developer
        number: 7640
        sha: 7d5da4ce183886b6de33172e7af2a01ca2e46708
        title: 'Another Pull Request'
      repo: installer
...

Overriding the Default Payload PullSpecs

It is possible to override the pull-spec used for both the initial and latest release payloads by manually submitting a PRPQR. This is done by supplying the spec.initial and spec.payload.base fields respectively:

1
2
3
4
5
6
7
8

...
spec:
  # useful for supplying a different pull-spec for the initial payload during 'upgrade' jobs
  initial: registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2024-02-01-213342 
  payload:
    # images built using PR code will be layered on top of the base
    base: registry.ci.openshift.org/ocp/release:4.13.0-0.nightly-2024-02-06-120750
...

Overriding Specific Image Tags in the Payload

It is also possible to override a specific image tag in the payload to any tag contained in an ocp ImageStream. Doing so requires manually submitting a PRPQR. This can be done by providing the tag overrides in the spec.payload.tags list:

1
2
3
4
5
6
7

...
spec:
  payload:
    tags:
      - name: "machine-os-content" # name of the tag to be overridden
        tag: "4.13-2024-02-04-192545" # ImageStream name in the 'ocp' namespace to override to
...

Sourcing Components from Specific Base

A PRPQR can be submitted without referencing an open PR. This is useful to pin one or more component repos to a specific baseRef and baseSHA in order to test the state of things at that point. Doing so requires manually applying the PRPQR CR:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

...
spec:
  jobs:
    ...
  pullRequests:
    - baseRef: master 
      baseSHA: 270de19d62fc7275f22de22a7eca270bd77dd05d
      org: openshift
      repo: kubernetes
...