Overriding Failing CI Jobs

This document talks about how to bypass a failing CI job so that a PR can merge, and when it is appropriate to do so.

How to override

Putting /override $ci-job-name (e.g. /override ci/prow/e2e-aws-single-node-serial, note that the ci/prow/ prefix is necessary) in a PR comment will cause tide to ignore the test failure and process the PR for merging if all other requirements are met. In order to merge, a PR must have been tested (or overridden) on top of the current head of the branch it is merging into. This means that even if you override a failing job, if another PR has merged into the repository in the meantime, the PR will still be retested(and potentially fail) during the merge attempt. This means that /override is more likely to be helpful in a low traffic repository where nothing else is likely to have merged in the meantime.

Who CAN override

Permission to use /override is determined by the approvers in the top level OWNERS file in the repository.

Who SHOULD override

While it’s determined by the OWNERS file, some people are better suited to make the decision about an override than others for a given situation, so when asking for an override (or being asked), consider who the right person is to assess the risk + implications of the override. That might be another approver, or perhaps even someone who’s not an approver but has a better understanding of the failure you want to override and/or the changes the PR is introducing, so they can properly determine whether it is safe/reasonable to override.

Sticky overrides

A regular /override expires when the base branch moves (i.e. when another PR merges), because Tide retests the PR against the new base and the override no longer applies. In a high-traffic repository this means a regular override often doesn’t survive long enough for the PR to merge.

/override-sticky solves this by marking the override so that it persists across Tide retests on the current HEAD SHA, even when the base branch moves. The override is cleared automatically when a new commit is pushed to the PR (new SHA = clean slate).

Commands

• /override-sticky <context> — forces a context to green and persists across base branch movement. Like /override, the ci/prow/ prefix is necessary when overriding prow contexts, and contexts with spaces must be quoted.
• /override-cancel [context] — removes an override (regular or sticky) by setting the status back to failure. If a context is given, only that override is removed. If no context is given, all overrides on the PR are removed.

When to use sticky vs. regular override

Use a regular /override in low-traffic repositories where nothing else is likely to merge before your PR does, or when you expect the next merge into the base branch to fix the failing context (since the retest against the new base will pick up the fix). Use /override-sticky in high-traffic repositories where the base branch moves frequently, causing regular overrides to be invalidated before the PR can merge.

The same authorization rules and caution described below apply to both commands — /override-sticky is not a way to bypass review, it simply prevents you from having to re-issue the override repeatedly.

Things to do before applying an override

1. Confirm that someone/team owns and is aware of the failure you are trying to override. Don’t just override something broken and let the rest of the org keep running into it until someone else reports it. You don’t have to solve it, but at least make sure there’s an active/current bug or that the team in question acknowledges the issue.
2. Confirm the failure the PR is hitting is unrelated to the PR in question. This seems obvious, but don’t skip this step.
3. Confirm that by doing a /override, you are not compromising the test coverage the PR is subjected to. This one is harder. Consider the case where e2e-aws is repeatedly failing because of some AWS issues. Is someone aware of/owning the issue? Probably(amazon). Is the failure the PR is hitting unrelated to the PR in question? Definitely. But if you /override e2e-aws, is the PR still going to get adequate test coverage to confirm it has not broken something else? No. So it should not be overridden. That’s a strawman example, but the more concrete example is the situation where a particular test has been failing consistently in master. With the PR it continues to fail in the same way. But unless you confirm otherwise, it’s possible that the PR introduces a new failure mode for the test, one that you can’t even see until the original test issue is resolved. In this case test coverage of the PR is being compromised by the existing test failure, and you should not be overriding to merge the PR.