Architecture Data Flow

Resources

• Periodic Jobs

• BigQuery

• DPCR Job Aggregation Configs (private repo)

  https://github.com/openshift/continuous-release-jobs/tree/master/config/clusters/dpcr/services/dpcr-ci-job-aggregation
  

• Origin Synthetic Backend Tests

Disruption Data Architecture

High Level Diagram

How The Data Flows

1. The Disruption Uploader is a CronJob that is set to run every 4 hours. All the Uploader jobs are run in the DPCR cluster, the current configuration can be found in the openshift/continuous-release-jobs private repo under config/clusters/dpcr/services/dpcr-ci-job-aggregation.

2. When e2e tests are done the results are uploaded to GCS and the results can be viewed in the artifacts folder for a particular job run.

   Clicking the artifact link on the top right of a prow job and navigating to the openshift-e2e-test folder will show you the disruption results. (ex. .../openshift-e2e-test/artifacts/junit/backend-disruption_[0-9]+-[0-9]+.json).

3. We only pull disruption data for job names specified in the Jobs table in BigQuery. (see Job Primer for more information on this process)

4. The disruption uploader will parse out the results from the e2e run backend-disruption json files and push them to the openshift-ci-data-analysis table in BigQuery.

5. We currently run a periodic disruption data analyzer job in the infra cluster. It gathers the recent disruption data and commits the results back to openshift/origin. The PR it generates will also include a report that will help show the differences from previous to current disruptions in a table format. (example PR).

   Note, the read only BigQuery secret used by this job is saved in Vault using the processes described in this HowTo.

6. The static query_results.json in openshift/origin are then used by the the matchers that the samplers invoke to find the best match for a given test (typically with “remains available using new/reused connections” or “should be nearly zero single second disruptions”) to check if we’re seeing noticeably worse disruption during the run.

How To Query The Data Manually

The process for gathering and updating disruption data is fully automated, however, if you wish to explore the BigQuery data set, below are some of the queries you can run. If you also want to run the job-run-aggregator locally, the README.md for the project will provide guidance.

Once you have access to BigQuery in the openshift-ci-data-analysis project, you can run the below query to fetch the latest results.

Query

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

SELECT
    BackendName,
    Release,
    FromRelease,
    Platform,
    Architecture,
    Network,
    Topology,
    ANY_VALUE(P95) AS P95,
    ANY_VALUE(P99) AS P99,
FROM (
    SELECT
        Jobs.Release,
        Jobs.FromRelease,
        Jobs.Platform,
        Jobs.Architecture,
        Jobs.Network,
        Jobs.Topology,
        BackendName,
        PERCENTILE_CONT(BackendDisruption.DisruptionSeconds, 0.95) OVER(PARTITION BY BackendDisruption.BackendName, Jobs.Network, Jobs.Platform, Jobs.Release, Jobs.FromRelease, Jobs.Topology) AS P95,
        PERCENTILE_CONT(BackendDisruption.DisruptionSeconds, 0.99) OVER(PARTITION BY BackendDisruption.BackendName, Jobs.Network, Jobs.Platform, Jobs.Release, Jobs.FromRelease, Jobs.Topology) AS P99,
    FROM
        openshift-ci-data-analysis.ci_data.BackendDisruption as BackendDisruption
    INNER JOIN
        openshift-ci-data-analysis.ci_data.BackendDisruption_JobRuns as JobRuns on JobRuns.Name = BackendDisruption.JobRunName
    INNER JOIN
        openshift-ci-data-analysis.ci_data.Jobs as Jobs on Jobs.JobName = JobRuns.JobName
    WHERE
        JobRuns.StartTime > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 21 DAY)
)
GROUP BY
BackendName, Release, FromRelease, Platform, Architecture, Network, Topology

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

SELECT * FROM openshift-ci-data-analysis.ci_data.Alerts_Unified_LastWeek_P95
where
  alertName = "etcdMembersDown" or
  alertName = "etcdGRPCRequestsSlow" or
  alertName = "etcdHighNumberOfFailedGRPCRequests" or
  alertName = "etcdMemberCommunicationSlow" or
  alertName = "etcdNoLeader" or
  alertName = "etcdHighFsyncDurations" or
  alertName = "etcdHighCommitDurations" or
  alertName = "etcdInsufficientMembers" or
  alertName = "etcdHighNumberOfLeaderChanges" or
  alertName = "KubeAPIErrorBudgetBurn" or
  alertName = "KubeClientErrors" or
  alertName = "KubePersistentVolumeErrors" or
  alertName = "MCDDrainError" or
  alertName = "PrometheusOperatorWatchErrors" or
  alertName = "VSphereOpenshiftNodeHealthFail"
order by
 AlertName, Release, FromRelease, Topology, Platform, Network

Downloading

Once the query is run, you can download the data locally.