Architecture Data Flow

A high level look at how the disruption historical data is gathered and updated.

Resources

Disruption Data Architecture

High Level Diagram

DPCR ClusterJob RunUploader CronJobsDisruption UploaderGCPjob artifactsBucketsBig QueryFetch jUnit Resultsfor Jobse2e test clustere2e testsStoreResultsopenshift-ci-data-analysisJob RunUpdate BigQueryDisruption TableUserQuery JSON ResultsGitHubopenshift/originquery_results.jsonCommit ResultsDisruption TableJobs TableFetch List ofJobs To Gather

How The Data Flows

  1. The Disruption Uploader is a CronJob that is set to run every 4 hours. All the Uploader jobs are run in the DPCR cluster, the current configuration can be found in the openshift/continuous-release-jobs private repo under config/clusters/dpcr/services/dpcr-ci-job-aggregation.

  2. When e2e tests are done the results are uploaded to GCS and the results can be viewed in the artifacts folder for a particular job run.

    Clicking the artifact link on the top right of a prow job and navigating to the openshift-e2e-test folder will show you the disruption results. (ex. .../openshift-e2e-test/artifacts/junit/backend-disruption_[0-9]+-[0-9]+.json).

  3. We only pull disruption data for job names specified in the Jobs table in BigQuery. (see Job Primer for more information on this process)

  4. The disruption uploader will parse out the results from the e2e run backend-disruption json files and push them to the openshift-ci-data-analysis table in BigQuery.

  5. Currently, backend disruption data is queried from BigQuery and downloaded in json format. The resulting json file is then committed to origin/pkg/synthetictests/allowedbackenddisruption/query_results.json for backend disruption or origin/pkg/synthetictests/allowedalerts/query_results.json for alert data (see how to query)

  6. The static query_results.json in openshift/origin are then used by the the matchers that the samplers invoke to find the best match for a given test (typically with “remains available using new/reused connections” or “should be nearly zero single second disruptions”) to check if we’re seeing noticeably worse disruption during the run.

How To Query The Data

Once you have access to BigQuery in the openshift-ci-data-analysis project, you can run the below query to fetch the latest results.

Query

origin/pkg/synthetictests/allowedbackenddisruption/types.go
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

SELECT
    BackendName,
    Release,
    FromRelease,
    Platform,
    Architecture,
    Network,
    Topology,
    ANY_VALUE(P95) AS P95,
    ANY_VALUE(P99) AS P99,
FROM (
    SELECT
        Jobs.Release,
        Jobs.FromRelease,
        Jobs.Platform,
        Jobs.Architecture,
        Jobs.Network,
        Jobs.Topology,
        BackendName,
        PERCENTILE_CONT(BackendDisruption.DisruptionSeconds, 0.95) OVER(PARTITION BY BackendDisruption.BackendName, Jobs.Network, Jobs.Platform, Jobs.Release, Jobs.FromRelease, Jobs.Topology) AS P95,
        PERCENTILE_CONT(BackendDisruption.DisruptionSeconds, 0.99) OVER(PARTITION BY BackendDisruption.BackendName, Jobs.Network, Jobs.Platform, Jobs.Release, Jobs.FromRelease, Jobs.Topology) AS P99,
    FROM
        openshift-ci-data-analysis.ci_data.BackendDisruption as BackendDisruption
    INNER JOIN
        openshift-ci-data-analysis.ci_data.BackendDisruption_JobRuns as JobRuns on JobRuns.Name = BackendDisruption.JobRunName
    INNER JOIN
        openshift-ci-data-analysis.ci_data.Jobs as Jobs on Jobs.JobName = JobRuns.JobName
    WHERE
        JobRuns.StartTime > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 21 DAY)
)
GROUP BY
BackendName, Release, FromRelease, Platform, Architecture, Network, Topology
origin/pkg/synthetictests/allowedalerts/types.go
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

SELECT * FROM openshift-ci-data-analysis.ci_data.Alerts_Unified_LastWeek_P95
where
  alertName = "etcdMembersDown" or
  alertName = "etcdGRPCRequestsSlow" or
  alertName = "etcdHighNumberOfFailedGRPCRequests" or
  alertName = "etcdMemberCommunicationSlow" or
  alertName = "etcdNoLeader" or
  alertName = "etcdHighFsyncDurations" or
  alertName = "etcdHighCommitDurations" or
  alertName = "etcdInsufficientMembers" or
  alertName = "etcdHighNumberOfLeaderChanges" or
  alertName = "KubeAPIErrorBudgetBurn" or
  alertName = "KubeClientErrors" or
  alertName = "KubePersistentVolumeErrors" or
  alertName = "MCDDrainError" or
  alertName = "PrometheusOperatorWatchErrors" or
  alertName = "VSphereOpenshiftNodeHealthFail"
order by
 AlertName, Release, FromRelease, Topology, Platform, Network

Downloading

Once the query is run, you can download the data locally.

BigQuery Download

Last modified June 22, 2022: Starter Disruption Documentation (#263) (292d250)