Architecture Data Flow
Resources
⚠️ Note!
You’ll need access to the appropriate groups to work with disruption data, please reach out to the TRT team for access.DPCR Job Aggregation Configs
(private repo)https://github.com/openshift/continuous-release-jobs/tree/master/config/clusters/dpcr/services/dpcr-ci-job-aggregation
Disruption Data Architecture
periodic jobs
, job aggregation
and BigQuery
to generate the disruption historical data.
It does not cover how the tests themselves are run against a cluster.High Level Diagram
How The Data Flows
openshift-tests
run disruption samplers
, these run GET
requests against a number of backends in the cluster every second and record the results to determine disruption. (see Testing Backends For Availability for more info)The
Disruption Uploader
is aCronJob
that is set to run every4 hours
. All theUploader
jobs are run in the DPCR cluster, the current configuration can be found in theopenshift/continuous-release-jobs
private repo underconfig/clusters/dpcr/services/dpcr-ci-job-aggregation
.When e2e tests are done the results are uploaded to
GCS
and the results can be viewed in the artifacts folder for a particular job run.Clicking the artifact link on the top right of a prow job and navigating to the
openshift-e2e-test
folder will show you the disruption results. (ex..../openshift-e2e-test/artifacts/junit/backend-disruption_[0-9]+-[0-9]+.json
).We only pull disruption data for job names specified in the
Jobs
table inBigQuery
. (see Job Primer for more information on this process)The disruption uploader will parse out the results from the e2e run backend-disruption json files and push them to the openshift-ci-data-analysis table in BigQuery.
Currently, backend disruption data is queried from BigQuery and downloaded in
json
format. The resultingjson
file is then committed to origin/pkg/synthetictests/allowedbackenddisruption/query_results.json for backend disruption or origin/pkg/synthetictests/allowedalerts/query_results.json for alert data (see how to query)The static
query_results.json
inopenshift/origin
are then used by the the matchers that thesamplers
invoke to find the best match for a given test (typically with “remains available using new/reused connections” or “should be nearly zero single second disruptions”) to check if we’re seeing noticeably worse disruption during the run.
How To Query The Data
Once you have access to BigQuery in the openshift-ci-data-analysis
project, you can run the below query to fetch the latest results.
Query
|
|
|
|
Downloading
Once the query is run, you can download the data locally.