Viewing job artifacts
Every Prow job executed by the CI system generates an artifacts directory containing information about that execution and its results. This document describes the contents of this directory and how they can be used to investigate the steps by the job.
The subjects explored here are documented, if somewhat superficially, in the
following files in the
kubernetes/test-infra repository, in reverse order of
usefulness to the casual Prow user:
The job examined will be a simple unit test from the
repository, which is executed as a pre-submit job. While each test has complete
control over the contents of its artifacts, that is only a subdirectory of the
final, root artifacts directory, and the overall structure of the latter does
not vary significantly between jobs.
When a job is triggered in a pull request,
of checks at the bottom of the page will contain a link to the
job details page.
The pages for post-submit and periodic jobs can be found via the main
page. These pages will be familiar to CI
users: they contain the most important information about the execution, such as
start time, result, duration, JUnit output, log, etc.
The top and bottom of the page have various links to other pages which contain auxiliary information. The sequence of links at the top depends on the type of job, but for most of them it is:
Job history: a list of all executions of the job across all pull requests, in reverse chronological order. This page can be used to determine the overall condition of the job: whether other recent executions passed or fail. For jobs added recently, it can also be useful for telling whether the job has ever succeeded.
Prow Job YAML: the
ProwJobobject created to execute the job. This page can be used to determine whether the expected version of the job was executed and to quickly inspect the parameters of the job, such as target cluster, timeouts, etc. This information is also available in
prowjob.jsonin the artifacts directory.
ProwJobis a runtime version of the content in
openshift/releaserepository, which in turn is usually automatically generated by
ci-operator-prowgenbased on the
ci-operator/configin the same repository.
Note that for
ci-operatorjobs, this and all references to “pod” in this section refer to the pod which executes
ci-operatoritself, not the one which executes the actual test (described below).
PR / PR History: a link to the original pull request and to a grid displaying all job executions for that pull request. The history page can be used to determine whether other executions of the job or other jobs for that PR succeeded, i.e. whether a failure is particular to a job or a general failure of the pull request.
Artifacts: a link to the directory which contains all output files created by the job. This page is described below.
The middle of the page contains the main build log. Particular error patterns
are highlighted and the full output is initially folded so that only relevant
parts are displayed. For
ci-operator jobs, this section contains the log
messages of level
info and above from its output, described below. The “raw build-log.txt” link contains the same
text, but served as a regular text file, and is also available as
build-log.txt in the artifacts directory.
In the case of the example job, the build log was as follows (parts of it will be used in examples in this and the next sections).
The bottom row of links/buttons can be used to display more information about the job:
Pod: a summary of the main fields of the
Podgenerated from the
ProwJobin the “Prow Job YAML” page, such as the start time and build node.
Volumes: the collection of volumes added to the test pod by Kubernetes, OpenShift, Prow, and
Eventobjects associated with the test pod. Useful in cases where the scheduling of the pod is delayed or impossible, volumes cannot be mounted, etc. This information is also available in the
podinfo.jsonfile in the artifacts directory.
Containers: not a single section, but one for each container in the test pod, with basic information such as the image used and the exit status. This list varies depending on the job, but common sections are:
clonerefs: the program used to clone repository code. If it is present here, it likely indicates the code is necessary for the execution of
ci-operatoritself, such as when part of its configuration is placed there.
initupload: this is an init container which signals that the job has successfully started and uploads some of the files which compose the result page and artifacts.
place-entrypoint: an init container used in the implementation of job execution, rarely interesting for job authors.
test: the actual test container, whose output is shown in the “build log” section. Somewhat surprisingly, the entry point program is not the one in the
ProwJobdefinition, but a program called
entrypoint, executed from a shared volume populated by the previous container.
This section decodes the
entrypointconfiguration and displays the test command which was executed, along with other information such as the timeout and grace periods.
sidecar: this container is executed in parallel with the main test and uploads information about the execution to the artifacts directory.
YAML: this is a dump of the full
Podobject as YAML. This information is also available in the
podinfo.jsonfile in the artifacts directory.
The “artifacts” link in the job result page leads to a page containing every file uploaded by that job. These files can provide more detailed information when the main job page is insufficient. The general structure of this directory is:
ci-operatorartifacts, described below.
build-log.txt: the build log as a text file, which also appears in the “build log” section of the main page.
clone-log.txt: if the Prow job itself clones the repository under test, this file will contain the main
clonerefsoutput. It corresponds to the
clonerefssection at the bottom of the main page.
clone-records.json: same as the previous item, but the raw output in JSON format.
finished.json: the file uploaded by the either the
sidecarcontainers when the test finishes.
podinfo.json: a dump of the
Eventobjects associated with the test, also found in the “Pod” and “Events” sections of the main page.
prowjob.json: a dump of the
ProwJobobject, also found in the “Prow Job YAML” link in the main page.
sidecar-logs.json: output of the
sidecarcontainer, which is responsible for uploading the files in this directory. It corresponds to the
sidecarsection at the bottom of the main page.
started.json: file uploaded by the
inituploadinit container when the test starts.
A test executed by
ci-operator will also have a directory called
among the files described in the previous section. This is the first directory
in the previous listing and has the following contents:
The files immediately under the directory are not commonly useful to test authors, they are mostly meant for test platform engineers:
ci-operator-step-graph.json: this is a dump of every Kubernetes/OpenShift object created by each step.
ci-operatordebug output. It is a superset of
build-log.txt, in JSON format. It can sometimes contain extra information useful for understanding the main log.
junit_operator.xml: the source of the JUnit section in the main page.
metadata.json: data file consumed by
ci-operator-step-graph.json therefore contains image import / build steps,
test steps, etc. This corresponds to the line which starts with “Running” in
unit are the steps in
the example job, which respectively import a base image, build the
and execute the
The directories at the top level contain detailed information for objects
created by the
build-logs: this directory contains one file for each OpenShift build created by the test containing the full log of the build.
ci-operatorwill include these logs in its output in case of failure, but they are available here directly as text files for all builds, even successful ones.
build-resources: this directory contains dumps of objects in the test namespace at the time the test finished, as listed below. Each file is named after the type of object dumped. Note that, since the test namespace can be and is commonly shared by many tests, these dumps may contain objects which the test did not itself create.
These files are useful to determine if the
Pod, etc. objects were created as expected by
ci-operatorbased on the image, test, etc. entries in the configuration file, as well as verifying that the expected versions of images were correctly imported and/or exported.
test: the name and contents of this directory varies based on the type of test. For container tests, the name will be simply
test. For template and multi-stage tests, it will be the name of the test. This directory is also created by a
sidecarprogram — this time attached to the test container, not
ci-operator— so it resembles the top-level artifacts directory.
sidecar-logs.jsonhave the same meaning.
artifactsdirectory contains all files written to the
$ARTIFACT_DIRdirectory by the test program. For multi-stage tests, a separate directory underneath this one will exist for each step.
Here are some problems which can be commonly answered by the files described in this page.
Problems cloning repository code
This can happen in two situations: when
ci-operator itself is executed and
when it creates the
src image. In both cases, this is done by the
program. For the first, the complete logs can be found in
clone-records.json at the root; for the latter, in
Test pod has not started
The immediate place to look is the “events” section at the bottom of the page,
which has all
Event objects associated with the test pod. The “Pod”,
“Volumes”, and “YAML” adjacent sections and the “Prow Job YAML” also provide
information about the generated pod.
Ultimately, for pull request jobs, the author has admin access to the test namespace and can inspect all objects in it. Obviously, at least one test has to successfully start in order for the test namespace name to be determined.
Test aborted due to timeout period
ci-operator pod, see the previous section. In addition, the
artifacts/build-resources/pods.json (or perhaps the other files in the same
directory) has information about the individual test pods created by
Build not generated as expected
A dump of all
Build objects created by
ci-operator are written to
artifacts/build-resources/builds.json, and all logs are written to
Incorrect credentials / parameters / dependencies
Credentials are mounted using
Secret mounts, while both parameters and
dependencies are passed to test containers as environment variables, so
artifacts/build-resources/pods.json contains exactly which values were
mounted/passed to each container.