Pod Scaler

Pod-scaler is a mutating admission webhook on build farm clusters (build01–build13). It reads historical CPU and memory usage from Prometheus (cached in GCS), recommends resources for recurring workloads, and may change requests and limits when a pod is created.

• Source: openshift/ci-tools/cmd/pod-scaler
• Build-farm admission manifest: openshift/release/clusters/build-clusters/build-shared/pod-scaler/
• Producer and UI on app.ci: clusters/app.ci/pod-scaler/

What changed recently

These landed on main in July 2026 and are worth knowing before you debug a job:

If admission pods crash-loop with flag provided but not defined: -authoritative-cpu-request-dry-run, the cluster still has the old release args. Sync to release main (or ArgoCD hard-sync build-shared-buildNN-pod-scaler) before chasing job failures.

For repo and job owners

You set CPU and memory in .ci-operator.yaml (test steps, image builds, registry steps). On build clusters, pod-scaler may raise those values at pod creation when history shows higher usage. Decreases are in dry-run on build farms right now (see above), so you should not see limits drop from authoritative mode until operators flip the apply flags to true.

What you will see

Pod-scaler does not resize running pods. Retrigger the job or start a new build to pick up changes.

When to edit your config

1. Set explicit resources for memory-sensitive or bursty work (image builds, heavy compiles). Defaults plus history are not a substitute for knowing your job.
2. Use disable_pod_scaler: true when one test target must keep fixed resources regardless of history.
3. Skip flags are not per-repo. You cannot set them in your repository; ask test-platform if you think a workload class needs cluster-wide decrease exemptions.

Per-job opt-out (disable_pod_scaler)

Set on the test as: name in .ci-operator.yaml:

Image build (*-images jobs):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

tests:
- as: images
  disable_pod_scaler: true
  steps:
    test:
    - as: src
      commands: # unchanged
      from: src
      resources:
        requests:
          cpu: 100m
          memory: 4Gi
        limits:
          memory: 8Gi

Multi-step test:

1
2
3
4
5
6
7

tests:
- as: e2e-aws
  disable_pod_scaler: true
  steps:
    workflow:
    - as: openshift-e2e-test
      ...

ci-operator sets ci-workload-autoscaler.openshift.io/scale: "false" on generated pods and OpenShift Build objects for that target.

Troubleshooting your jobs

compile: signal: killed or OOM after a pod-scaler image rollout

Image builds often burst above average history. Build farms already use peak usage basis for decrease math (when decreases are enabled), so limits track spikes better than p80 alone.

If a job still OOMs:

1. Raise resources.limits.memory (and requests if needed) in .ci-operator.yaml.
2. Set disable_pod_scaler: true on that test target if the job must stay fixed.
3. Check whether admission is healthy on the build cluster (see Verify deployment).

Resources look stuck high (e.g. 20Gi memory)

Admission caps recommendations at --memory-cap=20Gi and --cpu-cap=10. A pod stamped above the cap before a fix keeps that value until the next pod create; admission also clamps corrupt stamped values when it can derive a valid recommendation.

Resources did not change after you edited YAML

Pod-scaler keys on workload metadata (org, repo, branch, variant, target, container), not only the Prow job name. Confirm the pod is in a namespace with ci.openshift.io/scale-pods=true, is not opted out, and is a new pod.

Alert: pod-scaler-admission-resource-warning

Fires when a workload used roughly 10× declared CPU or memory. That usually means undersized YAML, a leak, or bad history — not “add cluster capacity.” See the pod-scaler admission SOP on openshift/release.

How pod-scaler works

Components

Admission runs as pod-scaler-consumer-admission in namespace ci. The webhook matches namespaces labeled ci.openshift.io/scale-pods=true.

Recommendation pipeline

1. Producer records container CPU/memory samples into histograms per workload fingerprint.
2. Consumer merges histograms into recommendations keyed by workload metadata (org, repo, branch, variant, target, container, measured vs unmeasured, etc.).
3. At admission, pod-scaler loads the best matching recommendation (max of measured and unmeasured when both exist).
4. Recommendations use p80 of merged history, then a buffer multiplier on top (hard-coded 1.2× today; upcoming releases expose this as --recommendation-buffer-percent, default 20).
5. Peak limits for digest use histogram max, capped before admission applies them.

Default behavior (increases)

• Admission increases configured requests/limits when the recommendation is higher (unless you opted out).
• Measured pods (~--percentage-measured, 10% on build farms) get extra CPU from --measured-pod-cpu-increase (50%). Authoritative decreases are skipped on measured pods only.
• Memory limits — with --mutate-resource-limits=true, limits are kept and forced to at least 2× memory request after mutation.

Safety rails

Cache load and repair

Consumer reloads GCS cache periodically (hourly tick; restart admission to force an immediate reload after incidents). On load, pod-scaler repairs corrupt cache entries (pkg/pod-scaler/cache_sanitize.go):

• Drops orphan histogram references.
• Removes histograms with impossible spikes (above cluster cap or above 10× baseline).
• Failed reload keeps the previous in-memory cache when one exists.

Already-stamped resources on old pods are unchanged until the next create.

Authoritative decrease mode

When --authoritative-*-request or --authoritative-*-limit is true, admission may lower requests or limits when scaled usage is below what is configured.

When the flag is false, admission logs what it would decrease (authoritative_*_dry_run events) but does not mutate. There are no separate *-dry-run CLI flags anymore — set the authoritative apply flag to false instead.

Build farms today: all four authoritative apply flags are false (decrease dry-run). Max-reduction and usage-basis flags are already wired for when operators enable apply.

Each decrease (when enabled) is bounded by:

Four independent toggles: CPU/memory × request/limit.

Usage basis for decreases

Flag: --pod-scaler-authoritative-decrease-usage-basis

Build farms run peak. Peak basis uses max(peak, request) so a stale low peak cannot shrink below the p80 request recommendation.

Workload skip for decreases

Four optional flags; comma-separated lists; empty = no skip.

Skip affects decreases only; increases still apply. None of these are set on build farms in release main today — they are available when operators need them (for example build image builds).

Workload types (skip matching)

Detection order (first match wins):

Class comes from ci-workload (scheduling pool): builds, tests, longtests, prowjobs, etc.

Operator reference

Release manifest (build farms)

Args live in pod-scaler-admission.yaml. Deployed by ArgoCD ApplicationSet appset-cluster-build-shared on core-ci (build-shared-buildNN-pod-scaler).

Current main excerpt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

- --cpu-cap=10
- --memory-cap=20Gi
- --mutate-resource-limits=true
- --percentage-measured=10
- --measured-pod-cpu-increase=50
- --authoritative-cpu-request=false
- --authoritative-cpu-limit=false
- --authoritative-memory-request=false
- --authoritative-memory-limit=false
- --authoritative-cpu-request-max-reduction-percent=0.50
- --authoritative-cpu-limit-max-reduction-percent=0.50
- --authoritative-memory-request-max-reduction-percent=0.50
- --authoritative-memory-limit-max-reduction-percent=0.50
- --pod-scaler-authoritative-decrease-usage-basis=peak

To apply decreases on one field, set that --authoritative-* flag to true. Roll one field at a time if you are unsure.

Image tag: quay-proxy.ci.openshift.org/openshift/ci:ci_pod-scaler_latest (keel @hourly on build farms).

Flag reference

Runtime and caps

Authoritative apply (four fields)

Max reduction (four fields)

All set to 0.50 on build farms.

Usage basis

--pod-scaler-authoritative-decrease-usage-basis=peak on build farms.

Deprecated aliases (still accepted)

Removed: *-dry-run flags. Use =false on the apply flag.

Verify deployment

On a build cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

CTX=build01   # or build02 … build13

oc --context "$CTX" -n ci get deploy pod-scaler-consumer-admission \
  -o jsonpath='ready={.status.readyReplicas}/{.spec.replicas}{"\n"}'

oc --context "$CTX" -n ci get deploy pod-scaler-consumer-admission \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | jq .

oc --context "$CTX" -n ci get pods -l app=pod-scaler-consumer-admission \
  -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[0].ready}{" "}{.status.containerStatuses[0].imageID}{"\n"}{end}'

Expect 2/2 ready, image digest matching ci_pod-scaler_latest, and no *-dry-run in args.

Producer on app.ci:

1

oc --context app.ci -n ci get deploy pod-scaler-producer pod-scaler-ui

Restart admission after cache incidents:

1

oc --context "$CTX" -n ci rollout restart deploy/pod-scaler-consumer-admission

Operator troubleshooting

Admission CrashLoopBackOff after image rollout

Check logs for unknown flags (old release args vs new binary). Fix: sync ArgoCD app to release main. If deploy args are correct but stale ReplicaSets remain, delete crashing pods or force-sync the Application.

ArgoCD shows Synced but args are wrong

ServerSideApply + ApplyOutOfSyncOnly can leave stale args field managers. Hard-sync the build-shared-buildNN-pod-scaler app on core-ci, or server-side apply the manifest with --force-conflicts.

Absurd CPU in logs (e.g. -9223372036854775808m)

Bad histogram quantile (NaN/Inf). Repair on cache load should drop it; roll admission and confirm producer is healthy.

Suspect corrupt cache in GCS

Consumer repair runs on every load. After producer fixes, restart admission. Operational cache prune scripts live in openshift/ci-tools/hack/ for incident response.

Admission logs worth searching