Files
zot/test
Andrei Aaron 95ce68ccb0 refactor(test/blackbox): extract shared push/pull helpers (#4132)
* refactor(test/blackbox): extract shared push/pull helpers

Move duplicated push/pull/regclient/oras/helm helper functions out of
fips140.bats, pushpull.bats and helpers_upgrade.bash into a single
helpers_pushpull.bash, then have all three suites load from there.

The helpers now share verify_prerequisites, get_zot_port, common assertion
helpers (catalog/tag presence, OCI index ref name) and the regclient
pagination listing. helpers_upgrade.bash keeps only the test_release_*
and test_new_* wrappers that compose those helpers.

Net effect: ~1000 lines of duplicated test scaffolding removed across
the four files; behavior of the existing test cases is preserved.

Refs: #3727
Signed-off-by: Akash Kumar <meakash7902@gmail.com>

* refactor(test/blackbox): share pushpull lifecycle and dedupe authn helpers

Build on the shared helpers_pushpull.bash to remove more duplicated
blackbox scaffolding:

- Add pushpull_setup_file/pushpull_teardown/pushpull_teardown_file keyed
  on PUSHPULL_FIPS_MODE so pushpull.bats and fips140.bats only set the
  flag and delegate lifecycle, instead of each carrying a near-identical
  setup_file/teardown.
- Add helpers_pushpull_authn.bash (loaded by pushpull_authn.bats and
  fips140_authn.bats) for shared htpasswd setup, FIPS vs non-FIPS config
  and teardown, and the regclient/OCI/ML test helpers; both authn suites
  collapse to one-line @test bodies keyed on PUSHPULL_AUTHN_FIPS_MODE.
- Make each authn helper self-sufficient by performing its own regctl
  login, removing the hidden dependency on the first test having logged
  in.
- Split the implicit manifest delete out of helper_pull_image_index_and_delete
  into a standalone helper_delete_manifest, surfaced as its own
  "delete image index" @test in the pushpull, fips140 and upgrade suites,
  so the delete is explicit and no longer an order-fragile side effect of
  a pull.

Refs: #3727
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): harden flaky oras pull and sync signature tests

Set org.opencontainers.image.title on oras artifact push and verify pull
both via oras pull and manifest/blob fetch. Add retry_until_success and
use it for periodic notation/cosign signature sync checks on slow CI.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): address Copilot review feedback in push/pull helpers

Use curl --fail for manifest deletes so HTTP errors fail the test. Remove
the duplicate regctl --format flag in helper_push_manifest_with_regclient.
Harden helper_authn_ml_artifacts with run and status checks, using a
binary-safe shell redirect for the ONNX artifact round-trip.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): isolate pushpull helper temp files and prerequisites

Write oras artifact and docker build files under BATS_TEST_TMPDIR
instead of the test working directory, and check git/docker in
verify_prerequisites for clearer failures when running bats directly.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test): use verify_prerequisites exit status in bats setup

Replace `$(verify_prerequisites)` with a direct call across blackbox
and scale-out suites.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): unify blackbox log path to zot/zot-log.json

Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in
pushpull, authn, and upgrade helpers.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): loop over tools in verify_prerequisites

Replace repeated command -v checks with a single loop over curl, jq,
git, and docker.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): fix exit code for retry_until_success

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): inline upgrade tests and drop helpers_upgrade

Call helper_* directly from upgrade.bats and upgrade_minimal.bats,
load helpers_pushpull in those suites, and move teardown there. Remove
helpers_upgrade.bash now that the release/new wrappers are gone.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): require explicit args on pushpull helpers

Drop parameter defaults from shared push/pull helpers and pass image,
repository, and pagination values explicitly at each call site.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): clarify docker push/pull negative test naming

Rename helper_push_docker_image to helper_build_docker_image_push_and_pull
and update test titles to reflect build plus expected push/pull failures
without the docker compatibility extension.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): isolate regctl config per BATS suite

regctl persists login and TLS settings on disk, so push/pull blackbox
tests could leak state into ~/.regctl/config.json across suites. Point
REGCTL_CONFIG at BATS_FILE_TMPDIR for pushpull, authn, and upgrade
suites, configure TLS once in authn setup, and drop redundant logout
and per-login TLS setup.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): harden upgrade suite log dump on failure

Touch zot-log.json during upgrade setup and only cat it in teardown when
the file exists, so a missing log cannot mask the underlying test failure.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

---------

Signed-off-by: Akash Kumar <meakash7902@gmail.com>
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Akash Kumar <meakash7902@gmail.com>
2026-06-15 10:56:11 +03:00
..

BATS Port Allocations

The ports.json file in this directory tracks the port ranges provided for each BATS test that runs concurrently to avoid overlaps. IANA registered ports range - 1024 to 49151

Adding a new BATS Test file

For the range, use a gap of 10 ports (e.g. 9000 to 9009) and leave a gap of 10 ports between the new range and the last range.

Avoid ranges in the 4000, 8000, and 10000 series as these may overlap with other services such as localstack, fixed port zot, and clustered zot.

For a new BATS test file, add a new entry to ports.json as follows:

replace TEST_DIR with just the directory name of the directory containing your test file. replace FILENAME with the name of the test file along with its extension e.g. new_test.bats

"TEST_DIR/FILENAME": {
  "svc1": {
    "begin": 20020,
    "end": 20029
  },
  "svc2": {
    "begin": 20040,
    "end": 20049
  }
}

A test file may have multiple services defined by a unique key. You can use any key for the service identifier, however, ensure that the same key is used in the BATS test file as an argument to the get_free_port_for_service function.