* refactor(test/blackbox): extract shared push/pull helpers
Move duplicated push/pull/regclient/oras/helm helper functions out of
fips140.bats, pushpull.bats and helpers_upgrade.bash into a single
helpers_pushpull.bash, then have all three suites load from there.
The helpers now share verify_prerequisites, get_zot_port, common assertion
helpers (catalog/tag presence, OCI index ref name) and the regclient
pagination listing. helpers_upgrade.bash keeps only the test_release_*
and test_new_* wrappers that compose those helpers.
Net effect: ~1000 lines of duplicated test scaffolding removed across
the four files; behavior of the existing test cases is preserved.
Refs: #3727
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* refactor(test/blackbox): share pushpull lifecycle and dedupe authn helpers
Build on the shared helpers_pushpull.bash to remove more duplicated
blackbox scaffolding:
- Add pushpull_setup_file/pushpull_teardown/pushpull_teardown_file keyed
on PUSHPULL_FIPS_MODE so pushpull.bats and fips140.bats only set the
flag and delegate lifecycle, instead of each carrying a near-identical
setup_file/teardown.
- Add helpers_pushpull_authn.bash (loaded by pushpull_authn.bats and
fips140_authn.bats) for shared htpasswd setup, FIPS vs non-FIPS config
and teardown, and the regclient/OCI/ML test helpers; both authn suites
collapse to one-line @test bodies keyed on PUSHPULL_AUTHN_FIPS_MODE.
- Make each authn helper self-sufficient by performing its own regctl
login, removing the hidden dependency on the first test having logged
in.
- Split the implicit manifest delete out of helper_pull_image_index_and_delete
into a standalone helper_delete_manifest, surfaced as its own
"delete image index" @test in the pushpull, fips140 and upgrade suites,
so the delete is explicit and no longer an order-fragile side effect of
a pull.
Refs: #3727
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): harden flaky oras pull and sync signature tests
Set org.opencontainers.image.title on oras artifact push and verify pull
both via oras pull and manifest/blob fetch. Add retry_until_success and
use it for periodic notation/cosign signature sync checks on slow CI.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): address Copilot review feedback in push/pull helpers
Use curl --fail for manifest deletes so HTTP errors fail the test. Remove
the duplicate regctl --format flag in helper_push_manifest_with_regclient.
Harden helper_authn_ml_artifacts with run and status checks, using a
binary-safe shell redirect for the ONNX artifact round-trip.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): isolate pushpull helper temp files and prerequisites
Write oras artifact and docker build files under BATS_TEST_TMPDIR
instead of the test working directory, and check git/docker in
verify_prerequisites for clearer failures when running bats directly.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test): use verify_prerequisites exit status in bats setup
Replace `$(verify_prerequisites)` with a direct call across blackbox
and scale-out suites.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): unify blackbox log path to zot/zot-log.json
Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in
pushpull, authn, and upgrade helpers.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): loop over tools in verify_prerequisites
Replace repeated command -v checks with a single loop over curl, jq,
git, and docker.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): fix exit code for retry_until_success
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): inline upgrade tests and drop helpers_upgrade
Call helper_* directly from upgrade.bats and upgrade_minimal.bats,
load helpers_pushpull in those suites, and move teardown there. Remove
helpers_upgrade.bash now that the release/new wrappers are gone.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): require explicit args on pushpull helpers
Drop parameter defaults from shared push/pull helpers and pass image,
repository, and pagination values explicitly at each call site.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): clarify docker push/pull negative test naming
Rename helper_push_docker_image to helper_build_docker_image_push_and_pull
and update test titles to reflect build plus expected push/pull failures
without the docker compatibility extension.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): isolate regctl config per BATS suite
regctl persists login and TLS settings on disk, so push/pull blackbox
tests could leak state into ~/.regctl/config.json across suites. Point
REGCTL_CONFIG at BATS_FILE_TMPDIR for pushpull, authn, and upgrade
suites, configure TLS once in authn setup, and drop redundant logout
and per-login TLS setup.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): harden upgrade suite log dump on failure
Touch zot-log.json during upgrade setup and only cat it in teardown when
the file exists, so a missing log cannot mask the underlying test failure.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
---------
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Akash Kumar <meakash7902@gmail.com>
Changes in this PR
==================
- Replaces the get_free_port bash function in BATS tests
with get_free_port_for_service that returns a random free port
in a given range for a test file and service defined in a
ports.json file.
- Updates all get_free_port calls to use the new function.
- A new README file for details on the ports.json file.
- Updates some tests using fixed ports to use dynamic ports.
- Adds a ports.json file with all the allocations.
- Adds a new common helper for port fetching.
Signed-off-by: Vishwas Rajashekar <30438425+vrajashkr@users.noreply.github.com>
Andrei Aaron