* refactor(test/blackbox): extract shared push/pull helpers
Move duplicated push/pull/regclient/oras/helm helper functions out of
fips140.bats, pushpull.bats and helpers_upgrade.bash into a single
helpers_pushpull.bash, then have all three suites load from there.
The helpers now share verify_prerequisites, get_zot_port, common assertion
helpers (catalog/tag presence, OCI index ref name) and the regclient
pagination listing. helpers_upgrade.bash keeps only the test_release_*
and test_new_* wrappers that compose those helpers.
Net effect: ~1000 lines of duplicated test scaffolding removed across
the four files; behavior of the existing test cases is preserved.
Refs: #3727
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* refactor(test/blackbox): share pushpull lifecycle and dedupe authn helpers
Build on the shared helpers_pushpull.bash to remove more duplicated
blackbox scaffolding:
- Add pushpull_setup_file/pushpull_teardown/pushpull_teardown_file keyed
on PUSHPULL_FIPS_MODE so pushpull.bats and fips140.bats only set the
flag and delegate lifecycle, instead of each carrying a near-identical
setup_file/teardown.
- Add helpers_pushpull_authn.bash (loaded by pushpull_authn.bats and
fips140_authn.bats) for shared htpasswd setup, FIPS vs non-FIPS config
and teardown, and the regclient/OCI/ML test helpers; both authn suites
collapse to one-line @test bodies keyed on PUSHPULL_AUTHN_FIPS_MODE.
- Make each authn helper self-sufficient by performing its own regctl
login, removing the hidden dependency on the first test having logged
in.
- Split the implicit manifest delete out of helper_pull_image_index_and_delete
into a standalone helper_delete_manifest, surfaced as its own
"delete image index" @test in the pushpull, fips140 and upgrade suites,
so the delete is explicit and no longer an order-fragile side effect of
a pull.
Refs: #3727
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): harden flaky oras pull and sync signature tests
Set org.opencontainers.image.title on oras artifact push and verify pull
both via oras pull and manifest/blob fetch. Add retry_until_success and
use it for periodic notation/cosign signature sync checks on slow CI.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): address Copilot review feedback in push/pull helpers
Use curl --fail for manifest deletes so HTTP errors fail the test. Remove
the duplicate regctl --format flag in helper_push_manifest_with_regclient.
Harden helper_authn_ml_artifacts with run and status checks, using a
binary-safe shell redirect for the ONNX artifact round-trip.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): isolate pushpull helper temp files and prerequisites
Write oras artifact and docker build files under BATS_TEST_TMPDIR
instead of the test working directory, and check git/docker in
verify_prerequisites for clearer failures when running bats directly.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test): use verify_prerequisites exit status in bats setup
Replace `$(verify_prerequisites)` with a direct call across blackbox
and scale-out suites.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): unify blackbox log path to zot/zot-log.json
Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in
pushpull, authn, and upgrade helpers.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): loop over tools in verify_prerequisites
Replace repeated command -v checks with a single loop over curl, jq,
git, and docker.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): fix exit code for retry_until_success
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): inline upgrade tests and drop helpers_upgrade
Call helper_* directly from upgrade.bats and upgrade_minimal.bats,
load helpers_pushpull in those suites, and move teardown there. Remove
helpers_upgrade.bash now that the release/new wrappers are gone.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): require explicit args on pushpull helpers
Drop parameter defaults from shared push/pull helpers and pass image,
repository, and pagination values explicitly at each call site.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): clarify docker push/pull negative test naming
Rename helper_push_docker_image to helper_build_docker_image_push_and_pull
and update test titles to reflect build plus expected push/pull failures
without the docker compatibility extension.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): isolate regctl config per BATS suite
regctl persists login and TLS settings on disk, so push/pull blackbox
tests could leak state into ~/.regctl/config.json across suites. Point
REGCTL_CONFIG at BATS_FILE_TMPDIR for pushpull, authn, and upgrade
suites, configure TLS once in authn setup, and drop redundant logout
and per-login TLS setup.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): harden upgrade suite log dump on failure
Touch zot-log.json during upgrade setup and only cat it in teardown when
the file exists, so a missing log cannot mask the underlying test failure.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
---------
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Akash Kumar <meakash7902@gmail.com>
Changes in this PR
==================
- Replaces the get_free_port bash function in BATS tests
with get_free_port_for_service that returns a random free port
in a given range for a test file and service defined in a
ports.json file.
- Updates all get_free_port calls to use the new function.
- A new README file for details on the ports.json file.
- Updates some tests using fixed ports to use dynamic ports.
- Adds a ports.json file with all the allocations.
- Adds a new common helper for port fetching.
Signed-off-by: Vishwas Rajashekar <30438425+vrajashkr@users.noreply.github.com>
* fix(headers): set Cross-Origin-Resource-Policy header for UI requests
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* ci: disable stacker tests in ecosystem tools as they do not work on Ubuntu 24
Workaround this:
The Ubuntu 22 github free runner is not powerful enough to run all these tests.
We need a real fix on stacker side
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
BREAKING CHANGE: the dist spec version in the config files needs to be bumped to 1.1.0
in order for the config verification to pass without warnings.
Also fix 1 dependabot alert for helm.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- Cosign supports 2 types of signature formats:
1. Using tag -> each new signature of the same manifest is
added as a new layer of the signature manifest having that
specific tag("{alghoritm}-{digest_of_signed_manifest}.sig")
2. Using referrers -> each new signature of the same manifest is
added as a new manifest
- For adding these cosign signature to metadb, we reserved index 0 of the
list of cosign signatures for tag-based signatures. When a new tag-based
signature is added for the same manifest, the element on first position
in its list of cosign signatures(in metadb) will be updated/overwritten.
When a new cosign signature(using referrers) will be added for the same
manifest this new signature will be appended to the list of cosign
signatures.
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
New examples of running tests:
1. To run a specific bats file (with and without verbose output):
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats BATS_VERBOSITY=2
2. To run the CI tests (with and without verbose output)
make run-blackbox-ci
make run-blackbox-ci BATS_VERBOSITY=2
BATS_TEST_FILE_PATH is used to pass on the test file to run using `run-blackbox-tests`
BATS_VERBOSITY controls the verbosity of the bats framework output, if unspecified the output only
contains test results and failure message in case of failures.
If BATS_VERBOSITY is 1, then also show commands as they are executed.
If BATS_VERBOSITY is 2, on top of the above it also shows output of passed tests.
Other changes in this PR:
- Update some of the tests to show logs after the run ends.
- Run the linters before the tests, as it saves time on failures when running in GH
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Andrei Aaron