* refactor(test/blackbox): extract shared push/pull helpers
Move duplicated push/pull/regclient/oras/helm helper functions out of
fips140.bats, pushpull.bats and helpers_upgrade.bash into a single
helpers_pushpull.bash, then have all three suites load from there.
The helpers now share verify_prerequisites, get_zot_port, common assertion
helpers (catalog/tag presence, OCI index ref name) and the regclient
pagination listing. helpers_upgrade.bash keeps only the test_release_*
and test_new_* wrappers that compose those helpers.
Net effect: ~1000 lines of duplicated test scaffolding removed across
the four files; behavior of the existing test cases is preserved.
Refs: #3727
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* refactor(test/blackbox): share pushpull lifecycle and dedupe authn helpers
Build on the shared helpers_pushpull.bash to remove more duplicated
blackbox scaffolding:
- Add pushpull_setup_file/pushpull_teardown/pushpull_teardown_file keyed
on PUSHPULL_FIPS_MODE so pushpull.bats and fips140.bats only set the
flag and delegate lifecycle, instead of each carrying a near-identical
setup_file/teardown.
- Add helpers_pushpull_authn.bash (loaded by pushpull_authn.bats and
fips140_authn.bats) for shared htpasswd setup, FIPS vs non-FIPS config
and teardown, and the regclient/OCI/ML test helpers; both authn suites
collapse to one-line @test bodies keyed on PUSHPULL_AUTHN_FIPS_MODE.
- Make each authn helper self-sufficient by performing its own regctl
login, removing the hidden dependency on the first test having logged
in.
- Split the implicit manifest delete out of helper_pull_image_index_and_delete
into a standalone helper_delete_manifest, surfaced as its own
"delete image index" @test in the pushpull, fips140 and upgrade suites,
so the delete is explicit and no longer an order-fragile side effect of
a pull.
Refs: #3727
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): harden flaky oras pull and sync signature tests
Set org.opencontainers.image.title on oras artifact push and verify pull
both via oras pull and manifest/blob fetch. Add retry_until_success and
use it for periodic notation/cosign signature sync checks on slow CI.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): address Copilot review feedback in push/pull helpers
Use curl --fail for manifest deletes so HTTP errors fail the test. Remove
the duplicate regctl --format flag in helper_push_manifest_with_regclient.
Harden helper_authn_ml_artifacts with run and status checks, using a
binary-safe shell redirect for the ONNX artifact round-trip.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): isolate pushpull helper temp files and prerequisites
Write oras artifact and docker build files under BATS_TEST_TMPDIR
instead of the test working directory, and check git/docker in
verify_prerequisites for clearer failures when running bats directly.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test): use verify_prerequisites exit status in bats setup
Replace `$(verify_prerequisites)` with a direct call across blackbox
and scale-out suites.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): unify blackbox log path to zot/zot-log.json
Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in
pushpull, authn, and upgrade helpers.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): loop over tools in verify_prerequisites
Replace repeated command -v checks with a single loop over curl, jq,
git, and docker.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): fix exit code for retry_until_success
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): inline upgrade tests and drop helpers_upgrade
Call helper_* directly from upgrade.bats and upgrade_minimal.bats,
load helpers_pushpull in those suites, and move teardown there. Remove
helpers_upgrade.bash now that the release/new wrappers are gone.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* refactor(test/blackbox): require explicit args on pushpull helpers
Drop parameter defaults from shared push/pull helpers and pass image,
repository, and pagination values explicitly at each call site.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): clarify docker push/pull negative test naming
Rename helper_push_docker_image to helper_build_docker_image_push_and_pull
and update test titles to reflect build plus expected push/pull failures
without the docker compatibility extension.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): isolate regctl config per BATS suite
regctl persists login and TLS settings on disk, so push/pull blackbox
tests could leak state into ~/.regctl/config.json across suites. Point
REGCTL_CONFIG at BATS_FILE_TMPDIR for pushpull, authn, and upgrade
suites, configure TLS once in authn setup, and drop redundant logout
and per-login TLS setup.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix(test/blackbox): harden upgrade suite log dump on failure
Touch zot-log.json during upgrade setup and only cat it in teardown when
the file exists, so a missing log cannot mask the underlying test failure.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
---------
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Akash Kumar <meakash7902@gmail.com>
* ci: add a upgrade bats test
Fixes https://github.com/project-zot/zot/issues/3601
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* ci: first check existing images
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* ci: refactor into common test code
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* refactor: Refactor upgrade tests to improve readability and maintainability
- Added section headers for release tests, upgrade process, and new tests in upgrade.bats and upgrade_minimal.bats.
- Replaced inline port retrieval with a function call to get_zot_port for consistency.
- Consolidated repeated test logic into dedicated functions (e.g., test_new_existing_pull_image, test_new_push_image).
- Removed unnecessary variable assignments and streamlined test cases for clarity.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
Changes in this PR
==================
- Replaces the get_free_port bash function in BATS tests
with get_free_port_for_service that returns a random free port
in a given range for a test file and service defined in a
ports.json file.
- Updates all get_free_port calls to use the new function.
- A new README file for details on the ports.json file.
- Updates some tests using fixed ports to use dynamic ports.
- Adds a ports.json file with all the allocations.
- Adds a new common helper for port fetching.
Signed-off-by: Vishwas Rajashekar <30438425+vrajashkr@users.noreply.github.com>
* chore: bump github.com/olekukonko/tablewriter from 0.0.5 to 1.0.7
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* fix: zli failed to connect to https server using test certificates
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
BREAKING CHANGE: the dist spec version in the config files needs to be bumped to 1.1.0
in order for the config verification to pass without warnings.
Also fix 1 dependabot alert for helm.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
New examples of running tests:
1. To run a specific bats file (with and without verbose output):
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats BATS_VERBOSITY=2
2. To run the CI tests (with and without verbose output)
make run-blackbox-ci
make run-blackbox-ci BATS_VERBOSITY=2
BATS_TEST_FILE_PATH is used to pass on the test file to run using `run-blackbox-tests`
BATS_VERBOSITY controls the verbosity of the bats framework output, if unspecified the output only
contains test results and failure message in case of failures.
If BATS_VERBOSITY is 1, then also show commands as they are executed.
If BATS_VERBOSITY is 2, on top of the above it also shows output of passed tests.
Other changes in this PR:
- Update some of the tests to show logs after the run ends.
- Run the linters before the tests, as it saves time on failures when running in GH
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Files were added to be built whether an extension is on or off.
New build tags were added for each extension, while minimal and extended disappeared.
added custom binary naming depending on extensions used and changed references from binary to binary-extended
added automated blackbox tests for sync, search, scrub, metrics
added contributor guidelines
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Andrei Aaron