Files
zot/test/scale-out/cloud_scale_out_no_auth.bats
Andrei Aaron 95ce68ccb0 refactor(test/blackbox): extract shared push/pull helpers (#4132)
* refactor(test/blackbox): extract shared push/pull helpers

Move duplicated push/pull/regclient/oras/helm helper functions out of
fips140.bats, pushpull.bats and helpers_upgrade.bash into a single
helpers_pushpull.bash, then have all three suites load from there.

The helpers now share verify_prerequisites, get_zot_port, common assertion
helpers (catalog/tag presence, OCI index ref name) and the regclient
pagination listing. helpers_upgrade.bash keeps only the test_release_*
and test_new_* wrappers that compose those helpers.

Net effect: ~1000 lines of duplicated test scaffolding removed across
the four files; behavior of the existing test cases is preserved.

Refs: #3727
Signed-off-by: Akash Kumar <meakash7902@gmail.com>

* refactor(test/blackbox): share pushpull lifecycle and dedupe authn helpers

Build on the shared helpers_pushpull.bash to remove more duplicated
blackbox scaffolding:

- Add pushpull_setup_file/pushpull_teardown/pushpull_teardown_file keyed
  on PUSHPULL_FIPS_MODE so pushpull.bats and fips140.bats only set the
  flag and delegate lifecycle, instead of each carrying a near-identical
  setup_file/teardown.
- Add helpers_pushpull_authn.bash (loaded by pushpull_authn.bats and
  fips140_authn.bats) for shared htpasswd setup, FIPS vs non-FIPS config
  and teardown, and the regclient/OCI/ML test helpers; both authn suites
  collapse to one-line @test bodies keyed on PUSHPULL_AUTHN_FIPS_MODE.
- Make each authn helper self-sufficient by performing its own regctl
  login, removing the hidden dependency on the first test having logged
  in.
- Split the implicit manifest delete out of helper_pull_image_index_and_delete
  into a standalone helper_delete_manifest, surfaced as its own
  "delete image index" @test in the pushpull, fips140 and upgrade suites,
  so the delete is explicit and no longer an order-fragile side effect of
  a pull.

Refs: #3727
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): harden flaky oras pull and sync signature tests

Set org.opencontainers.image.title on oras artifact push and verify pull
both via oras pull and manifest/blob fetch. Add retry_until_success and
use it for periodic notation/cosign signature sync checks on slow CI.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): address Copilot review feedback in push/pull helpers

Use curl --fail for manifest deletes so HTTP errors fail the test. Remove
the duplicate regctl --format flag in helper_push_manifest_with_regclient.
Harden helper_authn_ml_artifacts with run and status checks, using a
binary-safe shell redirect for the ONNX artifact round-trip.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): isolate pushpull helper temp files and prerequisites

Write oras artifact and docker build files under BATS_TEST_TMPDIR
instead of the test working directory, and check git/docker in
verify_prerequisites for clearer failures when running bats directly.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test): use verify_prerequisites exit status in bats setup

Replace `$(verify_prerequisites)` with a direct call across blackbox
and scale-out suites.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): unify blackbox log path to zot/zot-log.json

Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in
pushpull, authn, and upgrade helpers.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): loop over tools in verify_prerequisites

Replace repeated command -v checks with a single loop over curl, jq,
git, and docker.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): fix exit code for retry_until_success

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): inline upgrade tests and drop helpers_upgrade

Call helper_* directly from upgrade.bats and upgrade_minimal.bats,
load helpers_pushpull in those suites, and move teardown there. Remove
helpers_upgrade.bash now that the release/new wrappers are gone.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): require explicit args on pushpull helpers

Drop parameter defaults from shared push/pull helpers and pass image,
repository, and pagination values explicitly at each call site.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): clarify docker push/pull negative test naming

Rename helper_push_docker_image to helper_build_docker_image_push_and_pull
and update test titles to reflect build plus expected push/pull failures
without the docker compatibility extension.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): isolate regctl config per BATS suite

regctl persists login and TLS settings on disk, so push/pull blackbox
tests could leak state into ~/.regctl/config.json across suites. Point
REGCTL_CONFIG at BATS_FILE_TMPDIR for pushpull, authn, and upgrade
suites, configure TLS once in authn setup, and drop redundant logout
and per-login TLS setup.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): harden upgrade suite log dump on failure

Touch zot-log.json during upgrade setup and only cat it in teardown when
the file exists, so a missing log cannot mask the underlying test failure.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

---------

Signed-off-by: Akash Kumar <meakash7902@gmail.com>
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Akash Kumar <meakash7902@gmail.com>
2026-06-15 10:56:11 +03:00

84 lines
2.7 KiB
Bash

# note: intended to be run as "make run-cloud-scale-out-tests"
# makefile target installs & checks all necessary tooling
# extra tools that are not covered in Makefile target needs to be added in verify_prerequisites()
NUM_ZOT_INSTANCES=6
ZOT_LOG_DIR=/tmp/zot-ft-logs/no-auth
load helpers_zot
load helpers_cloud
load helpers_haproxy
load ../port_helper
function launch_zot_server() {
local zot_server_address=${1}
local zot_server_port=${2}
local zot_root_dir=${ZOT_ROOT_DIR}
mkdir -p ${zot_root_dir}
mkdir -p ${ZOT_LOG_DIR}
local zot_config_file="${BATS_FILE_TMPDIR}/zot_config_${zot_server_address}_${zot_server_port}.json"
local zot_log_file="${ZOT_LOG_DIR}/zot-${zot_server_address}-${zot_server_port}.log"
create_zot_cloud_base_config_file ${zot_server_address} ${zot_server_port} ${zot_root_dir} ${zot_config_file} ${zot_log_file}
update_zot_cluster_member_list_in_config_file ${zot_config_file} ${ZOT_CLUSTER_MEMBERS_PATCH_FILE}
echo "launching zot server ${zot_server_address}:${zot_server_port}" >&3
echo "config file: ${zot_config_file}" >&3
echo "log file: ${zot_log_file}" >&3
zot_serve ${zot_config_file}
wait_zot_reachable ${zot_server_port}
}
function setup() {
# verify prerequisites are available
if ! verify_prerequisites; then
exit 1
fi
# setup S3 bucket and DynamoDB tables
setup_cloud_services
# generate the free ports list
zot_srv_ports=()
for ((i=0;i<${NUM_ZOT_INSTANCES};i++)); do
port=$(get_free_port_for_service "zot${i}")
zot_srv_ports+=("${port}")
done
generate_zot_cluster_member_list ${NUM_ZOT_INSTANCES} ${ZOT_CLUSTER_MEMBERS_PATCH_FILE} "${zot_srv_ports[@]}"
for inst in "${zot_srv_ports[@]}"; do
launch_zot_server 127.0.0.1 ${inst}
done
# list all zot processes that were started
ps -ef | grep ".*zot.*serve.*" | grep -v grep >&3
haproxy_port=$(get_free_port_for_service "haproxy")
echo ${haproxy_port} > ${BATS_FILE_TMPDIR}/haproxy.port
generate_haproxy_config ${HAPROXY_CFG_FILE} "http" ${haproxy_port} "${zot_srv_ports[@]}"
haproxy_start ${HAPROXY_CFG_FILE}
# list HAproxy processes that were started
ps -ef | grep "haproxy" | grep -v grep >&3
}
function teardown() {
local zot_root_dir=${ZOT_ROOT_DIR}
haproxy_stop_all
zot_stop_all
rm -rf ${zot_root_dir}
teardown_cloud_services
}
@test "Check for successful zb run on haproxy frontend" {
haproxy_port=`cat ${BATS_FILE_TMPDIR}/haproxy.port`
# zb_run <test_name> <zot_address> <concurrency> <num_requests> <credentials (optional)>
zb_run "cloud-scale-out-no-auth-bats" "http://127.0.0.1:${haproxy_port}" 3 5
}