Files
zot/test/blackbox/redis_session_store.bats
Andrei Aaron 95ce68ccb0 refactor(test/blackbox): extract shared push/pull helpers (#4132)
* refactor(test/blackbox): extract shared push/pull helpers

Move duplicated push/pull/regclient/oras/helm helper functions out of
fips140.bats, pushpull.bats and helpers_upgrade.bash into a single
helpers_pushpull.bash, then have all three suites load from there.

The helpers now share verify_prerequisites, get_zot_port, common assertion
helpers (catalog/tag presence, OCI index ref name) and the regclient
pagination listing. helpers_upgrade.bash keeps only the test_release_*
and test_new_* wrappers that compose those helpers.

Net effect: ~1000 lines of duplicated test scaffolding removed across
the four files; behavior of the existing test cases is preserved.

Refs: #3727
Signed-off-by: Akash Kumar <meakash7902@gmail.com>

* refactor(test/blackbox): share pushpull lifecycle and dedupe authn helpers

Build on the shared helpers_pushpull.bash to remove more duplicated
blackbox scaffolding:

- Add pushpull_setup_file/pushpull_teardown/pushpull_teardown_file keyed
  on PUSHPULL_FIPS_MODE so pushpull.bats and fips140.bats only set the
  flag and delegate lifecycle, instead of each carrying a near-identical
  setup_file/teardown.
- Add helpers_pushpull_authn.bash (loaded by pushpull_authn.bats and
  fips140_authn.bats) for shared htpasswd setup, FIPS vs non-FIPS config
  and teardown, and the regclient/OCI/ML test helpers; both authn suites
  collapse to one-line @test bodies keyed on PUSHPULL_AUTHN_FIPS_MODE.
- Make each authn helper self-sufficient by performing its own regctl
  login, removing the hidden dependency on the first test having logged
  in.
- Split the implicit manifest delete out of helper_pull_image_index_and_delete
  into a standalone helper_delete_manifest, surfaced as its own
  "delete image index" @test in the pushpull, fips140 and upgrade suites,
  so the delete is explicit and no longer an order-fragile side effect of
  a pull.

Refs: #3727
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): harden flaky oras pull and sync signature tests

Set org.opencontainers.image.title on oras artifact push and verify pull
both via oras pull and manifest/blob fetch. Add retry_until_success and
use it for periodic notation/cosign signature sync checks on slow CI.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): address Copilot review feedback in push/pull helpers

Use curl --fail for manifest deletes so HTTP errors fail the test. Remove
the duplicate regctl --format flag in helper_push_manifest_with_regclient.
Harden helper_authn_ml_artifacts with run and status checks, using a
binary-safe shell redirect for the ONNX artifact round-trip.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): isolate pushpull helper temp files and prerequisites

Write oras artifact and docker build files under BATS_TEST_TMPDIR
instead of the test working directory, and check git/docker in
verify_prerequisites for clearer failures when running bats directly.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test): use verify_prerequisites exit status in bats setup

Replace `$(verify_prerequisites)` with a direct call across blackbox
and scale-out suites.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): unify blackbox log path to zot/zot-log.json

Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in
pushpull, authn, and upgrade helpers.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): loop over tools in verify_prerequisites

Replace repeated command -v checks with a single loop over curl, jq,
git, and docker.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): fix exit code for retry_until_success

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): inline upgrade tests and drop helpers_upgrade

Call helper_* directly from upgrade.bats and upgrade_minimal.bats,
load helpers_pushpull in those suites, and move teardown there. Remove
helpers_upgrade.bash now that the release/new wrappers are gone.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* refactor(test/blackbox): require explicit args on pushpull helpers

Drop parameter defaults from shared push/pull helpers and pass image,
repository, and pagination values explicitly at each call site.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): clarify docker push/pull negative test naming

Rename helper_push_docker_image to helper_build_docker_image_push_and_pull
and update test titles to reflect build plus expected push/pull failures
without the docker compatibility extension.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): isolate regctl config per BATS suite

regctl persists login and TLS settings on disk, so push/pull blackbox
tests could leak state into ~/.regctl/config.json across suites. Point
REGCTL_CONFIG at BATS_FILE_TMPDIR for pushpull, authn, and upgrade
suites, configure TLS once in authn setup, and drop redundant logout
and per-login TLS setup.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

* fix(test/blackbox): harden upgrade suite log dump on failure

Touch zot-log.json during upgrade setup and only cat it in teardown when
the file exists, so a missing log cannot mask the underlying test failure.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

---------

Signed-off-by: Akash Kumar <meakash7902@gmail.com>
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Akash Kumar <meakash7902@gmail.com>
2026-06-15 10:56:11 +03:00

254 lines
7.5 KiB
Bash

# Note: Intended to be run as "make run-blackbox-tests" or "make run-blackbox-ci"
# Makefile target installs & checks all necessary tooling
# Extra tools that are not covered in Makefile target needs to be added in verify_prerequisites()
load helpers_zot
load helpers_redis
load ../port_helper
function verify_prerequisites() {
if [ ! $(command -v curl) ]; then
echo "you need to install curl as a prerequisite to running the tests" >&3
return 1
fi
if [ ! $(command -v docker) ]; then
echo "you need to install docker as a prerequisite to running the tests" >&3
return 1
fi
if [ ! $(command -v valkey-cli) ]; then
echo "you need to install valkey-cli as a prerequisite to running the tests" >&3
return 1
fi
return 0
}
HTPASSWD_PATH=/tmp/zotpasswd
CURL_COOKIES_DIR=/tmp/zotcookies
REDIS_TEST_CONTAINER_NAME="redis_sessions_server_local"
function setup_file() {
# Verify prerequisites are available
if ! verify_prerequisites; then
exit 1
fi
mkdir -p ${CURL_COOKIES_DIR}
# Create htpasswd file for basic auth
htpasswd -bBn test test123 > ${HTPASSWD_PATH}
# Setup redis server
redis_port=$(get_free_port_for_service "redis")
redis_start ${REDIS_TEST_CONTAINER_NAME} ${redis_port}
# Setup zot server
local zot_root_dir=${BATS_FILE_TMPDIR}/zot
local zot_redis_session_config_file=${BATS_FILE_TMPDIR}/zot_redis_session_config.json
zot_port=$(get_free_port_for_service "zot")
echo ${zot_port} > ${BATS_FILE_TMPDIR}/zot.port
echo ${redis_port} > ${BATS_FILE_TMPDIR}/redis.port
mkdir -p ${zot_root_dir}
cat >${zot_redis_session_config_file} <<EOF
{
"distSpecVersion": "1.1.1",
"storage": {
"rootDirectory": "${zot_root_dir}"
},
"http": {
"address": "127.0.0.1",
"port": "${zot_port}",
"auth": {
"htpasswd": {
"path": "${HTPASSWD_PATH}"
},
"sessionDriver": {
"name": "redis",
"url": "redis://localhost:${redis_port}",
"keyprefix": "zotsession"
}
}
},
"log": {
"level": "debug",
"output": "/tmp/blackbox.log"
},
"extensions": {
"ui": {
"enable": true
},
"search": {
"enable": true
}
}
}
EOF
zot_serve ${ZOT_PATH} ${zot_redis_session_config_file}
wait_zot_reachable ${zot_port}
}
function get_zot_port() {
cat ${BATS_FILE_TMPDIR}/zot.port
}
function get_redis_port() {
cat ${BATS_FILE_TMPDIR}/redis.port
}
function get_session_count() {
port=$(get_redis_port)
valkey-cli -u "redis://localhost:${port}" --scan --pattern 'zotsession:*' | wc -l
}
function perform_login() {
zot_port=$(get_zot_port)
user_num=$1
# The authorization header carries a base 64 encode of test:test123
curl -s -o /dev/null -w '%{http_code}' --cookie-jar "${CURL_COOKIES_DIR}/zot-cookie-${user_num}" \
"http://localhost:${zot_port}/v2/" \
-H 'Accept: application/json, text/plain, */*' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Accept-Encoding: gzip, deflate, br, zstd' \
-H 'Authorization: Basic dGVzdDp0ZXN0MTIz' \
-H 'X-ZOT-API-CLIENT: zot-ui' \
-H 'Connection: keep-alive' \
-H "Referer: http://localhost:${zot_port}/login" \
-H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Priority: u=0' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache'
}
function perform_logout() {
zot_port=$(get_zot_port)
user_num=$1
curl -s -o /dev/null -w '%{http_code}' --cookie "${CURL_COOKIES_DIR}/zot-cookie-${user_num}" \
-X POST \
"http://localhost:${zot_port}/zot/auth/logout" \
-H 'Accept: application/json, text/plain, */*' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Accept-Encoding: gzip, deflate, br, zstd' \
-H "Origin: http://localhost:${zot_port}/login" \
-H 'X-ZOT-API-CLIENT: zot-ui' \
-H 'Connection: keep-alive' \
-H "Referer: http://localhost:${zot_port}/home" \
-H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Priority: u=0' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache' \
-H 'Content-Length: 0'
}
function perform_authenticated_globalsearch() {
zot_port=$(get_zot_port)
user_num=$1
url="http://localhost:${zot_port}"
url+='/v2/_zot/ext/search?query={GlobalSearch(query:%22%22,%20requestedPage:%20{limit:3%20offset:0%20sortBy:%20DOWNLOADS}%20)%20{Page%20{TotalCount%20ItemCount}%20Repos%20{Name%20LastUpdated%20Size%20Platforms%20{%20Os%20Arch%20}%20IsStarred%20IsBookmarked%20NewestImage%20{%20Tag%20Vulnerabilities%20{MaxSeverity%20Count}%20Description%20IsSigned%20SignatureInfo%20{%20Tool%20IsTrusted%20Author%20}%20Licenses%20Vendor%20Labels%20}%20StarCount%20DownloadCount}}}'
curl -g -s -o /dev/null -w '%{http_code}' --cookie "${CURL_COOKIES_DIR}/zot-cookie-${user_num}" \
"${url}" \
-H 'Accept: application/json' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Accept-Encoding: gzip, deflate, br, zstd' \
-H 'X-ZOT-API-CLIENT: zot-ui' \
-H 'Connection: keep-alive' \
-H "Referer: http://localhost:${zot_port}/home" \
-H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'Pragma: no-cache' \
-H 'Cache-Control: no-cache'
}
@test "verify bulk user authentication cycle" {
num_users=20
# Note: queries are forked and run concurrently for load
for i in $(seq 1 ${num_users}); do
(
# User tries to access the global search URL without login
echo "user $i unauthenticated URL check"
status=$(perform_authenticated_globalsearch $i)
[ 401 -eq "${status}" ]
# User login
echo "user $i login"
status=$(perform_login $i)
[ 200 -eq "${status}" ]
) &
done
# wait for background processes to complete
sleep 0.1
echo "waiting for background process completion"
wait $(jobs -p)
for i in $(seq 1 ${num_users}); do
# Retry authenticated global search URL
(
echo "user $i authenticated URL check"
status=$(perform_authenticated_globalsearch $i)
[ 200 -eq "${status}" ]
) &
done
# wait for background processes to complete
sleep 0.1
echo "waiting for background process completion"
wait $(jobs -p)
cookies_count=$(get_session_count)
echo "total cookies ${cookies_count}"
[ "${cookies_count}" -eq "${num_users}" ]
for i in $(seq 1 ${num_users}); do
# All users logout
(
status=$(perform_logout $i)
[ 200 -eq "${status}" ]
) &
done
# wait for background processes to complete
sleep 0.1
echo "waiting for background process completion"
wait $(jobs -p)
for i in $(seq 1 ${num_users}); do
# All users verify no access to URL
(
status=$(perform_authenticated_globalsearch $i)
[ 401 -eq "${status}" ]
) &
done
# wait for background processes to complete
sleep 0.1
echo "waiting for background process completion"
wait $(jobs -p)
cookies_count=$(get_session_count)
echo "total cookies ${cookies_count}"
[ 0 -eq "${cookies_count}" ]
}
function teardown_file() {
zot_stop_all
redis_stop ${REDIS_TEST_CONTAINER_NAME}
rm ${HTPASSWD_PATH}
rm -r ${CURL_COOKIES_DIR}
}