refactor(makefile): consolidate the make targets used for bats tests (#1746)

New examples of running tests:

1. To run a specific bats file (with and without verbose output):
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats
make run-blackbox-tests BATS_TEST_FILE_PATH=test/blackbox/delete_images.bats BATS_VERBOSITY=2

2. To run the CI tests (with and without verbose output)
make run-blackbox-ci
make run-blackbox-ci BATS_VERBOSITY=2

BATS_TEST_FILE_PATH is used to pass on the test file to run using `run-blackbox-tests`
BATS_VERBOSITY controls the verbosity of the bats framework output, if unspecified the output only
contains test results and failure message in case of failures.
If BATS_VERBOSITY is 1, then also show commands as they are executed.
If BATS_VERBOSITY is 2, on top of the above it also shows output of passed tests.

Other changes in this PR:
- Update some of the tests to show logs after the run ends.
- Run the linters before the tests, as it saves time on failures when running in GH

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
This commit is contained in:
Andrei Aaron
2023-09-07 21:06:21 +03:00
committed by GitHub
parent f5b63963be
commit 6115eed4ec
22 changed files with 199 additions and 211 deletions
+3 -36
View File
@@ -69,42 +69,9 @@ jobs:
sudo du -sh /var/lib/docker/
du -sh /home/runner/work/
set +x
- name: Run referrers tests
- name: Run CI tests
run: |
make test-bats-referrers
- name: Run metadata tests
run: |
make test-bats-metadata
- name: Run push-pull tests
run: |
make test-push-pull
- name: Run push-pull-authn tests
run: |
make test-push-pull-authn
- name: Run metrics tests
run: |
make test-bats-metrics
- name: Run cve tests
run: |
make test-bats-cve
- name: Run sync test
run: |
make test-bats-sync
- name: Run scrub tests
run: |
make test-bats-scrub
- name: Run anonymous-push-pull tests
run: |
make test-anonymous-push-pull
- name: Run detect-manifest-collision tests
run: |
make test-detect-manifest-collision
- name: Run annotations tests
run: |
make test-annotations
- name: Run garbage collect tests
run: |
make test-garbage-collect
make run-blackbox-ci
- name: Install localstack
run: |
pip install --upgrade pyopenssl
@@ -117,7 +84,7 @@ jobs:
echo "Startup complete"
- name: Run cloud-only tests
run: |
make test-cloud-only
make run-blackbox-cloud-ci
env:
AWS_ACCESS_KEY_ID: fake
AWS_SECRET_ACCESS_KEY: fake