GCS storage support (#3798)

feat(storage): add a GCS driver

test(storage): add unit tests for GCS driver

test(storage): add missing unit tests for GCS driver & resolve lint issues

fix: configuration validation for GCS Storage

test(storage): resolve panic by test due to setupGCS ignoring returned error

test(storage): add dummy gcs credentials

test: add darwin support for macos to run tests

ci: update workflows to pin gcs emulator version

lint: resolve long line lengths & formatting issues

test: move error for gcs mock earlier with an error

test: stop test using local google credentials and use mock instead

test: add missing dummy creds

test(storage): use storage-testbench for GCS, isolate GCS tests, fix driver Delete

- Switch GCS emulator from fake-gcs-server to storage-testbench in CI.
  Run the GCS emulator only in the privileged-test job; remove it from
  minimal and extended test jobs.

- Consolidate GCS tests under pkg/storage/gcs (needprivileges,linux).
  Add TestMain with HTTPS proxy and /etc/hosts so tests talk to
  storage-testbench; move GCS-specific cases from storage_test.go and
  scrub_test.go into gcs_test.go. Run GCS tests via a second privileged-test
  invocation and collect coverage in coverage-needprivileges-gcs.txt.

- Make GCS driver Delete idempotent and normalize errors. Treat
  PathNotFoundError from Delete as success so that deleting an already-gone
  path (e.g. after GC under eventual consistency) does not fail. Add
  formatErr to map 404/not found to PathNotFoundError and use it for all
  driver methods so callers get consistent storage driver errors.

- Drop GCS branches and helpers from storage_test.go and scrub_test.go so
  non-privileged tests only use local/S3; GCS is tested only in
  pkg/storage/gcs with storage-testbench.

- Set GCSMOCK_ENDPOINT without /storage/v1/, as the rest of the URL is set in tests.

- Show errors in case of failure to create bucket.

- Consolidate StorageDriverMock structs inside the pkg/test/mocks package.

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Steven Marks <steve.marks@qomodo.io>

.github/actions/setup-gcs-storage-testbench/action.yaml

@@ -0,0 +1,10 @@
+name: 'Setup storage-testbench'
+description: 'Download & run Google Cloud Storage testbench container'
+runs:
+  using: "composite"
+  steps:
+    - shell: bash
+      run: |
+        docker run -d --name storage-testbench -p 9000:9000 -p 9090:9090 ghcr.io/project-zot/ci-images/gcs-storage-testbench:v0.60.0-zot
+        echo "Waiting for storage-testbench..."
+        timeout 30s bash -c 'until curl -s http://localhost:9000/storage/v1/b; do sleep 1; done'

.github/actions/teardown-gcs-storage-testbench/action.yaml

@@ -0,0 +1,8 @@
+name: 'Teardown storage-testbench'
+description: 'Stop & remove storage-testbench container'
+runs:
+  using: "composite"
+  steps:
+    - if: always()
+      shell: bash
+      run: docker rm -f storage-testbench || true

.github/workflows/test.yaml

@@ -148,13 +148,23 @@ jobs:
         run: |
             cd $GITHUB_WORKSPACE
             go mod download
+      - uses: ./.github/actions/setup-gcs-storage-testbench
       - name: run zot privileged tests
-        run: sudo env "PATH=$PATH" make privileged-test
+        run: >
+          sudo env
+          "PATH=$PATH"
+          "GCSMOCK_ENDPOINT=$GCSMOCK_ENDPOINT"
+          "STORAGE_EMULATOR_HOST=$STORAGE_EMULATOR_HOST"
+          make privileged-test
+        env:
+          GCSMOCK_ENDPOINT: http://localhost:9000/
+          STORAGE_EMULATOR_HOST: localhost:9000
       - name: upload coverage
         uses: actions/upload-artifact@v6
         with:
           name: coverage-needprivileges
-          path: coverage-needprivileges.txt
+          path: coverage-needprivileges-*.txt
+      - uses: ./.github/actions/teardown-gcs-storage-testbench
   test-coverage:
     name: Collect all test coverage
     runs-on: ubuntu-latest