mirror of
https://github.com/project-zot/zot.git
synced 2026-06-17 21:17:58 +08:00
main
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ramkumar Chinchani
|
55b68228da |
feat(storage): redirect blob pulls to backend URLs (#4092)
* feat(storage): redirect blob pulls to backend URLs * fix: rebase conflicts Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * refactor: rename redirect field Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * test: relax brittle TestPeriodicGC substore log assertion Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * feat(storage): improve blob redirect config handling and validation Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix(storage): address PR review feedback for blob redirect Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * feat(storage): apply latest PR review fixes for blob redirect Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * test: fix blob redirect and verify test regressions Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix(storage): enforce redirectBlobURL validation and add redirect tests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix(storage): fix err113/noctx lint errors in storage driver tests - Replace httptest.NewRequest with httptest.NewRequestWithContext in s3, gcs, and imagestore driver tests (noctx) - Replace dynamic errors.New in s3 driver test with a package-level static sentinel error (err113) Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * test(storage): use temp dirs in imagestore redirect tests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix: handle ranged blob redirects and add regression tests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix: validate blob digest consistently in GetBlob Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * test: fix GetBlobPartialFn mock return values for range requests The test 'does not redirect ranged blob requests' was failing because the mock was returning incorrect length values. For a range request 'bytes=0-0' (1 byte), it was returning 4 bytes, which caused a length mismatch check in GetBlob to return HTTP 500. Fix the mock to dynamically calculate the correct length: to - from + 1 Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix(storage): preserve signed URL bytes in normalizeBlobRedirectURL Preserve the original URL bytes from backend storage drivers (important for signed/presigned URLs) while only lowercasing the scheme prefix. URL re-serialization via net/url can invalidate signatures through path escaping or canonicalization. Add regression tests covering signed URL query parameters and mixed-case scheme handling. Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix(storage): address PR review comments for blob redirect - Return signed redirect URLs unchanged; validate scheme/CRLF/host only, no URL normalization that would corrupt signed URL bytes - Add inline comments for all non-obvious decisions: range bypass, soft fallback on invalid URL, local driver empty return, subpath resolution, redirectBlobURL config constraint on local/empty driver - Expand TestNormalizeBlobRedirectURL to cover allowed schemes (http/https), parse failure, missing host, and CRLF injection cases - Add TestIsBlobRedirectEnabled covering subpath-only enablement with default store disabled Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * test(storage): address remaining blob redirect review comments Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * fix: gofumpt formatting in routes_test.go Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> --------- Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> Co-authored-by: Akash Kumar <meakash7902@gmail.com> |
||
|
Benoit Tigeot
|
6c1f1bdd40 |
feat(metrics): add Prometheus GC metrics (#3863)
* feat(metrics): add Prometheus GC metrics Track garbage collection activity with three new metrics: - zot_gc_runs_total (counter, label: error) — GC run count - zot_gc_duration_seconds (summary) — GC run duration - zot_gc_deleted_total (counter, label: type) — items deleted by type: blob, manifest, upload MetricServer is added to GarbageCollect and wired through all callers (controller, verify-feature retention, tests). Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * fix(test): add missing metrics var in GCS GC tests TestGCSGarbageCollectImageIndex and TestGCSGarbageCollectChainedImageIndexes were missing the metrics variable required by NewGarbageCollect after the MetricServer parameter was added. Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * fix(test): add defer metrics.Stop() in GC tests Prevent goroutine/port leaks by stopping MetricsServer in storage_test.go (3 functions) and gcs_test.go (also add missing metrics declaration in TestGCSGarbageCollectImageManifest). Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * fix(test): cover `CleanRepo` error path Add test that exercises the error branch in `CleanRepo` where `cleanRepo` fails, covering the metrics calls and log lines flagged by Codecov. Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * test: Cover GC error paths for codecov Add three tests in gc_internal_test.go to cover previously untested error branches in `removeBlobUploads` and `removeUnreferencedBlobs`: `ListBlobUploads` failure, `addIndexBlobsToReferences` failure, and `PathNotFoundError` from `GetAllBlobs`. Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * test(gc): cover remaining error paths Cover `StatBlobUpload`, `digest.Validate()`, `isBlobOlderThan`, and `CleanupRepo` error branches in `removeBlobUploads` and `removeUnreferencedBlobs`. `removeUnreferencedBlobs` now at 100% coverage, `removeBlobUploads` from 78.3% to 91.3%. Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * test: cover `sanityChecks` label name mismatch Try to avoid -0.09% coverage regression on `minimal.go` by exercising the uncovered branch in `sanityChecks` where label names have correct count but wrong values. Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * test(gc): exercise real GC path in metrics test TestGCMetrics was calling metric helpers directly instead of running actual garbage collection, so it couldn't catch wiring regressions where `CleanRepo` stops recording metrics. Now uploads an orphaned blob and runs `gc.CleanRepo` end-to-end, verifying metrics appear on the Prometheus endpoint. Suggestion from Copilot: https://github.com/project-zot/zot/pull/3863#discussion_r3129324719 Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * fix(gc): skip deletion metrics when DryRun is enabled https://github.com/project-zot/zot/pull/3863#discussion_r3129324684 Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * fix(test): stop leaked MetricsServer goroutines in GCS tests https://github.com/project-zot/zot/pull/3863#discussion_r3129324657 Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * refactor(test): drop unnecessary zlog import alias Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * fix(monitoring): expose metric types outside build tag `MetricsCopy` and related types were only visible under `\!metrics`, causing a typecheck failure when golangci-lint runs with `-tags metrics`. Moving the type definitions to `common.go` makes them unconditionally available. Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * fix(monitoring): remove extra blank line for gci Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * test(gc): cover both dry-run and real deletion metrics And fix issue with build tag with metrics Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> * Satisfy testpackage linter for gc metrics test The `testpackage` linter allows `package gc` only in files named `*_internal_test.go`; rename to follow that convention. Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> --------- Signed-off-by: Benoit Tigeot <benoit.tigeot@lifen.fr> |
||
|
Ramkumar Chinchani
|
9aff5b8d08 |
chore: fix dependabot alerts (#4048)
* chore: fix dependabot alerts Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: fix dependabot alerts Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: fix dependabot alerts Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: fix golangci-lint findings from CI Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: fix golangci-lint gosec warnings Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: update code to use slices package and address gosec linting issues Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * build: fix makefile target Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: update tests to use context in HTTP requests and add gosec annotations Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: update tests to use context in HTTP requests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: update tests to use context in HTTP requests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: update tests to use context in HTTP requests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: update tests to use context in HTTP requests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: bump zui version Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: update test helpers and improve security settings in tests Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> * chore: add gosec linting directive for test path construction Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> --------- Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com> |
||
|
Sebastian Thees
|
e188f45890 |
fix(storage/gcs): fix double-prefixed rootdirectory and EOF handling in Walk for GCS (#3903)
* fix(storage): resolve double-prefixing issue for GCS rootdirectory Preserve double-prefixing for S3 to maintain backward compatibility with existing data. For GCS, always use "/" as rootDir to avoid double-prefixing, as GCS rootdirectory usage is a newer feature without legacy data. Signed-off-by: Sebastian Thees <thees@users.noreply.github.com> * fix(gcs): handle io.EOF correctly in Walk method Ensure io.EOF is returned unwrapped to allow proper error handling with errors.Is() upstream. Signed-off-by: Sebastian Thees <thees@users.noreply.github.com> * fix(storage): set sensible default ("/zot") for GCS when storageDriver.rootdirectory is unset or empty or "/" Signed-off-by: Sebastian Thees <thees@users.noreply.github.com> * fix(imagestore): avoid warning logs for expected cache miss scenarios Refine logging to use debug level for expected cache misses, preventing unnecessary warnings. Signed-off-by: Sebastian Thees <thees@users.noreply.github.com> --------- Signed-off-by: Sebastian Thees <thees@users.noreply.github.com> |
||
|
Andrei Aaron
|
a5cc8ab810 |
feat: support pushing multiple tags for a single manifest (#3885)
* feat: support pushing multiple tags for a single manifest See https://github.com/opencontainers/distribution-spec/pull/600 Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com> * fix: constants not replaced in swagger output Also godot mandates comments ending in dots, which produces bad results in the swagger generated files, see the extra ". which is now fixed below: ``` diff --git a/swagger/docs.go b/swagger/docs.go index 84b08277..fb2c45c3 100644 --- a/swagger/docs.go +++ b/swagger/docs.go @@ -114,7 +114,7 @@ const docTemplate = `{ } }, "400": { - "description": "bad request\".", + "description": "bad request", "schema": { "type": "string" } @@ -200,7 +200,7 @@ const docTemplate = `{ } }, "400": { - "description": "bad request\".", + "description": "bad request", "schema": { "type": "string" } diff --git a/swagger/swagger.json b/swagger/swagger.json index cfeb3900..247f95fa 100644 --- a/swagger/swagger.json +++ b/swagger/swagger.json @@ -106,7 +106,7 @@ } }, "400": { - "description": "bad request\".", + "description": "bad request", "schema": { "type": "string" } @@ -192,7 +192,7 @@ } }, "400": { - "description": "bad request\".", + "description": "bad request", "schema": { "type": "string" } diff --git a/swagger/swagger.yaml b/swagger/swagger.yaml index 57641c2f..09b30dcc 100644 --- a/swagger/swagger.yaml +++ b/swagger/swagger.yaml @@ -310,7 +310,7 @@ paths: schema: type: string "400": - description: bad request". + description: bad request schema: type: string "500": @@ -366,7 +366,7 @@ paths: schema: type: string "400": - description: bad request". + description: bad request schema: type: string "500": ``` Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com> --------- Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com> |
||
|
Andrei Aaron
|
5e57656bff |
GCS storage support (#3798)
feat(storage): add a GCS driver test(storage): add unit tests for GCS driver test(storage): add missing unit tests for GCS driver & resolve lint issues fix: configuration validation for GCS Storage test(storage): resolve panic by test due to setupGCS ignoring returned error test(storage): add dummy gcs credentials test: add darwin support for macos to run tests ci: update workflows to pin gcs emulator version lint: resolve long line lengths & formatting issues test: move error for gcs mock earlier with an error test: stop test using local google credentials and use mock instead test: add missing dummy creds test(storage): use storage-testbench for GCS, isolate GCS tests, fix driver Delete - Switch GCS emulator from fake-gcs-server to storage-testbench in CI. Run the GCS emulator only in the privileged-test job; remove it from minimal and extended test jobs. - Consolidate GCS tests under pkg/storage/gcs (needprivileges,linux). Add TestMain with HTTPS proxy and /etc/hosts so tests talk to storage-testbench; move GCS-specific cases from storage_test.go and scrub_test.go into gcs_test.go. Run GCS tests via a second privileged-test invocation and collect coverage in coverage-needprivileges-gcs.txt. - Make GCS driver Delete idempotent and normalize errors. Treat PathNotFoundError from Delete as success so that deleting an already-gone path (e.g. after GC under eventual consistency) does not fail. Add formatErr to map 404/not found to PathNotFoundError and use it for all driver methods so callers get consistent storage driver errors. - Drop GCS branches and helpers from storage_test.go and scrub_test.go so non-privileged tests only use local/S3; GCS is tested only in pkg/storage/gcs with storage-testbench. - Set GCSMOCK_ENDPOINT without /storage/v1/, as the rest of the URL is set in tests. - Show errors in case of failure to create bucket. - Consolidate StorageDriverMock structs inside the pkg/test/mocks package. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com> Co-authored-by: Steven Marks <steve.marks@qomodo.io> |