feat(storage): add a GCS driver
test(storage): add unit tests for GCS driver
test(storage): add missing unit tests for GCS driver & resolve lint issues
fix: configuration validation for GCS Storage
test(storage): resolve panic by test due to setupGCS ignoring returned error
test(storage): add dummy gcs credentials
test: add darwin support for macos to run tests
ci: update workflows to pin gcs emulator version
lint: resolve long line lengths & formatting issues
test: move error for gcs mock earlier with an error
test: stop test using local google credentials and use mock instead
test: add missing dummy creds
test(storage): use storage-testbench for GCS, isolate GCS tests, fix driver Delete
- Switch GCS emulator from fake-gcs-server to storage-testbench in CI.
Run the GCS emulator only in the privileged-test job; remove it from
minimal and extended test jobs.
- Consolidate GCS tests under pkg/storage/gcs (needprivileges,linux).
Add TestMain with HTTPS proxy and /etc/hosts so tests talk to
storage-testbench; move GCS-specific cases from storage_test.go and
scrub_test.go into gcs_test.go. Run GCS tests via a second privileged-test
invocation and collect coverage in coverage-needprivileges-gcs.txt.
- Make GCS driver Delete idempotent and normalize errors. Treat
PathNotFoundError from Delete as success so that deleting an already-gone
path (e.g. after GC under eventual consistency) does not fail. Add
formatErr to map 404/not found to PathNotFoundError and use it for all
driver methods so callers get consistent storage driver errors.
- Drop GCS branches and helpers from storage_test.go and scrub_test.go so
non-privileged tests only use local/S3; GCS is tested only in
pkg/storage/gcs with storage-testbench.
- Set GCSMOCK_ENDPOINT without /storage/v1/, as the rest of the URL is set in tests.
- Show errors in case of failure to create bucket.
- Consolidate StorageDriverMock structs inside the pkg/test/mocks package.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Steven Marks <steve.marks@qomodo.io>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: migrate from github.com/rs/zerolog to golang-native log/slog
We have been using zerolog for a really long time.
golang now has structured logging using slog.
Best to move to this in interests of long-term support.
This is a tech debt item.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: a few changes on top
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: address comments
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* feat: add support for docker images
Issue #724
A new config section under "HTTP" called "Compat" is added which
currently takes a list of possible compatible legacy media-types.
https://github.com/opencontainers/image-spec/blob/main/media-types.md#compatibility-matrix
Only "docker2s2" (Docker Manifest V2 Schema V2) is currently supported.
Garbage collection also needs to be made aware of non-OCI compatible
layer types.
feat: add cve support for non-OCI compatible layer types
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
*
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* test: add more docker compat tests
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat: add additional validation checks for non-OCI images
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* ci: make "full" images docker-compatible
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types
refactor(gc): drop umoci dependency, implemented internal gc
added retentionDelay config option that specifies
the garbage collect delay for images without tags
this will also clean manifests which are part of an index image
(multiarch) that no longer exist.
fix(dedupe): skip blobs under .sync/ directory
if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage
fix(storage): do not allow image indexes modifications
when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Moved boltdb to a driver implementation for such interface
Added CreateCacheDatabaseDriver in controller
Fixed default directory creation (boltDB will only create the file, not the dir
Added coverage tests
Added example config for boltdb
Re-added caching on subpaths, rewrote CreateCacheDatabaseDriver
Fix tests
Made cacheDriver argument mandatory for NewImageStore, added more validation, added defaults
Moved cache interface to own file, removed useRelPaths from config
Got rid of cache config, refactored
Moved cache to own package and folder
Renamed + removed cache factory to backend, replaced CloudCache to RemoteCache
Moved storage constants back to storage package
moved cache interface and factory to storage package, changed remoteCache defaulting
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
- Digests were represented by different ways
- We needed a uniform way to represent the digests and enforce a format
- also replace usage of github.com/google/go-containerregistry/pkg/v1
with github.com/opencontainers/image-spec/specs-go/v1
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)
Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Andrei Aaron