* fix(storage): resolve double-prefixing issue for GCS rootdirectory
Preserve double-prefixing for S3 to maintain backward compatibility with existing data. For GCS, always use "/" as rootDir to avoid double-prefixing, as GCS rootdirectory usage is a newer feature without legacy data.
Signed-off-by: Sebastian Thees <thees@users.noreply.github.com>
* fix(gcs): handle io.EOF correctly in Walk method
Ensure io.EOF is returned unwrapped to allow proper error handling with errors.Is() upstream.
Signed-off-by: Sebastian Thees <thees@users.noreply.github.com>
* fix(storage): set sensible default ("/zot") for GCS when storageDriver.rootdirectory is unset or empty or "/"
Signed-off-by: Sebastian Thees <thees@users.noreply.github.com>
* fix(imagestore): avoid warning logs for expected cache miss scenarios
Refine logging to use debug level for expected cache misses, preventing unnecessary warnings.
Signed-off-by: Sebastian Thees <thees@users.noreply.github.com>
---------
Signed-off-by: Sebastian Thees <thees@users.noreply.github.com>
feat(storage): add a GCS driver
test(storage): add unit tests for GCS driver
test(storage): add missing unit tests for GCS driver & resolve lint issues
fix: configuration validation for GCS Storage
test(storage): resolve panic by test due to setupGCS ignoring returned error
test(storage): add dummy gcs credentials
test: add darwin support for macos to run tests
ci: update workflows to pin gcs emulator version
lint: resolve long line lengths & formatting issues
test: move error for gcs mock earlier with an error
test: stop test using local google credentials and use mock instead
test: add missing dummy creds
test(storage): use storage-testbench for GCS, isolate GCS tests, fix driver Delete
- Switch GCS emulator from fake-gcs-server to storage-testbench in CI.
Run the GCS emulator only in the privileged-test job; remove it from
minimal and extended test jobs.
- Consolidate GCS tests under pkg/storage/gcs (needprivileges,linux).
Add TestMain with HTTPS proxy and /etc/hosts so tests talk to
storage-testbench; move GCS-specific cases from storage_test.go and
scrub_test.go into gcs_test.go. Run GCS tests via a second privileged-test
invocation and collect coverage in coverage-needprivileges-gcs.txt.
- Make GCS driver Delete idempotent and normalize errors. Treat
PathNotFoundError from Delete as success so that deleting an already-gone
path (e.g. after GC under eventual consistency) does not fail. Add
formatErr to map 404/not found to PathNotFoundError and use it for all
driver methods so callers get consistent storage driver errors.
- Drop GCS branches and helpers from storage_test.go and scrub_test.go so
non-privileged tests only use local/S3; GCS is tested only in
pkg/storage/gcs with storage-testbench.
- Set GCSMOCK_ENDPOINT without /storage/v1/, as the rest of the URL is set in tests.
- Show errors in case of failure to create bucket.
- Consolidate StorageDriverMock structs inside the pkg/test/mocks package.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Steven Marks <steve.marks@qomodo.io>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* feat: add support for docker images
Issue #724
A new config section under "HTTP" called "Compat" is added which
currently takes a list of possible compatible legacy media-types.
https://github.com/opencontainers/image-spec/blob/main/media-types.md#compatibility-matrix
Only "docker2s2" (Docker Manifest V2 Schema V2) is currently supported.
Garbage collection also needs to be made aware of non-OCI compatible
layer types.
feat: add cve support for non-OCI compatible layer types
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
*
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* test: add more docker compat tests
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat: add additional validation checks for non-OCI images
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* ci: make "full" images docker-compatible
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
- Cosign supports 2 types of signature formats:
1. Using tag -> each new signature of the same manifest is
added as a new layer of the signature manifest having that
specific tag("{alghoritm}-{digest_of_signed_manifest}.sig")
2. Using referrers -> each new signature of the same manifest is
added as a new manifest
- For adding these cosign signature to metadb, we reserved index 0 of the
list of cosign signatures for tag-based signatures. When a new tag-based
signature is added for the same manifest, the element on first position
in its list of cosign signatures(in metadb) will be updated/overwritten.
When a new cosign signature(using referrers) will be added for the same
manifest this new signature will be appended to the list of cosign
signatures.
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types
refactor(gc): drop umoci dependency, implemented internal gc
added retentionDelay config option that specifies
the garbage collect delay for images without tags
this will also clean manifests which are part of an index image
(multiarch) that no longer exist.
fix(dedupe): skip blobs under .sync/ directory
if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage
fix(storage): do not allow image indexes modifications
when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Suppose we push two identical manifests (sharing same digest) but with
different tags, then deleting by digest should throw an error otherwise
we end up deleting all image tags (with gc) or dangling references
(without gc)
This behaviour is controlled via Authorization, added a new policy
action named detectManifestsCollision which enables this behaviour
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
- Digests were represented by different ways
- We needed a uniform way to represent the digests and enforce a format
- also replace usage of github.com/google/go-containerregistry/pkg/v1
with github.com/opencontainers/image-spec/specs-go/v1
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)
Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.
maintidx linter disabled
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
previously mount blob will look for blob that is provided in http request and try to hard link that path
but ideally we should look for path from our cache and do the hard link of that particular path.
this commit does the same.
added support to point multiple storage locations in zot by running multiple instance of zot in background.
see examples/config-multiple.json for more info about config.
Closes#181
In use cases, when there are large images with shared layers across
repositories, clients may benefit from not re-uploading the same blobs
over and over again.
We ensure this by hard linking when check-blob api is called.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
The storage layer is protected with read-write locks.
However, we may be holding the locks over unnecessarily large critical
sections.
The typical workflow is that a blob is first uploaded via a per-client
private session-id meaning the blob is not publicly visible yet. When
the blob being uploaded is very large, the transfer takes a long time
while holding the lock.
Private session-id based uploads don't really need locks, and hold locks
only when blobs are published after the upload is complete.
dstRecord :- blob path stored in cache.
dst :- blob path that is trying to be uploaded.
Currently, if the actual blob on disk may have been removed by GC/delete, during syncing the cache dst is being passed to DeleteBlob function and retry section is being continuously called because DeleteBlob function never deletes dst path (doesn't exist in db), dstRecord should be passed into DeleteBlob function because dstRecord is actual blob path stored in db.
If dst and dstRecord path value is same then this issue will not be produced and DeleteBlob method will delete the blob info from cache but if both are different then DeleteBlob method will try to delete dst path which is not in cache.
Note:- boltdb delete method return nil even when value doesn't exist (https://godoc.org/github.com/boltdb/bolt#Bucket.Delete)
We perform inline garbage collection of orphan blobs. However, the
dist-spec poses a problem because blobs begin their life as orphan blobs
and then a manifest is add which refers to these blobs.
We use umoci's GC() to perform garbage collection and policy support
has been added recently which can control whether a blob can be skipped
for GC.
In this patch, we use a time-based policy to skip blobs.
Go version changed to 1.14.4
Golangci-lint changed to 1.26.0
Bazel version changed to 3.0.0
Bazel rules_go version changed to 0.23.3
Bazel gazelle version changed to v0.21.0
Bazel build tools version changed to 0.25.1
Bazel skylib version changed to 1.0.2
Sebastian Thees