* chore(metadb): add writer version to interface
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(metadb): add writer version to db mock
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(metadb): implement writer version for bolt, redis, and dynamodb
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* feat(metadb): add optional fast restart path that skips storage walk when binary identity matches metaDB stamp
binary identity is determined by the current release tag/commit and stored in metaDB after a successful storage parse. When fast restart is enabled, the next startup will skip the parse if the stored identity matches the current binary
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(cli): serve: add a way to force reparse storage
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* refactor(meta): version: split to avoid global state mutation in tests
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): version: include commit in writerVersion to distinguish retags
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(config): add IsFastRestartEnabled() test
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): skip writer-version stamp when storage parse is incomplete
ParseStorage returns nil even when individual repos fail to parse or are only partially parsed (a missing manifest blob), so MaybeParseStorage would stamp a partially-populated metaDB as good. On the next restart fastRestart trusts the stamp, skips the storage walk, and never recovers.
Track per-repo outcomes via parseStats and stamp only when the walk fully populated the metaDB, otherwise log and continue so the next restart reparses
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(docs): readme: remove trailing comma from JSON config
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): dynamodb: use context.Background instead of context.TODO
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): invalidate fast restart on storage config changes
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(meta): dynamodb: use context.Background() instead of context.TODO()
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* docs(meta): dynamodb: add comment about nil AttributeValue handling in GetWriterVersion
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore: rename writer-version stamp to fast-restart stamp
also replaces the version/commit tracking to use BinaryVersion instead of WriterVersion
This should make things more clear
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(config): ensure FastRestart is on GlobalStorageConfig
This is not a per-subpath setting
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(metadb): redis: tests: ensure clients are closed
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
---------
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
Adds a configurable maximum repository count per registry instance.
When maxRepos is set on StorageConfig, manifest pushes that would create
a new repository beyond the limit are rejected with HTTP 429
TOOMANYREQUESTS. Pushes to existing repositories are always allowed.
Implemented as an always-available feature in pkg/api (not a build-tag
extension). MaxRepos is a field on StorageConfig, enabled when > 0.
- repoQuotaMiddleware on the dist-spec router intercepts manifest PUTs.
New-repo pushes are serialized with a sync.Mutex to prevent concurrent
requests from exceeding the limit.
- Adds CountRepos(ctx) to the MetaDB interface with efficient
implementations: BoltDB (Stats().KeyN), Redis (HLen), DynamoDB
(Scan with Select=COUNT).
- Config.IsQuotaEnabled() added, wired into controller.go metaDB init.
- Four integration tests (enforcement, concurrency, disabled,
unconfigured) and backend-specific CountRepos tests for BoltDB, Redis,
and DynamoDB.
Signed-off-by: Bachir Khiati <bachir.khiati@gmail.com>
1. Parse repos without metadata in ParseStorage
The timestamp check in ParseStorage was skipping repos that exist in
storage but don't have metadata. When GetRepoLastUpdated returns zero
time (no metadata), we should always parse the repo to create its
metadata. Check if metaLastUpdated is zero before comparing timestamps.
If zero, always parse regardless of storageLastUpdated.
2. Change the logic of how LastUpdated is computed in RepoSummary
It is not the latest tagged timestamp from the available images or
the last updated image created timestamp, based on whichever is the
latest.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
feat(meta): add TaggedTimestamp field and preserve during re-parsing
Add TaggedTimestamp field to track when image tags were created, exposed
through GraphQL API. Previously, when zot restarted and re-parsed storage,
ResetRepoReferences would clear all tags, causing timestamp information to
be lost and reset to the service restart time for existing images.
This change adds TaggedTimestamp support and modifies ResetRepoReferences to
selectively preserve tags that still exist in storage, maintaining their
TaggedTimestamp values. Tags that no longer exist in storage are removed as
before.
Changes:
- Add TaggedTimestamp field to GraphQL ImageSummary schema
- Update GraphQL conversion functions to populate TaggedTimestamp with
fallback to PushTimestamp when unavailable
- Updated ResetRepoReferences interface to accept tagsToKeep parameter
- Modified ParseRepo to collect tags from storage before resetting
- Updated all backend implementations (Redis, DynamoDB, BoltDB) to preserve
tags in tagsToKeep instead of clearing all tags
- Updated tests and mocks to match new signature
This ensures TaggedTimestamp accurately reflects when tags were originally
created, and exposes this information through the GraphQL API.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
fix: make sure metadb statistics are initialized on image download, and minor metadb fixes for Docker v2 manifest compatibility
Looking into potential causes of https://github.com/project-zot/zot/issues/3163
1. One possible reason is the statistics were not properly initialized in the first place because of (unknown and/or unavoidable) errors on image push.
To workaround this add logic to initialize the statistics on the call to download them.
2. Some images have the download statistics while others dont, one cause could be a bug in the logic handling manifest mediatypes in the search extension.
Add compatibility checks for Docker v2 manifest types in metadb convert functions, and more tests for covering the Docker mediatype use case.
Side fixes:
- Ensure PushedBy Statistics entries are properly initialized in SetRepoReference
- Fix and issue in the image upload test functions, they were uploading docker images with oci mediatypes in call headers
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
GC and scrub should not stop if a manifest or index is missing from storage.
Other similar changes are also included.
WRT metadb, the missing manifests cannot be added, and the results returned from metadb
do not include the descriptors for these manifests.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: migrate from github.com/rs/zerolog to golang-native log/slog
We have been using zerolog for a really long time.
golang now has structured logging using slog.
Best to move to this in interests of long-term support.
This is a tech debt item.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: a few changes on top
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: address comments
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
when a UI client tries to view image details
for an image with multiple tags pointing to the same digest
we make a query to dynamodb having duplicate keys (same digest)
resulting in an error and the client is redirect back to image
overview.
closes: #2464
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
wait for workers to finish before exiting
should fix tests reporting they couldn't remove rootDir because it's being
written by tasks
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- MetaDB stores the time of the last update of a repo
- During startup we check if the layout has been updated after the last recorded change in the db
- If this is the case, the repo is parsed and updated in the DB otherwise it's skipped
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
test "zotregistry.io/zot/pkg/test/common" - in tests
. "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.
2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
. "zotregistry.io/zot/pkg/test/image-utils"
3. "zotregistry.io/zot/pkg/test/deprecated" - curently used as
"zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.
4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
"zotregistry.io/zot/pkg/test/signature"
5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc) curently used as
authutils "zotregistry.io/zot/pkg/test/auth"
6. "zotregistry.io/zot/pkg/test/oci-utils" - curently used as
ociutils "zotregistry.io/zot/pkg/test/oci-utils"
Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.
Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- using secrets manager for storing public keys and certificates
- adding a default truststore for notation verification and upload all certificates to this default truststore
- removig `truststoreName` query param from notation api for uploading certificates
(cherry picked from commit eafcc1a213)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
fix(authz): fix isAdmin not using groups to determine if a user is admin.
fix(authz): return 401 instead of 403
403 is correct as per HTTP spec
However authz is not part of dist-spec and clients know only about 401
So this is a compromise.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Jacob McSwain