* chore(metadb): add writer version to interface
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(metadb): add writer version to db mock
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(metadb): implement writer version for bolt, redis, and dynamodb
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* feat(metadb): add optional fast restart path that skips storage walk when binary identity matches metaDB stamp
binary identity is determined by the current release tag/commit and stored in metaDB after a successful storage parse. When fast restart is enabled, the next startup will skip the parse if the stored identity matches the current binary
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(cli): serve: add a way to force reparse storage
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* refactor(meta): version: split to avoid global state mutation in tests
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): version: include commit in writerVersion to distinguish retags
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(config): add IsFastRestartEnabled() test
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): skip writer-version stamp when storage parse is incomplete
ParseStorage returns nil even when individual repos fail to parse or are only partially parsed (a missing manifest blob), so MaybeParseStorage would stamp a partially-populated metaDB as good. On the next restart fastRestart trusts the stamp, skips the storage walk, and never recovers.
Track per-repo outcomes via parseStats and stamp only when the walk fully populated the metaDB, otherwise log and continue so the next restart reparses
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(docs): readme: remove trailing comma from JSON config
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): dynamodb: use context.Background instead of context.TODO
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(meta): invalidate fast restart on storage config changes
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore(meta): dynamodb: use context.Background() instead of context.TODO()
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* docs(meta): dynamodb: add comment about nil AttributeValue handling in GetWriterVersion
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* chore: rename writer-version stamp to fast-restart stamp
also replaces the version/commit tracking to use BinaryVersion instead of WriterVersion
This should make things more clear
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(config): ensure FastRestart is on GlobalStorageConfig
This is not a per-subpath setting
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
* fix(metadb): redis: tests: ensure clients are closed
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
---------
Signed-off-by: Jacob McSwain <jacob@mcswain.dev>
Adds a configurable maximum repository count per registry instance.
When maxRepos is set on StorageConfig, manifest pushes that would create
a new repository beyond the limit are rejected with HTTP 429
TOOMANYREQUESTS. Pushes to existing repositories are always allowed.
Implemented as an always-available feature in pkg/api (not a build-tag
extension). MaxRepos is a field on StorageConfig, enabled when > 0.
- repoQuotaMiddleware on the dist-spec router intercepts manifest PUTs.
New-repo pushes are serialized with a sync.Mutex to prevent concurrent
requests from exceeding the limit.
- Adds CountRepos(ctx) to the MetaDB interface with efficient
implementations: BoltDB (Stats().KeyN), Redis (HLen), DynamoDB
(Scan with Select=COUNT).
- Config.IsQuotaEnabled() added, wired into controller.go metaDB init.
- Four integration tests (enforcement, concurrency, disabled,
unconfigured) and backend-specific CountRepos tests for BoltDB, Redis,
and DynamoDB.
Signed-off-by: Bachir Khiati <bachir.khiati@gmail.com>
1. Parse repos without metadata in ParseStorage
The timestamp check in ParseStorage was skipping repos that exist in
storage but don't have metadata. When GetRepoLastUpdated returns zero
time (no metadata), we should always parse the repo to create its
metadata. Check if metaLastUpdated is zero before comparing timestamps.
If zero, always parse regardless of storageLastUpdated.
2. Change the logic of how LastUpdated is computed in RepoSummary
It is not the latest tagged timestamp from the available images or
the last updated image created timestamp, based on whichever is the
latest.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
feat(meta): add TaggedTimestamp field and preserve during re-parsing
Add TaggedTimestamp field to track when image tags were created, exposed
through GraphQL API. Previously, when zot restarted and re-parsed storage,
ResetRepoReferences would clear all tags, causing timestamp information to
be lost and reset to the service restart time for existing images.
This change adds TaggedTimestamp support and modifies ResetRepoReferences to
selectively preserve tags that still exist in storage, maintaining their
TaggedTimestamp values. Tags that no longer exist in storage are removed as
before.
Changes:
- Add TaggedTimestamp field to GraphQL ImageSummary schema
- Update GraphQL conversion functions to populate TaggedTimestamp with
fallback to PushTimestamp when unavailable
- Updated ResetRepoReferences interface to accept tagsToKeep parameter
- Modified ParseRepo to collect tags from storage before resetting
- Updated all backend implementations (Redis, DynamoDB, BoltDB) to preserve
tags in tagsToKeep instead of clearing all tags
- Updated tests and mocks to match new signature
This ensures TaggedTimestamp accurately reflects when tags were originally
created, and exposes this information through the GraphQL API.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
fix: make sure metadb statistics are initialized on image download, and minor metadb fixes for Docker v2 manifest compatibility
Looking into potential causes of https://github.com/project-zot/zot/issues/3163
1. One possible reason is the statistics were not properly initialized in the first place because of (unknown and/or unavoidable) errors on image push.
To workaround this add logic to initialize the statistics on the call to download them.
2. Some images have the download statistics while others dont, one cause could be a bug in the logic handling manifest mediatypes in the search extension.
Add compatibility checks for Docker v2 manifest types in metadb convert functions, and more tests for covering the Docker mediatype use case.
Side fixes:
- Ensure PushedBy Statistics entries are properly initialized in SetRepoReference
- Fix and issue in the image upload test functions, they were uploading docker images with oci mediatypes in call headers
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
GC and scrub should not stop if a manifest or index is missing from storage.
Other similar changes are also included.
WRT metadb, the missing manifests cannot be added, and the results returned from metadb
do not include the descriptors for these manifests.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: migrate from github.com/rs/zerolog to golang-native log/slog
We have been using zerolog for a really long time.
golang now has structured logging using slog.
Best to move to this in interests of long-term support.
This is a tech debt item.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: a few changes on top
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: address comments
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* feat: add redis cache support
https://github.com/project-zot/zot/pull/2005
Fixes https://github.com/project-zot/zot/issues/2004
* feat: add redis cache support
Currently, we have dynamoDB as the remote shared cache but ideal only
for the cloud use case.
For on-prem use case, add support for redis.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat(redis): added blackbox tests for redis
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* feat(redis): dummy implementation of MetaDB interface for redis cache
Signed-off-by: Alexei Dodon <adodon@cisco.com>
* feat: check validity of driver configuration on metadb instantiation
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat: multiple fixes for redis cache driver implementation
- add missing method GetAllBlobs
- add redis cache tests, with and without mocking
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): redis implementation for MetaDB
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): use redsync to block concurrent write access to the redis DB
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): update .github/workflows/cluster.yaml to also test redis
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(metadb): add keyPrefix parameter for redis and remove unneeded method meta.Crate()
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): support RedisCluster configuration and add unit tests
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): more tests for redis metadb implementation
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): add more examples and update examples/README.md
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): move option parsing and redis client initialization under pkg/api/config/redis
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* chore(cachedb): move Cache interface to pkg/storage/types
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): reorganize code in pkg/storage/cache.go
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): call redis.SetLogger() with the zot logger as parameter
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): rename pkg/meta/redisdb to pkg/meta/redis
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Alexei Dodon <adodon@cisco.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: a <a@tuxpa.in>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alexei Dodon <adodon@cisco.com>
Jacob McSwain