Replace MakeTempFile usage with MakeTempFilePath and MakeTempFileWithContent
helpers that automatically handle file lifecycle. This prevents resource
leaks by ensuring temporary files are properly closed.
Shoudld also make the tests easier to read.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
feat: add support for sha256 and sha512 htpasswd formats
Fixes issue #3495
We currently support only bcrypt htpasswd hashes, however bcrypt is not
FIPS-140 approved since it uses Blowfish.
This PR adds support for sha256 and sha512 formats and enforces that
bcrypt be disabled when fips140 mode is enabled.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: migrate from github.com/rs/zerolog to golang-native log/slog
We have been using zerolog for a really long time.
golang now has structured logging using slog.
Best to move to this in interests of long-term support.
This is a tech debt item.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: a few changes on top
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: address comments
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* feat: add redis cache support
https://github.com/project-zot/zot/pull/2005
Fixes https://github.com/project-zot/zot/issues/2004
* feat: add redis cache support
Currently, we have dynamoDB as the remote shared cache but ideal only
for the cloud use case.
For on-prem use case, add support for redis.
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat(redis): added blackbox tests for redis
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* feat(redis): dummy implementation of MetaDB interface for redis cache
Signed-off-by: Alexei Dodon <adodon@cisco.com>
* feat: check validity of driver configuration on metadb instantiation
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat: multiple fixes for redis cache driver implementation
- add missing method GetAllBlobs
- add redis cache tests, with and without mocking
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): redis implementation for MetaDB
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): use redsync to block concurrent write access to the redis DB
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): update .github/workflows/cluster.yaml to also test redis
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(metadb): add keyPrefix parameter for redis and remove unneeded method meta.Crate()
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): support RedisCluster configuration and add unit tests
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): more tests for redis metadb implementation
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): add more examples and update examples/README.md
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): move option parsing and redis client initialization under pkg/api/config/redis
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* chore(cachedb): move Cache interface to pkg/storage/types
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): reorganize code in pkg/storage/cache.go
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): call redis.SetLogger() with the zot logger as parameter
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(redis): rename pkg/meta/redisdb to pkg/meta/redis
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Alexei Dodon <adodon@cisco.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: a <a@tuxpa.in>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alexei Dodon <adodon@cisco.com>
* feat: add support for docker images
Issue #724
A new config section under "HTTP" called "Compat" is added which
currently takes a list of possible compatible legacy media-types.
https://github.com/opencontainers/image-spec/blob/main/media-types.md#compatibility-matrix
Only "docker2s2" (Docker Manifest V2 Schema V2) is currently supported.
Garbage collection also needs to be made aware of non-OCI compatible
layer types.
feat: add cve support for non-OCI compatible layer types
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
*
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* test: add more docker compat tests
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat: add additional validation checks for non-OCI images
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* ci: make "full" images docker-compatible
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* fix(scheduler): data race when pushing new tasks
the problem here is that scheduler can be closed in two ways:
- canceling the context given as argument to scheduler.RunScheduler()
- running scheduler.Shutdown()
because of this shutdown can trigger a data race between calling scheduler.inShutdown()
and actually pushing tasks into the pool workers
solved that by keeping a quit channel and listening on both quit channel and ctx.Done()
and closing the worker chan and scheduler afterwards.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* refactor(scheduler): refactor into a single shutdown
before this we could stop scheduler either by closing the context
provided to RunScheduler(ctx) or by running Shutdown().
simplify things by getting rid of the external context in RunScheduler().
keep an internal context in the scheduler itself and pass it down to all tasks.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
---------
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
test "zotregistry.io/zot/pkg/test/common" - in tests
. "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.
2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
. "zotregistry.io/zot/pkg/test/image-utils"
3. "zotregistry.io/zot/pkg/test/deprecated" - curently used as
"zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.
4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
"zotregistry.io/zot/pkg/test/signature"
5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc) curently used as
authutils "zotregistry.io/zot/pkg/test/auth"
6. "zotregistry.io/zot/pkg/test/oci-utils" - curently used as
ociutils "zotregistry.io/zot/pkg/test/oci-utils"
Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.
Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- using secrets manager for storing public keys and certificates
- adding a default truststore for notation verification and upload all certificates to this default truststore
- removig `truststoreName` query param from notation api for uploading certificates
(cherry picked from commit eafcc1a213)
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types
refactor(gc): drop umoci dependency, implemented internal gc
added retentionDelay config option that specifies
the garbage collect delay for images without tags
this will also clean manifests which are part of an index image
(multiarch) that no longer exist.
fix(dedupe): skip blobs under .sync/ directory
if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage
fix(storage): do not allow image indexes modifications
when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* fix: remove inline GC and set a default value of gc interval
- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
* ci: add test for gc with short interval
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
---------
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved - see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey
mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled
userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)
apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys
other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Andrei Aaron