feat: Sync images with a background context
This means syncs/pulls will not be cancelled anymore when the requesting client disconnects.
The timeout used can be configured per registry
Signed-off-by: Lars Francke <git@lars-francke.de>
fix: make sure metadb statistics are initialized on image download, and minor metadb fixes for Docker v2 manifest compatibility
Looking into potential causes of https://github.com/project-zot/zot/issues/3163
1. One possible reason is the statistics were not properly initialized in the first place because of (unknown and/or unavoidable) errors on image push.
To workaround this add logic to initialize the statistics on the call to download them.
2. Some images have the download statistics while others dont, one cause could be a bug in the logic handling manifest mediatypes in the search extension.
Add compatibility checks for Docker v2 manifest types in metadb convert functions, and more tests for covering the Docker mediatype use case.
Side fixes:
- Ensure PushedBy Statistics entries are properly initialized in SetRepoReference
- Fix and issue in the image upload test functions, they were uploading docker images with oci mediatypes in call headers
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Old message example:
```
{"time":"2025-11-17T22:29:29.434073547Z","level":"debug","message":"image is not scannable","digest":"sha256:99155f2825186175c7f7e4df4f77d5d0ba68a018603afd9c014063d4ca11c0d6","mediaType":"application/vnd.oci.image.manifest.v1+json","error":"image 'nginx@sha256:99155f2825186175c7f7e4df4f77d5d0ba68a018603afd9c014063d4ca11c0d6' scanning is not supported for given image media type","caller":"zotregistry.dev/zot/v2/pkg/extensions/search/cve/cve.go:513","func":"zotregistry.dev/zot/v2/pkg/extensions/search/cve.BaseCveInfo.GetCVESummaryForImageMedia","goroutine":1246}
```
New message example:
```
{"time":"2025-11-17T22:25:10.33273421Z","level":"debug","message":"image is not scannable","digest":"sha256:99155f2825186175c7f7e4df4f77d5d0ba68a018603afd9c014063d4ca11c0d6","mediaType":"application/vnd.oci.image.manifest.v1+json","error":"image 'nginx@sha256:99155f2825186175c7f7e4df4f77d5d0ba68a018603afd9c014063d4ca11c0d6' scanning is not supported for given media type: layer media type 'application/octet-stream'","caller":"zotregistry.dev/zot/v2/pkg/extensions/search/cve/cve.go:513","func":"zotregistry.dev/zot/v2/pkg/extensions/search/cve.BaseCveInfo.GetCVESummaryForImageMedia","goroutine":1316}
```
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
GC and scrub should not stop if a manifest or index is missing from storage.
Other similar changes are also included.
WRT metadb, the missing manifests cannot be added, and the results returned from metadb
do not include the descriptors for these manifests.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
See: https://github.com/project-zot/zot/issues/2506
Note we are not loosing anything functionality-wise by making this change.
Initially we considered the tags are in the annotations present in the referrers
but the only annotations we set on referrers are the ones inside the manifests themselves,
not the ones in the manifest descriptors, so the tags were not presetn anyway.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
feat: add support for sha256 and sha512 htpasswd formats
Fixes issue #3495
We currently support only bcrypt htpasswd hashes, however bcrypt is not
FIPS-140 approved since it uses Blowfish.
This PR adds support for sha256 and sha512 formats and enforces that
bcrypt be disabled when fips140 mode is enabled.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
- Use custom authURL/tokenURL from config instead of hardcoded github.com endpoints
- Properly configure GitHub Enterprise API base URL from auth endpoints
Fixes OAuth2 authentication with GitHub Enterprise Server and other
self-hosted OAuth2 providers.
Signed-off-by: Mathias Bogaert <mathias.bogaert@gmail.com>
The "func" field in log output is incorrect in some cases and
showing internal logging methods instead of the actual calling
function. This is happening because the caller information is
being captured statically (5 call frames) which can be different
for different call patterns.
Changes:
* Move the caller capture to the event creation
* Use deterministic skip (3 frames) for event creation
* Add test cases to verify that the caller is captured correctly
Note: tests needed to written outside convey to avoid call insertion
Signed-off-by: Ravi Chamarthy <ravi@chamarthy.dev>
feat: add verify-feature retention subcommand with comprehensive testing and validation
Add a `verify-feature retention` subcommand that allows users to preview and
validate retention policy changes without running the actual Zot server.
The command runs GC and retention tasks in dry-run mode for immediate feedback.
- Run verify-feature retention standalone without starting the server
- Preview retention policy decisions in dry-run mode
- Configurable GC interval override via command-line flag
- Optional timeout for task completion
- Configurable log output (stdout or file)
Basic usage:
```bash
zot verify-feature retention <config-file>
```
With log file output:
```bash
zot verify-feature retention -l /var/log/zot-retention-check.log <config-file>
```
With GC interval override (runs GC tasks every 30 seconds):
```bash
zot verify-feature retention -i 30s <config-file>
```
With timeout (wait up to 5 minutes for tasks to complete):
```bash
zot verify-feature retention -t 5m <config-file>
```
Combined flags:
```bash
zot verify-feature retention -l /var/log/zot-retention-check.log -i 1m -t 10m <config-file>
```
The command supports overriding GC settings from the config:
- `-i, --gc-interval`: Override the GC interval setting (applies to all storage paths including subpaths)
- Refactored `RunGCTasks` from `controller.go` to be reusable
- Added `checkServerRunning` validation to prevent conflicts
- Implemented signal handling for graceful shutdown
- Added configuration sanitization and logging
- Set GCMaxSchedulerDelay programmatically (not user-configurable)
Added tests for coverage on main function:
- Negative test cases (no args, bad config, GC disabled, server running)
- Both BoltDB and Redis
- Retention enabled scenarios with complex image setups
- Retention disabled scenarios
- Delete referrers functionality
- Subpaths configuration
- GC interval override validation
Run the verify-feature retention tests:
```bash
go test -v ./pkg/cli/server -run TestRetentionCheck
```
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Make the Secure flag for session cookies configurable based on Zot's
TLS settings. This allows cookies to work properly when Zot is
accessed over HTTP (without TLS).
Changes:
- Add SecureSession field to AuthConfig to allow explicit control
- Add UseSecureSession() method that returns true when TLS is
configured, or uses SecureSession setting if provided
- Update saveUserLoggedSession() to accept and use secure parameter
- Add tests for UseSecureSession() in config_test.go
- Enhance authn tests to verify cookie Secure flag behavior
- Fix TestAuthnSessionErrors by creating new client without cookies
The logic is:
- If TLS is configured, cookies always have Secure=true
- If TLS is not configured but SecureSession is explicitly set,
use that value
- Otherwise, default to Secure=false for HTTP-only deployments
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: make config read/write thread safe and fix some other similar issues
1. The config config has a lock, and safe methods to update and read the attributes
2. The config has methods to retrieve copies of specific attributes, such as the extyensions config, the auth config, and the authz config.
These are needed, as the config object may mutate in the middle of an auth/authz requests, and we avoid partial configuration being applied for that request.
3. Fix an issue with the monitoring server not stopping when the controller is shut down.
4. Fix an issue with the HTPasswdWatcher not stopping when the background tasks are supposed to finish.
5. Fix some tests using hardcoded ports.
Moved some of the methods which were on the main config to the auth, access control and extension configs
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
Most users don't make the difference between retention deleting untagged manifests vs GC deleting other blobs.
This causes confusion since the GC delay and the retention delay (used for untagged manifests and orphan referrers) have different defaults, and are set separately in the zot configuration.
Most users don't configrue retention policies, and they still expect untagged manifests to be deleted at GC time.
With this change, if retention delay is not specified in the config file, the value used is the GC delay.
If GC delay is also unspecified in the config file, the default GC delay is used for both.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Description
====================
zot currently stores session cookies in memory or in a local directory.
For cases where the session cookies should be independent of the
instance where they were created such as multiple instances of zot, or a
fully stateless zot instance, there is a need to support a remote
session storage.
This change adds support for using Redis and Redis-compatible services as a
remote session driver as well as introduces a new configuration option
for it.
What has changed
=======================
- New config added under Auth config to specify configuration for
the session driver.
- Examples README updated with details of the new Auth config.
- The config supports only 2 drivers in this change - local and redis
- Using the local driver is backwards compatible and behaves the same
way that zot currently works for local session storage.
- Omitting this config does not result in an error. In this case, zot
behaves as it normally does for local session storage.
- When configured, zot can use redis for persisting cookie
information for zot UI.
- The cookie in the store is deleted on logout or after the max
expiry time for the cookie.
- Configuration for the redis session driver accepts the same configuration
values as that of the remote meta cache.
- A separate connection is established for the session driver. An
existing connection for meta cache will not be re-used for the
session driver.
- A key prefix is configurable for the redis session driver. The value will be
converted into a string for use. If no value is provided, a default
prefix of "zotsession" will be used.
- Redis sessions does not support hash key or encryption in this change.
- New BATS test added to verify zot behavior with Redis session store.
- Github workflow updated to install valkey-tools dependency for BATS.
Signed-off-by: Vishwas Rajashekar <dev@vrajashkr.com>
* fix: migrate from github.com/rs/zerolog to golang-native log/slog
We have been using zerolog for a really long time.
golang now has structured logging using slog.
Best to move to this in interests of long-term support.
This is a tech debt item.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: a few changes on top
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: address comments
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* chore: increase/stabilize coverage for the local storage driver
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* chore: add/stabilize coverage for soring ImageSummary objects
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* chore: stabilize coverage in sync tests
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
---------
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
- fixes#3347: removeUntaggedManifests() did not consider compatible manifest types
- add AsDockerImage() to Image and MultiarchImage for testing
- extend TestGarbageCollectAndRetentionMetaDB to test docker image and multiarch image
Signed-off-by: Stephan Merker <stephan.merker@sap.com>
fix: allow zot to build on a FreeBSD host (#3246)
The build works as long as the protoc package is installed on the build
host. This also fixes lint checks when building on FreeBSD, working
around common lint complaints caused by the fact that rlim_t is int64 on
FreeBSD.
Signed-off-by: Doug Rabson <dfr@rabson.org>
* feat: healthz server
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
* fix: startup and readiness probe activation points
Enable startup probe at end of Controller.Init and readiness probe at
end of Controller.Run
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
* fix: rewrote to reuse same HTTP listener
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
---------
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
Using just the last repository is not enough as in the case when it is deleted
(either by GC or some other way), GetNextRepository returns empty string
causing the generator to be marked completed without any errors.
An alternative would have been to start over from the first repository,
but this can take hours if multiple repositories need to be deleted,
not to mention the processing power and I/O and S3 load this could take.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* chore: bump github.com/olekukonko/tablewriter from 0.0.5 to 1.0.7
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* fix: zli failed to connect to https server using test certificates
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
It is to fix#3185.
This fixes the case where MetaDB is not instantiated (none of the conditions match),
and we want to retain tags only by pattern (which should not need to use MetaBD).
Without this fix you could only use retention to delete untagged manifests.
If you specified only the key "patterns" under "keepTags", zot would crash.
It was possible to not specify "keepTags" all, which would retain all tags,
but it was not possible to retains specific tags.
Basically the case quoted below, from the documentation, was broken::
https://zotregistry.dev/v2.1.4/articles/retention/#configuration-example
```
When you specify a regex pattern with no rules other than the default, all tags matching the pattern are retained.
```
This would only work if MetaDb was instantiated by an unrelated configured feature.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Ramkumar Chinchani