1. Parse repos without metadata in ParseStorage
The timestamp check in ParseStorage was skipping repos that exist in
storage but don't have metadata. When GetRepoLastUpdated returns zero
time (no metadata), we should always parse the repo to create its
metadata. Check if metaLastUpdated is zero before comparing timestamps.
If zero, always parse regardless of storageLastUpdated.
2. Change the logic of how LastUpdated is computed in RepoSummary
It is not the latest tagged timestamp from the available images or
the last updated image created timestamp, based on whichever is the
latest.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
- Refactored HTTP client from global cache to struct-based approach (global state was shared between tests, including what certificates to use)
- Enhanced pkg/test/tls to support ECDSA and ED25519 key types
- Replaced static certificate files with dynamic generation in golang tests
- Fixed test cleanup issues and improved resource management
This eliminates dependency on external cert generation scripts and
improves test maintainability.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: make config read/write thread safe and fix some other similar issues
1. The config config has a lock, and safe methods to update and read the attributes
2. The config has methods to retrieve copies of specific attributes, such as the extyensions config, the auth config, and the authz config.
These are needed, as the config object may mutate in the middle of an auth/authz requests, and we avoid partial configuration being applied for that request.
3. Fix an issue with the monitoring server not stopping when the controller is shut down.
4. Fix an issue with the HTPasswdWatcher not stopping when the background tasks are supposed to finish.
5. Fix some tests using hardcoded ports.
Moved some of the methods which were on the main config to the auth, access control and extension configs
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* feat: healthz server
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
* fix: startup and readiness probe activation points
Enable startup probe at end of Controller.Init and readiness probe at
end of Controller.Run
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
* fix: rewrote to reuse same HTTP listener
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
---------
Signed-off-by: Asgeir Nilsen <asgeir@twingine.no>
* feat(cluster): initial commit for scale-out cluster
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
* feat(cluster): support shared storage scale out
This change introduces support for shared storage backed
zot cluster scale out.
New feature
Multiple stateless zot instances can run using the same shared
storage backend where each instance looks at a specific set
of repositories based on a siphash of the repository name to improve
scale as the load is distributed across multiple instances.
For a given config, there will only be one instance that can perform
dist-spec read/write on a given repository.
What's changed?
- introduced a transparent request proxy for dist-spec endpoints based on
siphash of repository name.
- new config for scale out cluster that specifies list of
cluster members.
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
refactor(http): refactor http client to take options struct
This commit updates the arguments for the `CreateHTTPClient`
function to consume a struct which can be extended as required.
It replaces the certPath argument with a struct of 3 paths for
client ertificate, client key, and ca cert. It also adds
a TLSEnabled option for when an HTTP Client is required
without any further TLS config.
Existing consumers of this function have been updated so that
they can work as they do today. This change is a no-op for
existing features.
This allows for certificate paths to be customised and
allows other modules to re-use the same HTTP client and get
the benefits of mTLS support and per-host certificates.
Signed-off-by: Vishwas Rajashekar <vrajashe@cisco.com>
For CLI output is similar to:
CRITICAL 0, HIGH 1, MEDIUM 1, LOW 0, UNKNOWN 0, TOTAL 2
ID SEVERITY TITLE
CVE-2023-0464 HIGH openssl: Denial of service by excessive resou...
CVE-2023-0465 MEDIUM openssl: Invalid certificate policies in leaf...
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
wait for workers to finish before exiting
should fix tests reporting they couldn't remove rootDir because it's being
written by tasks
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- Cosign supports 2 types of signature formats:
1. Using tag -> each new signature of the same manifest is
added as a new layer of the signature manifest having that
specific tag("{alghoritm}-{digest_of_signed_manifest}.sig")
2. Using referrers -> each new signature of the same manifest is
added as a new manifest
- For adding these cosign signature to metadb, we reserved index 0 of the
list of cosign signatures for tag-based signatures. When a new tag-based
signature is added for the same manifest, the element on first position
in its list of cosign signatures(in metadb) will be updated/overwritten.
When a new cosign signature(using referrers) will be added for the same
manifest this new signature will be appended to the list of cosign
signatures.
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
ci(notation): update to latest notation version
fix(sync): add layers info when syncing signatures
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
test "zotregistry.io/zot/pkg/test/common" - in tests
. "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.
2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
. "zotregistry.io/zot/pkg/test/image-utils"
3. "zotregistry.io/zot/pkg/test/deprecated" - curently used as
"zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.
4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
"zotregistry.io/zot/pkg/test/signature"
5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc) curently used as
authutils "zotregistry.io/zot/pkg/test/auth"
6. "zotregistry.io/zot/pkg/test/oci-utils" - curently used as
ociutils "zotregistry.io/zot/pkg/test/oci-utils"
Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.
Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
fix(authz): fix isAdmin not using groups to determine if a user is admin.
fix(authz): return 401 instead of 403
403 is correct as per HTTP spec
However authz is not part of dist-spec and clients know only about 401
So this is a compromise.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
- the size of the binary-minimal becomes 32MB
- "signatures" package is renamed into "imagetrust" and moved under extensions
- if the binary is not built using "imagetrust" tag then the signatures verification will
not be performed
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
BREAKING CHANGE: The functionality provided by the mgmt endpoint has beed redesigned - see details below
BREAKING CHANGE: The API keys endpoint has been moved - see details below
BREAKING CHANGE: The mgmt extension config has been removed - endpoint is now enabled by having both the search and the ui extensions enabled
BREAKING CHANGE: The API keys configuration has been moved from extensions to http>auth>apikey
mgmt and imagetrust extensions:
- separate the _zot/ext/mgmt into 3 separate endpoints: _zot/ext/auth, _zot/ext/notation, _zot/ext/cosign
- signature verification logic is in a separate `imagetrust` extension
- better hanling or errors in case of signature uploads: logging and error codes (more 400 and less 500 errors)
- add authz on signature uploads (and add a new middleware in common for this purpose)
- remove the mgmt extension configuration - it is now enabled if the UI and the search extensions are enabled
userprefs estension:
- userprefs are enabled if both search and ui extensions are enabled (as opposed to just search)
apikey extension is removed and logic moved into the api folder
- Move apikeys code out of pkg/extensions and into pkg/api
- Remove apikey configuration options from the extensions configuration and move it inside the http auth section
- remove the build label apikeys
other changes:
- move most of the logic adding handlers to the extensions endpoints out of routes.go and into the extensions files.
- add warnings in case the users are still using configurations with the obsolete settings for mgmt and api keys
- add a new function in the extension package which could be a single point of starting backgroud tasks for all extensions
- more clear methods for verifying specific extensions are enabled
- fix http methods paired with the UI handlers
- rebuild swagger docs
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
This change introduces OpenID authn by using providers such as Github,
Gitlab, Google and Dex.
User sessions are now used for web clients to identify
and persist an authenticated users session, thus not requiring every request to
use credentials.
Another change is apikey feature, users can create/revoke their api keys and use them
to authenticate when using cli clients such as skopeo.
eg:
login:
/auth/login?provider=github
/auth/login?provider=gitlab
and so on
logout:
/auth/logout
redirectURL:
/auth/callback/github
/auth/callback/gitlab
and so on
If network policy doesn't allow inbound connections, this callback wont work!
for more info read documentation added in this commit.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>
Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off
Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration
feat(userdb): complete functionality for userdb integration
- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked
feat(userdb): rest api calls for toggling stars/bookmarks on/off
test(userdb): blackbox tests
test(userdb): move preferences tests in a different file with specific build tags
feat(repodb): add is-starred and is-bookmarked fields to repo-meta
- removed duplicated logic for determining if a repo is starred/bookmarked
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
Andrei Aaron