ImageIndex2ImageSummary was missing LastPullTimestamp assignment, causing
multi-arch image queries to always return null for this field. Also adds
the PushedBy field (already stored in MetaDB) to the GraphQL schema and
both conversion paths (manifest and index).
Signed-off-by: cainydev <wajo432@gmail.com>
1. Parse repos without metadata in ParseStorage
The timestamp check in ParseStorage was skipping repos that exist in
storage but don't have metadata. When GetRepoLastUpdated returns zero
time (no metadata), we should always parse the repo to create its
metadata. Check if metaLastUpdated is zero before comparing timestamps.
If zero, always parse regardless of storageLastUpdated.
2. Change the logic of how LastUpdated is computed in RepoSummary
It is not the latest tagged timestamp from the available images or
the last updated image created timestamp, based on whichever is the
latest.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
feat(meta): add TaggedTimestamp field and preserve during re-parsing
Add TaggedTimestamp field to track when image tags were created, exposed
through GraphQL API. Previously, when zot restarted and re-parsed storage,
ResetRepoReferences would clear all tags, causing timestamp information to
be lost and reset to the service restart time for existing images.
This change adds TaggedTimestamp support and modifies ResetRepoReferences to
selectively preserve tags that still exist in storage, maintaining their
TaggedTimestamp values. Tags that no longer exist in storage are removed as
before.
Changes:
- Add TaggedTimestamp field to GraphQL ImageSummary schema
- Update GraphQL conversion functions to populate TaggedTimestamp with
fallback to PushTimestamp when unavailable
- Updated ResetRepoReferences interface to accept tagsToKeep parameter
- Modified ParseRepo to collect tags from storage before resetting
- Updated all backend implementations (Redis, DynamoDB, BoltDB) to preserve
tags in tagsToKeep instead of clearing all tags
- Updated tests and mocks to match new signature
This ensures TaggedTimestamp accurately reflects when tags were originally
created, and exposes this information through the GraphQL API.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
This commit modernizes code across multiple packages by:
- Using Go 1.18+ features (slices.IndexFunc, strings.Cut)
- Pre-allocating slices and maps with known capacity
- Consolidating defensive checks and improving code clarity
- Fixing test data and build tag issues
CLI client improvements:
- Pre-allocate slices in search functions and service methods
- Replace strings.Split with strings.Cut for username:password parsing
- Use range-based iteration instead of manual index loops
Search extension optimizations:
- Cache sort functions in pagination modules
- Pre-allocate page buffers and maps
- Consolidate defensive checks in filterBaseImages/filterDerivedImages
- Fix image bas and derived logic allowing out of sequence layers for base images
- Fix image pagination reporting images groupped by repos when sorted by update time
- Remove duplicate resolver_test.go file
Monitoring extension:
- Replace manual loops with slices.IndexFunc
- Pre-allocate bucketsFloat2String map
Sync extension:
- Pre-allocate slice in parseRegistryURLs
Test utilities:
- Fix build tags in oci_layout.go
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: migrate to Go module v2 for proper semantic versioning
This change updates the module path from 'zotregistry.dev/zot' to
'zotregistry.dev/zot/v2' to comply with Go's semantic versioning rules.
According to Go's module versioning requirements, major version v2+
must include the major version in the module path. The current
module path 'zotregistry.dev/zot' only supports v0.x.x and v1.x.x
versions, making existing v2.x.x tags (like v2.1.8) unusable.
Changes:
- Updated go.mod module path to zotregistry.dev/zot/v2
- Updated all internal import paths across 280+ Go source files
- Updated configuration files (golangcilint.yaml, gqlgen.yml)
- Updated README.md Go reference badge
This fix enables proper use of existing v2.x.x Git tags and allows
external packages to import zot v2+ versions without compatibility
errors.
Resolves: Go module import compatibility for v2+ versions
Fixes: #3071
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: regenerate GraphQL files with updated v2 import paths
The gqlgen tool needs to regenerate the GraphQL schema files after
the module path change to use the new v2 imports.
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
---------
Signed-off-by: Luca Muscariello <muscariello@ieee.org>
* fix: migrate from github.com/rs/zerolog to golang-native log/slog
We have been using zerolog for a really long time.
golang now has structured logging using slog.
Best to move to this in interests of long-term support.
This is a tech debt item.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: a few changes on top
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* fix: address comments
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
Fallback to Created field and the History entries in the image config
only if the annotation "org.opencontainers.image.created" is not available
closes#2210
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
For CLI output is similar to:
CRITICAL 0, HIGH 1, MEDIUM 1, LOW 0, UNKNOWN 0, TOTAL 2
ID SEVERITY TITLE
CVE-2023-0464 HIGH openssl: Denial of service by excessive resou...
CVE-2023-0465 MEDIUM openssl: Invalid certificate policies in leaf...
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- added a new field 'IsDeletable' for graphql ImageSummary struct.
- apply cors on DeleteManifest route
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
wait for workers to finish before exiting
should fix tests reporting they couldn't remove rootDir because it's being
written by tasks
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
test "zotregistry.io/zot/pkg/test/common" - in tests
. "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.
2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
. "zotregistry.io/zot/pkg/test/image-utils"
3. "zotregistry.io/zot/pkg/test/deprecated" - curently used as
"zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.
4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
"zotregistry.io/zot/pkg/test/signature"
5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc) curently used as
authutils "zotregistry.io/zot/pkg/test/auth"
6. "zotregistry.io/zot/pkg/test/oci-utils" - curently used as
ociutils "zotregistry.io/zot/pkg/test/oci-utils"
Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.
Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
1. Only scan CVEs for images returned by graphql calls
Since pagination was refactored to account for image indexes, we had started
to run the CVE scanner before pagination was applied, resulting in
decreased ZOT performance if CVE information was requested
2. Increase in medory-cache of cve results to 1m, from 10k digests.
3. Update CVE model to use CVSS severity values in our code.
Previously we relied upon the strings returned by trivy directly,
and the sorting they implemented.
Since CVE severities are standardized, we don't need to pass around
an adapter object just for pagination and sorting purposes anymore.
This also improves our testing since we don't mock the sorting functions anymore.
4. Fix a flaky CLI test not waiting for the zot service to start.
5. Add the search build label on search/cve tests which were missing it.
6. The boltdb update method was used in a few places where view was supposed to be called.
7. Add logs for start and finish of parsing MetaDB.
8. Avoid unmarshalling twice to obtain annotations for multiarch images.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
- image command is now deprecated in favor of 'images'
- cve command is now deprecated in favor of 'cves'
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
https://github.com/project-zot/zot/issues/1591
- I will rename "IMAGE NAME" to "REPOSITORY" in order to make the header easier to parse
- The order of the images cannot be predicted if zot is getting them 1 by 1 using the REST API for manifests, so they cannot be sorted when printed. We could wait on all calls to return but that may take minutes, and printing partial results as they become available is better.
- The order of the images can be predicted when relying on the zot specific search API, but that is not available in all zot servers depending on build options. I added sorting ascending by default. We are planning to implement configurable sorting in a separate PR - see the work under https://github.com/project-zot/zot/pull/1577
- With regards to the column widths/alignments that was discussed before, and the issue is we don't know the values beforehand for the REST API based responses. As mentioned above printing partial results as they become available is better.
- The column widths/alignments are partially fixed in this PR for the search API, but we should properly fix this in - see https://github.com/project-zot/zot/pull/851https://github.com/project-zot/zot/issues/1592
- Fix missing space after help message
https://github.com/project-zot/zot/issues/1598
- Fix table headers showing for json/yaml format
- Fix spacing shown with json format, use 1 row per shown entry in order to be compatible with json lines format: https://jsonlines.org/
- Add document header `---` to every image shown in yaml format to separate the entries
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>
Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off
Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration
feat(userdb): complete functionality for userdb integration
- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked
feat(userdb): rest api calls for toggling stars/bookmarks on/off
test(userdb): blackbox tests
test(userdb): move preferences tests in a different file with specific build tags
feat(repodb): add is-starred and is-bookmarked fields to repo-meta
- removed duplicated logic for determining if a repo is starred/bookmarked
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
* refactor(repodb): moving common utilities under pkg/meta
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* refactor(repodb): moved update, version components under pkg/meta
- updated wrapper initialization to recieve a log object in constructor
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* refactor(repodb): moved repodb initialization from controller to pkg/meta/repodb
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
---------
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Changed repodb to store more information about the referrer needed for the referrers query
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
If we push an artifact and give it a tag, repodb would crash because of the null pointer dereferencing
Now when iterating over the tags of a repo and stumbling upon a unsupported media type, it's being ignored
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): implement pagination for ImageList and integrate it with RepoDB
- it can now return all images from all repos, when provided repo parameter is ""
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit c003dcec9f805564946935e7eb091632f605035e)
(cherry picked from commit 72feba979b9ddd452465a652bb31f439584a046c)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* ci(timeouts): increase ci-cd workflow timeout for the build and test step
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
add page info to dynamo-> feat(repodb): add PageInfo to GlobalSearch and RepoListWithNewestImage results
(cherry picked from commit 4fed42bb4bbc68199281d9d9a4e09b97fbd4759b)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
* feat(repodb): implement a DB for image specific information using boltdb
(cherry picked from commit e3cb60b856)
Some other fixes/improvements on top (Andrei)
Global search: The last updated attribute on repo level is now computed correctly.
Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic
RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries
CVE: scan OCIUncompressedLayer instead of skiping them (used in tests)
bug(repodb): do no try to increment download counters for signatures
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Add filtering to global search API (Laurentiu)
(cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f)
Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Fix pagination bug
- when limit was bigger than the repo count result contained empty results
- now correctly returns only maximum available number of repo results
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Add history to the fields returned from RepoDB
Consolidate fields used in packages
- pkg/extensions/search/common/common_test
- pkg/extensions/search/common/common
Refactor duplicate code in GlobalSearch verification
Add vulnerability scan results to image:tag reply
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Refactor ExpandedRepoInfo to using RepoDB
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005)
Init RepoDB at startup
- sync with storage
- ignore images without a tag
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650)
Update request to get image:tag to use repodb
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Sync RepoDB logging
- added logging for errors
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb)
sync-repodb minor error checking fix
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Improve tests for syncing RepoDB with storage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e)
Update scoring rule for repos
- now prioritize matches to the end of the repo name
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c)
Upgrade search filters to permit multiple values
- multiple values for os and arch
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0)
feature(repodb): add pagination for RepoListWithNewestImage
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee)
test(fix): fix tests failing since repodb is used for listing all repos
1. One of the tests was verifying disk/oci related erros and is not applicable
2. Another test was actually broken in an older PR, the default store and
the substore were using the same repo names (the substore ones were unprefixed),
which should not be the case, this was causing a single entry to show
in the RepoDB instead of two separate entries for each test image
Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
chore: move code reponsible for transforming objects to gql_generated types to separate package
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Process input for global search
- Clean input: query, filter strings
- Add validation for global search input
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a)
fix: only call cve scanning for data shown to the user
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
GQL omit scanning for CVE if field is not required
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1)
Fix filtering logic in RepoDB
- filter parameter was set to false instead of being calculator from the later image
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a)
bug(repodb): Checking signature returns error if signed image is not found
- we considere a signature image orfan when the image it signs is not found
- we need this to ignore such signatures in certain cases
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5)
feat(repodb): CVE logic to use repoDB
Also update some method signatures to remove usage of:
github.com/google/go-containerregistry/pkg/v1
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): refactor repodb update logic
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): minor fixes
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): move repodb logic inside meta directory under pkg
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): change factory class for repodb initialization with factory metrod
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): simplify repodb configuration
- repodb now shares config parameters with the cache
- config taken directly from storage config
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(authors): fix authors information to work properly with repodb
Ideally this commit would be squshed in the repodb commit
but as-is it is easier to cherry-pick on other branches
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): dynamodb support for repodb
- clean-up repodb code + coverage improvements
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(dynamo): tables used by dynamo are created automatically if they don't exists
- if the table exists nothing happens
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* test(repodb): coverage tests
- minor fix for CVEListForImage to fix the tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): add descriptor with media type
- to represent images and multi-arch images
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): support signatures on repo level
- added to follow the behavior of signing and signature verification tools
that work on a manifest level for each repo
- all images with different tags but the same manifest will be signed at once
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): old repodb version migration support
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): tests for coverage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): WIP fixing tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): work on patchRepoDB tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): create dynamo tables only for linux amd
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(ci): fix a typo in ci-cd.yml
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
John Wagner