1. Only scan CVEs for images returned by graphql calls
Since pagination was refactored to account for image indexes, we had started
to run the CVE scanner before pagination was applied, resulting in
decreased ZOT performance if CVE information was requested
2. Increase in medory-cache of cve results to 1m, from 10k digests.
3. Update CVE model to use CVSS severity values in our code.
Previously we relied upon the strings returned by trivy directly,
and the sorting they implemented.
Since CVE severities are standardized, we don't need to pass around
an adapter object just for pagination and sorting purposes anymore.
This also improves our testing since we don't mock the sorting functions anymore.
4. Fix a flaky CLI test not waiting for the zot service to start.
5. Add the search build label on search/cve tests which were missing it.
6. The boltdb update method was used in a few places where view was supposed to be called.
7. Add logs for start and finish of parsing MetaDB.
8. Avoid unmarshalling twice to obtain annotations for multiarch images.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Initial code was contributed by Bogdan BIVOLARU <104334+bogdanbiv@users.noreply.github.com>
Moved implementation from a separate db to repodb by Andrei Aaron <aaaron@luxoft.com>
Not done yet:
- run/test dynamodb implementation, only boltdb was tested
- add additional coverage for existing functionality
- add web-based APIs to toggle the stars/bookmarks on/off
Initially graphql mutation was discussed for the missing API but
we decided REST endpoints would be better suited for configuration
feat(userdb): complete functionality for userdb integration
- dynamodb rollback changes to user starred repos in case increasing the total star count fails
- dynamodb increment/decrement repostars in repometa when user stars/unstars a repo
- dynamodb check anonymous user permissions are working as intendend
- common test handle anonymous users
- RepoMeta2RepoSummary set IsStarred and IsBookmarked
feat(userdb): rest api calls for toggling stars/bookmarks on/off
test(userdb): blackbox tests
test(userdb): move preferences tests in a different file with specific build tags
feat(repodb): add is-starred and is-bookmarked fields to repo-meta
- removed duplicated logic for determining if a repo is starred/bookmarked
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
- derivedImageList and baseImageList now use FilterTags to obtain results,
each with its own filter function
- images that have the exact same manifest as the one provided as a
parameter are no longer considered base images or derived images
- both calls can be made with specific pagination parameters, and the
response will include PageInfo
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
fix(tests): fix one of the pagination tests
The results were not reliable as the 2 returned tags were sorted by created date/time
which was not set, resulting in an unpredictable order
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
(cherry picked from commit be504200a1127371422aeb0e5c0219e2a1ead20a)
(cherry picked from commit ed8d797e639f262a63840120afe92da7db9a7600)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
* feat(repodb): implement pagination for ImageList and integrate it with RepoDB
- it can now return all images from all repos, when provided repo parameter is ""
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit c003dcec9f805564946935e7eb091632f605035e)
(cherry picked from commit 72feba979b9ddd452465a652bb31f439584a046c)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* ci(timeouts): increase ci-cd workflow timeout for the build and test step
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
add page info to dynamo-> feat(repodb): add PageInfo to GlobalSearch and RepoListWithNewestImage results
(cherry picked from commit 4fed42bb4bbc68199281d9d9a4e09b97fbd4759b)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
* feat(repodb): add pagination for ImageListForDigest and implement FilterTags
ImageListForDigest can now return paginated results, directly from DB.
It uses FilterTags, a new method to filter tags (obviously) based on
the criteria provided in the filter function.
Pagination of tags is now slightly different, it shows all results if
no limit and offset are provided.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
bug(tests): cli tests for digests expecting wrong size
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
(cherry picked from commit 369216df931a4053c18278a8d89f86d2e1e6a436)
fix(repodb): do not include repo metadata in search results if no matching tags are identified
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* fix(repodb): Fix an issue in FilterTags where repometa was not proceesed correctly
The filter function was called only once per manifest digest.
The function is supposed to also take into consideration repometa,
but only the first repometa-manifestmeta pair was processed.
Also increase code coverage.
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Alex Stan <alexandrustan96@yahoo.ro>
* feat(repodb): implement a DB for image specific information using boltdb
(cherry picked from commit e3cb60b856)
Some other fixes/improvements on top (Andrei)
Global search: The last updated attribute on repo level is now computed correctly.
Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic
RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries
CVE: scan OCIUncompressedLayer instead of skiping them (used in tests)
bug(repodb): do no try to increment download counters for signatures
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Add filtering to global search API (Laurentiu)
(cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f)
Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Fix pagination bug
- when limit was bigger than the repo count result contained empty results
- now correctly returns only maximum available number of repo results
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Add history to the fields returned from RepoDB
Consolidate fields used in packages
- pkg/extensions/search/common/common_test
- pkg/extensions/search/common/common
Refactor duplicate code in GlobalSearch verification
Add vulnerability scan results to image:tag reply
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Refactor ExpandedRepoInfo to using RepoDB
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005)
Init RepoDB at startup
- sync with storage
- ignore images without a tag
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650)
Update request to get image:tag to use repodb
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Sync RepoDB logging
- added logging for errors
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb)
sync-repodb minor error checking fix
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Improve tests for syncing RepoDB with storage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e)
Update scoring rule for repos
- now prioritize matches to the end of the repo name
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c)
Upgrade search filters to permit multiple values
- multiple values for os and arch
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0)
feature(repodb): add pagination for RepoListWithNewestImage
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee)
test(fix): fix tests failing since repodb is used for listing all repos
1. One of the tests was verifying disk/oci related erros and is not applicable
2. Another test was actually broken in an older PR, the default store and
the substore were using the same repo names (the substore ones were unprefixed),
which should not be the case, this was causing a single entry to show
in the RepoDB instead of two separate entries for each test image
Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
chore: move code reponsible for transforming objects to gql_generated types to separate package
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Process input for global search
- Clean input: query, filter strings
- Add validation for global search input
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a)
fix: only call cve scanning for data shown to the user
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
GQL omit scanning for CVE if field is not required
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1)
Fix filtering logic in RepoDB
- filter parameter was set to false instead of being calculator from the later image
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a)
bug(repodb): Checking signature returns error if signed image is not found
- we considere a signature image orfan when the image it signs is not found
- we need this to ignore such signatures in certain cases
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5)
feat(repodb): CVE logic to use repoDB
Also update some method signatures to remove usage of:
github.com/google/go-containerregistry/pkg/v1
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): refactor repodb update logic
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): minor fixes
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): move repodb logic inside meta directory under pkg
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): change factory class for repodb initialization with factory metrod
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): simplify repodb configuration
- repodb now shares config parameters with the cache
- config taken directly from storage config
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(authors): fix authors information to work properly with repodb
Ideally this commit would be squshed in the repodb commit
but as-is it is easier to cherry-pick on other branches
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): dynamodb support for repodb
- clean-up repodb code + coverage improvements
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(dynamo): tables used by dynamo are created automatically if they don't exists
- if the table exists nothing happens
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* test(repodb): coverage tests
- minor fix for CVEListForImage to fix the tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): add descriptor with media type
- to represent images and multi-arch images
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): support signatures on repo level
- added to follow the behavior of signing and signature verification tools
that work on a manifest level for each repo
- all images with different tags but the same manifest will be signed at once
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): old repodb version migration support
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): tests for coverage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): WIP fixing tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): work on patchRepoDB tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): create dynamo tables only for linux amd
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(ci): fix a typo in ci-cd.yml
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
Currently there is no push-back on queries that should contain image names but
have only the repo name. This commit adds a check that will return an error for images
w/o a reference(tag or digest).
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
UI can now make use of OCI artifacts and references using `Referrers` gQL query.
It returns a list of descriptors that refer on their `subject` field to another
digest.
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
- Digests were represented by different ways
- We needed a uniform way to represent the digests and enforce a format
- also replace usage of github.com/google/go-containerregistry/pkg/v1
with github.com/opencontainers/image-spec/specs-go/v1
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)
Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Return this data as part of GlobalSearch and RepoListWithNewestImage
query results.
This commit also includes refactoring of the CVE scanning logic in
order to better encapsulate trivy specific logic, remove CVE scanning
logic from the graphql resolver.
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Ramkumar Chinchani