now gc stress on s3 storage is using minio for ci/cd builds
gc stress on s3 storage is using localstack for nightly builds
fixed(gc): make sure we don't remove repo if there are blobs
being uploaded or the number of blobs gc'ed is not 0
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
without removing its index.json reference
fix that by also reporting if manifests with unknown mediatypes
are referenced in index.json
this will make gc delete manifest blobs with deleteImageManifest() method
instead of deleteBlob(), which also removes index.json entries.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
unified both local and s3 ImageStore logic into a single ImageStore
added a new driver interface for common file/dirs manipulations
to be implemented by different storage types
refactor(gc): drop umoci dependency, implemented internal gc
added retentionDelay config option that specifies
the garbage collect delay for images without tags
this will also clean manifests which are part of an index image
(multiarch) that no longer exist.
fix(dedupe): skip blobs under .sync/ directory
if startup dedupe is running while also syncing is running
ignore blobs under sync's temporary storage
fix(storage): do not allow image indexes modifications
when deleting a manifest verify that it is not part of a multiarch image
and throw a MethodNotAllowed error to the client if it is.
we don't want to modify multiarch images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
- the size of the binary-minimal becomes 32MB
- "signatures" package is renamed into "imagetrust" and moved under extensions
- if the binary is not built using "imagetrust" tag then the signatures verification will
not be performed
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
* fix: remove inline GC and set a default value of gc interval
- remove inline GC
- add a default value of GC interval
- run the GC periodically by default with the default value if no interval provided
- generate GC tasks with a random delay(0-30s) between
- add IsReady() method to scheduler.TaskGenerator interface
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
* ci: add test for gc with short interval
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
---------
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
https://github.com/project-zot/zot/issues/1591
- I will rename "IMAGE NAME" to "REPOSITORY" in order to make the header easier to parse
- The order of the images cannot be predicted if zot is getting them 1 by 1 using the REST API for manifests, so they cannot be sorted when printed. We could wait on all calls to return but that may take minutes, and printing partial results as they become available is better.
- The order of the images can be predicted when relying on the zot specific search API, but that is not available in all zot servers depending on build options. I added sorting ascending by default. We are planning to implement configurable sorting in a separate PR - see the work under https://github.com/project-zot/zot/pull/1577
- With regards to the column widths/alignments that was discussed before, and the issue is we don't know the values beforehand for the REST API based responses. As mentioned above printing partial results as they become available is better.
- The column widths/alignments are partially fixed in this PR for the search API, but we should properly fix this in - see https://github.com/project-zot/zot/pull/851https://github.com/project-zot/zot/issues/1592
- Fix missing space after help message
https://github.com/project-zot/zot/issues/1598
- Fix table headers showing for json/yaml format
- Fix spacing shown with json format, use 1 row per shown entry in order to be compatible with json lines format: https://jsonlines.org/
- Add document header `---` to every image shown in yaml format to separate the entries
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
dist-spec APIs independently allow deletion of blobs and manifests.
Doing the former when in use by an image manifest or index is simply
error-prone. So disallow it.
Fixes issue #1509
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Co-authored-by: Ramkumar Chinchani <rchincha@cisco.com>
We use locks to protect OCI layouts. However, our critical sections have
too long and cover code instead of data.
Fixes issue #1595
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
when pushing manifests, zot will validate blobs (layers + config blob) are
present in repo, currently it opens(in case of filesystem storage) or download(
in case of cloud storage) each blob.
fixed that by adding a new method ImageStore.CheckBlobPresence() on storage
to check blobs presence without checking the cache like ImageStore.CheckBlob() method does.
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
* fix: missing Oci-Subject header pushing index with subject
Signed-off-by: Josh Dolitsky <josh@dolit.ski>
* fix(s3): Add a test to cover handling pushing indexes with a subject
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
---------
Signed-off-by: Josh Dolitsky <josh@dolit.ski>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
fix(storage/local): also put deduped blobs in cache, not just origin blobs
this caused an error when trying to delete deduped blobs
from multiple repositories
fix(storage/s3): check blob is present in cache before deleting
this is an edge case where dedupe is false but cacheDriver is not nil
(because in s3 we open the cache.db if storage find it in rootDir)
it caused an error when trying to delete blobs uploaded with dedupe false
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
Currently, when pushing an image, validation is performed to check that
a layer/blob in the manifest already exists. For non-distributable
layers, that check needs to be skipped.
Fixes issue #1394
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
"If filtering is requested and applied, the response MUST include an
annotation (org.opencontainers.referrers.filtersApplied) denoting that
an artifactType filter was applied.""
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
1. chore(trivy): update trivy library version
The trivy team switched github.com/urfave/cli for viper so
there are some other code changes as well.
Since we don't use github.com/urfave/cli directly in our software
we needed to add a tools.go in order for "go mod tidy" to not delete it.
See this pattern explained in:
- https://github.com/99designs/gqlgen#quick-start
- https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
- https://github.com/go-modules-by-example/index/blob/master/010_tools/README.md#walk-through
The jobs using "go get -u" have been updated to use "go install", since go get
modifies the go.mod by upgrading some of the packages, but downgrading trivy to an older
version with broken dependencies
2. fix(storage) Update local storage to ignore folder names not compliant with dist spec
Also updated trivy to download the DB and cache results under the rootDir/_trivy folder
3. fix(s3): one of the s3 tests was missing the skipIt call
This caused a failure when running locally without s3 being available
4. make sure the offline scanning is enabled, and zot only downloads the trivy DB
on the regular schedule, and doesn't download the DB on every image scan
ci: increase build and test timeout as tests are reaching the limit more often
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
* feat(repodb): implement a DB for image specific information using boltdb
(cherry picked from commit e3cb60b856)
Some other fixes/improvements on top (Andrei)
Global search: The last updated attribute on repo level is now computed correctly.
Global search: Fix and enhance tests: validate more fields, and fix CVE verification logic
RepoListWithNewestImage: The vendors and platforms at repo level are no longer containing duplicate entries
CVE: scan OCIUncompressedLayer instead of skiping them (used in tests)
bug(repodb): do no try to increment download counters for signatures
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Add filtering to global search API (Laurentiu)
(cherry picked from commit a87976d635ea876fe8ced532e8adb7c3bb24098f)
Original work by Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Fix pagination bug
- when limit was bigger than the repo count result contained empty results
- now correctly returns only maximum available number of repo results
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Add history to the fields returned from RepoDB
Consolidate fields used in packages
- pkg/extensions/search/common/common_test
- pkg/extensions/search/common/common
Refactor duplicate code in GlobalSearch verification
Add vulnerability scan results to image:tag reply
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Refactor ExpandedRepoInfo to using RepoDB
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit fd7dc85c3a9d028fd8860d3791cad4df769ed005)
Init RepoDB at startup
- sync with storage
- ignore images without a tag
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 359898facd6541b2aa99ee95080f7aabf28c2650)
Update request to get image:tag to use repodb
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Sync RepoDB logging
- added logging for errors
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 2e128f4d01712b34c70b5468285100b0657001bb)
sync-repodb minor error checking fix
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Improve tests for syncing RepoDB with storage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit b18408c6d64e01312849fc18b929e3a2a7931e9e)
Update scoring rule for repos
- now prioritize matches to the end of the repo name
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 6961346ccf02223132b3b12a2132c80bd1b6b33c)
Upgrade search filters to permit multiple values
- multiple values for os and arch
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 3ffb72c6fc0587ff827a03fe4f76a13b27b876a0)
feature(repodb): add pagination for RepoListWithNewestImage
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
(cherry picked from commit 32c917f2dc65363b0856345289353559a8027aee)
test(fix): fix tests failing since repodb is used for listing all repos
1. One of the tests was verifying disk/oci related erros and is not applicable
2. Another test was actually broken in an older PR, the default store and
the substore were using the same repo names (the substore ones were unprefixed),
which should not be the case, this was causing a single entry to show
in the RepoDB instead of two separate entries for each test image
Root cause in: https://github.com/project-zot/zot/commit/b61aff62cd5088314cee39fae42cec0ad855bd7c#diff-b86e11fa5a3102b336caebec3b30a9d35e26af554dd8658f124dba2404b7d24aR88
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
chore: move code reponsible for transforming objects to gql_generated types to separate package
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Process input for global search
- Clean input: query, filter strings
- Add validation for global search input
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit f1ca8670fbe4a4a327ea25cf459237dbf23bb78a)
fix: only call cve scanning for data shown to the user
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
GQL omit scanning for CVE if field is not required
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 5479ce45d6cb2abcf5fbccadeaf6f3393c3f6bf1)
Fix filtering logic in RepoDB
- filter parameter was set to false instead of being calculator from the later image
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit a82d2327e34e5da617af0b7ca78a2dba90999f0a)
bug(repodb): Checking signature returns error if signed image is not found
- we considere a signature image orfan when the image it signs is not found
- we need this to ignore such signatures in certain cases
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit d0418505f76467accd8e1ee34fcc2b2a165efae5)
feat(repodb): CVE logic to use repoDB
Also update some method signatures to remove usage of:
github.com/google/go-containerregistry/pkg/v1
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): refactor repodb update logic
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): minor fixes
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): move repodb logic inside meta directory under pkg
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): change factory class for repodb initialization with factory metrod
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): simplify repodb configuration
- repodb now shares config parameters with the cache
- config taken directly from storage config
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(authors): fix authors information to work properly with repodb
Ideally this commit would be squshed in the repodb commit
but as-is it is easier to cherry-pick on other branches
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
* feat(repodb): dynamodb support for repodb
- clean-up repodb code + coverage improvements
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(dynamo): tables used by dynamo are created automatically if they don't exists
- if the table exists nothing happens
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* test(repodb): coverage tests
- minor fix for CVEListForImage to fix the tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): add descriptor with media type
- to represent images and multi-arch images
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): support signatures on repo level
- added to follow the behavior of signing and signature verification tools
that work on a manifest level for each repo
- all images with different tags but the same manifest will be signed at once
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): old repodb version migration support
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): tests for coverage
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): WIP fixing tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* feat(repodb): work on patchRepoDB tests
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(repodb): create dynamo tables only for linux amd
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
* fix(ci): fix a typo in ci-cd.yml
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
Co-authored-by: Andrei Aaron <andaaron@cisco.com>
Co-authored-by: Andrei Aaron <aaaron@luxoft.com>
peusebiu