Commit Graph

38 Commits

Author SHA1 Message Date
peusebiu 024b13efe6 fix(sync): syncing OCI artifacts with distribution package fails (#1013)
sync OCI artifacts using REST APIs

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-12-09 11:38:00 -08:00
peusebiu e96c80c344 feat(sync,s3): added s3 logic for ORAS and OCI artifacts (#985)
added sync logic for OCI artifacts

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-11-14 22:21:49 -08:00
peusebiu 2d877aaea1 fix(sync): also sync on demand digests, not only tags, closes #902 (#932)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-27 09:39:59 -07:00
Andrei Aaron ac6c6a844c refactor(digests): standardise representation of digests to digest.Digest (#898)
- Digests were represented by different ways
  - We needed a uniform way to represent the digests and enforce a format
  - also replace usage of github.com/google/go-containerregistry/pkg/v1
    with github.com/opencontainers/image-spec/specs-go/v1

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)

Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2022-10-22 13:46:13 -07:00
peusebiu 5f99f9a445 fix(sync): fixed broken logic to get tags for repo (#900)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-22 00:26:14 -07:00
Andrei Aaron 38b00e3507 chore(lint): gci to separate zot from other imports (#870)
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
2022-10-20 09:39:20 -07:00
peusebiu 3c0c51fcbe fix(sync): also sync image index mediatype (#847)
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-06 09:41:16 -07:00
peusebiu c146448f01 fix(sync): revert code which removed image destination feature (#840)
Added an end to end test for this feature, closes #793

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-10-05 11:03:24 -07:00
Nicol 33a431ef43 Update go version to 1.19 (#829)
* ci: Update go version to 1.19

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Fix lint issues

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Added needprivileges to lint, made needprivileges pass lint

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Nicol Draghici <idraghic@cisco.com>

Signed-off-by: Nicol Draghici <idraghic@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-10-05 13:21:14 +03:00
LaurentiuNiculae 885f139e0e Remove forking logger (#825)
- no longer needed, the race conditions were fixed

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2022-09-29 13:28:39 -07:00
Lisca Ana-Roberta 87fc941b3c image level lint: enforce manifest mandatory annotations
closes #536

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
2022-07-27 11:48:04 +03:00
Catalin Hofnar a8a65a6c37 Modified sync log calls to include error type (#336)
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-06-15 09:45:49 -07:00
Petu Eusebiu da4acaf178 sync: preserve upstream digests after syncing images
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-05-25 10:19:36 -07:00
Petu Eusebiu bd730150a8 sync: allow HTTP redirects when GETing signatures blobs
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-05-25 10:19:36 -07:00
Shivam Mishra 36c9631000 ext: use distribution spec route prefix for extension api
Following the spec defined here https://github.com/opencontainers/distribution-spec/tree/main/extensions

Signed-off-by: Shivam Mishra <shimish2@cisco.com>
2022-05-22 16:35:16 -07:00
Catalin Hofnar 20a60cbad4 Enhance sync logic - stop blob redownloads and re-pushes (#479 #480)
Changed imagesToCopyFromUpstream to return a map[string][]types.ImageReference from just an array of refs
Rewrote some logic in sync.go to use the new signature of imagesToCopyFromUpstream
Split getLocalImageRef by adding function getLocalCachePath
Adapted tests for new changes, added some tests
Merged #481

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
2022-05-16 10:05:01 -07:00
Alex Stan d325c8b5f4 Fix problems signaled by new linter version v1.45.2
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.

maintidx linter disabled

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
2022-04-27 09:55:44 -07:00
Petu Eusebiu f53dc9eb8d sync: Add a new flag to enforce syncing only signed images, closes #455
sync: When checking if a image is already synced also check for changes in upstream signatures.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-24 10:50:01 -07:00
laurentiuNiculae 0d148e1d6b new config option for sync-destination
Signed-off-by: laurentiuNiculae <themelopeus@gmail.com>
2022-03-21 08:12:34 -07:00
Petu Eusebiu 6d04ab3cdc sync: support reloading sync config when the config file changes
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-18 11:24:11 -07:00
Petu Eusebiu 45968e0bb7 sync: fix inconsistent test, used inject error fw for hard to reach test cases
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-03-01 09:56:27 -08:00
Petu Eusebiu 35eeedb22a sync: pull only missing images, not everything, closes #335
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-02-14 08:09:18 -08:00
Alexei Dodon 47c9b6244e Added config enable=true/false for extensions with default value as enabled closes #258
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2022-02-09 09:53:49 -08:00
Petu Eusebiu f89925fb27 sync: periodically retry if on-demand fails inline, closes #281
sync: don't return error on sync signatures, just skip them, closes #375
sync: sync signatures on demand
sync on demand: in case of parallel requests pull image just once, closes #344

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-02-07 09:04:13 -08:00
Petu Eusebiu a0e65379c8 sync: for a prefix, allow multiple registries as a list instead of only one, closes #343
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-02-01 09:45:09 -08:00
Petu Eusebiu 1109bb4dde sync: Added support for syncing notary/cosign signatures, closes #261
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2022-01-13 08:45:59 -08:00
Ramkumar Chinchani ac3801ea2d lint: upgrade golangci-lint
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-20 17:20:35 -08:00
Petu Eusebiu 5f04092e71 Fix negative WaitGroup counter at runtime
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-17 16:52:34 -08:00
Petu Eusebiu c86f44cc53 Disable sync periodically polling when pollInterval is not configured
Filtering out sync on demand images based on content configuration

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-14 08:59:50 -08:00
Petu Eusebiu 627cb97ef1 Add wait group for graceful shutdown, closes #302
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-08 10:04:52 -08:00
Petu Eusebiu 63a75216ed sync: allow for saving to a subpath, closes #307
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-12-07 10:13:20 -08:00
Ramkumar Chinchani 96226af869 move references to zotregistry.io and project-zot
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2021-12-05 10:52:27 -08:00
Petu Eusebiu fff6107310 Sync prefix can be an exact match or a glob pattern, closes #297
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-29 13:10:13 -08:00
Petu Eusebiu f0ef10fa50 sync: cleanup the orphaned private download dir on failure, closes 282
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-25 10:31:45 -08:00
Petu Eusebiu 7ada50e9c8 sync: skip tls verify on /v2/_catalog when a registry is configured with tls-verify false
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-25 10:31:45 -08:00
Petu Eusebiu 5c07e19c8d Changed sync behaviour, it used to copy images over http interface
now it copies to a local cache and then it copies over storage APIs

- accept all images with or without signatures
- disable sync writing to stdout
- added more logs
- fixed switch statement in routes
- fixed enabling sync multiple times for storage subpaths

closes #266

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-11-15 09:32:43 -08:00
Petu Eusebiu f7ae491d22 Fix data race in sync tests
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-10-25 10:50:05 -07:00
Petu Eusebiu 19003e8a71 Added new extension "sync"
Periodically poll registries and pull images according to sync's config
Added sync on demand, syncing when clients asks for an image which
zot doesn't have.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2021-10-21 10:32:46 -07:00