github/zot
1
0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2026-06-16 20:38:08 +08:00
Code Issues Packages Projects Releases Wiki Activity
604 Commits 38 Branches 132 Tags
2d877aaea10d471391ff3a1d51e32d1902aa98b9
Commit Graph

91 Commits

Author SHA1 Message Date
Andrei Aaron ac6c6a844c refactor(digests): standardise representation of digests to digest.Digest (#898) 
- Digests were represented by different ways
  - We needed a uniform way to represent the digests and enforce a format
  - also replace usage of github.com/google/go-containerregistry/pkg/v1
    with github.com/opencontainers/image-spec/specs-go/v1

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
(cherry picked from commit 96b2f29d6d57070a913ce419149cd481c0723815)
(cherry picked from commit 3d41b583daea654c98378ce3dcb78937d71538e8)

Co-authored-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
 2022-10-22 13:46:13 -07:00
Andrei Aaron 38b00e3507 chore(lint): gci to separate zot from other imports (#870) 
Signed-off-by: Andrei Aaron <andaaron@cisco.com>
 2022-10-20 09:39:20 -07:00
peusebiu 92d97d48d6 fix(s3): remove tracking multipart uploads (#883) 
Remove sticky sessions from clustering

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-10-20 09:36:58 -07:00
peusebiu fd87a22829 fix(storage): resolve cache/storage inconsistencies on HEAD request (#794) 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-10-11 10:05:39 -07:00
Nicol 33a431ef43 Update go version to 1.19 (#829) 
* ci: Update go version to 1.19

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Fix lint issues

Signed-off-by: Nicol Draghici <idraghic@cisco.com>

* ci: Added needprivileges to lint, made needprivileges pass lint

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Signed-off-by: Nicol Draghici <idraghic@cisco.com>

Signed-off-by: Nicol Draghici <idraghic@cisco.com>
Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
Co-authored-by: Catalin Hofnar <catalin.hofnar@gmail.com>
 2022-10-05 13:21:14 +03:00
peusebiu 8237f8d20a storage: Move common code in helper functions, closes #730 (#820) 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-09-30 10:35:16 -07:00
Andreea Lupu f686ab6bf6 initial design for task scheduler (#700) 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-09-22 22:27:56 -07:00
Ramkumar Chinchani 90c8390c29 routes: support resumable pull 
Some embedded clients use the "Range" header for blob pulls in order to
resume downloads.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-09-15 15:51:47 -07:00
Petu Eusebiu 7912b6a3fb Refactor s3 dedupe 
Under digest bucket create 2 buckets, one for storing origin blob
and one for storing deduped blobs.

PutBlob() - puts an origin blob in both buckets
          - puts a deduped blob in deduped bucket
GetBlob() - returns blobs only from origin bucket
DeleteBlob() - deletes an origin blob from both buckets
             and moves one deduped blob into origin bucket
             - deletes a deduped blob from deduped bucket

[storage] When deleting an origin blob, next time we GetBlob() we get a
deduped blob with no content and we will move the content from
the deleted origin blob to it (inside s3.go).

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-09-14 14:43:58 -07:00
peusebiu 5479e2c785 s3: fix dedupe failing to manage blobs correctly (#772) 
in order to know which blob is 'real' (has content)
we need to know which was the first blob inserted in cache,
because that is always the real one.

because we can not modify the keys order in boltdb we'll do this
by marking the first blob inserted with a value

when GetBlob() return the blob which is marked
when PutBlob() if is the first one, mark it
when DeleteBlob() in case deleted is marked then mark the next blob

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-09-08 01:12:14 +03:00
slab713 8ffb053cec Replaced deprecated io/ioutil functions (#768) 
Signed-off-by: slab713 <109306207+slab713@users.noreply.github.com>
 2022-09-02 15:56:02 +03:00
Ramkumar Chinchani 2ff8e8b7d2 fix dependabot alerts 
https://github.com/project-zot/zot/pull/737
https://github.com/project-zot/zot/pull/738
https://github.com/project-zot/zot/pull/739
https://github.com/project-zot/zot/pull/740

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-08-23 09:38:30 -07:00
Ramkumar Chinchani 5c01c4eab4 support OCI image index at manifest endpoint (#638) 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-08-20 11:18:48 +03:00
Andrei Aaron bd9ad998cd Fix file handlers not being closed after calls to ImageStore.GetBlob 
This is to fixes hitting the FD limit when reading blobs from the disk in the graphql API

Signed-off-by: Andrei Aaron <andaaron@cisco.com>
 2022-08-19 09:22:13 -07:00
alexstan12 16e9822c7f Add fuzz tests for storage_fs (#601) 
This commit uses native go fuzzing to fuzz test implementations
of storage in storage_fs.

moved fuzzing testdata for storage_fs in separate repo

added make target and script for importing fuzz data and running all fuzz tests

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
 2022-07-27 20:37:55 +03:00
Lisca Ana-Roberta 87fc941b3c image level lint: enforce manifest mandatory annotations 
closes #536

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-07-27 11:48:04 +03:00
Laurentiu Niculae 7e3d063319 freeform querry api 
Signed-off-by: Laurentiu Niculae <themelopeus@gmail.com>
 2022-07-20 10:03:11 -07:00
Alex Stan a31869f270 fix GetReferrers function to be able to retrieve referrers of any specified artifactType 
Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
 2022-07-19 09:44:23 -07:00
Ramkumar Chinchani eed48c1715 refactor filenames to reflect functionality 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-06-21 21:42:54 -07:00
Andreea-Lupu 081ba0b2f2 fix periodic background tasks - gc and scrub 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-06-02 08:58:02 -07:00
Lisca Ana-Roberta 62775cc095 fixed failed tests for all skopeo versions 
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed failed tests for all skopeo versions

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

echo skopeo version

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo verifications modified makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

modified how to get digest and fixed makefile

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

skopeo failed tests fixed

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

changed function name

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed lost modifications

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>

fixed code coverage and dead code

Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-05-27 08:19:07 -07:00
Lisca Ana-Roberta e5a14670db code coverage improvement 
Signed-off-by: Lisca Ana-Roberta <ana.kagome@yahoo.com>
 2022-05-26 08:54:05 -07:00
Petu Eusebiu 5e22acbbc4 s3: added logic for deduping blobs 
Because s3 doesn't support hard links we store duplicated blobs
as empty files. When the original blob is deleted its content is
moved to the the next duplicated blob and so on.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-05-24 17:00:10 -07:00
Alex Stan d325c8b5f4 Fix problems signaled by new linter version v1.45.2 
PR (linter: upgrade linter version #405) triggered lint job which failed
with many errors generated by various linters. Configurations were added to
golangcilint.yaml and several refactorings were made in order to improve the
results of the linter.

maintidx linter disabled

Signed-off-by: Alex Stan <alexandrustan96@yahoo.ro>
 2022-04-27 09:55:44 -07:00
Andreea-Lupu 5e35dfa28f make gc periodic 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-04-18 10:25:29 -07:00
Petu Eusebiu c62dae06c9 s3: fix initRepo not creating index.json in some edge cases 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-04-14 15:06:40 -07:00
Catalin Hofnar 475d97b1ad Separate make commands that require sudo 
Reworked privileged cert test so it runs in Go by moving make logic to Go logic

Signed-off-by: Catalin Hofnar <catalin.hofnar@gmail.com>
 2022-04-05 10:12:43 -07:00
Andreea-Lupu 9454c77be2 make scrub inline and periodic 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2022-04-01 13:38:24 -07:00
Alexei Dodon ad519e2d3e Leave zot repositories in a consistent state after zot hits fd limit closes #359 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-03-31 13:25:15 -07:00
Ramkumar Chinchani b2a4388522 gc: add a unit test 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-03-24 12:02:52 -07:00
Petu Eusebiu f53dc9eb8d sync: Add a new flag to enforce syncing only signed images, closes #455 
sync: When checking if a image is already synced also check for changes in upstream signatures.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-24 10:50:01 -07:00
Eng Zer Jun 0d77b60de7 test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-03-07 12:28:49 -08:00
Petu Eusebiu 9cffbcaccb s3: bugfix, use sync.Map instead of map for storing multi part uploads references 
add storage lock in GetIndexContent

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-03 09:12:07 -08:00
Petu Eusebiu 45968e0bb7 sync: fix inconsistent test, used inject error fw for hard to reach test cases 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-03-01 09:56:27 -08:00
Ramkumar Chinchani 38a110314b gc: add a gcDelay param 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-14 14:03:22 -08:00
Ramkumar Chinchani d2aa016cdb storage: flush/sync contents to disk on file close 
Behavior controlled by configuration (default=off)
It is a trade-off between performance and consistency.

References:
[1] https://github.com/golang/go/issues/20599

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-07 12:08:01 -08:00
Ramkumar Chinchani c73e71b018 ci/cd: add a basic push-pull testing using client tools 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-07 10:03:24 -08:00
Ramkumar Chinchani 87084f286b storage: improve/fix oci image validation 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-02-02 13:31:41 -08:00
Ramkumar Chinchani 45fe129c63 notaryv2: fix 'notation list' 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-31 14:33:21 -08:00
Ramkumar Chinchani 4d576a4817 storage: return errors instead of panics 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2022-01-24 13:33:45 -08:00
Petu Eusebiu 1109bb4dde sync: Added support for syncing notary/cosign signatures, closes #261 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2022-01-13 08:45:59 -08:00
Alexei Dodon c4d34b7269 Added storage latency histogram metric 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2022-01-10 11:55:39 -08:00
Ramkumar Chinchani cac7fe4854 storage: use sha256-simd from minio 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-28 22:25:11 -08:00
Ramkumar Chinchani ac3801ea2d lint: upgrade golangci-lint 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-20 17:20:35 -08:00
Andreea-Lupu c61c3836db implement scrub to check manifest/blob integrity 
Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
 2021-12-09 11:18:09 -08:00
Ramkumar Chinchani 96226af869 move references to zotregistry.io and project-zot 
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-05 10:52:27 -08:00
Ramkumar Chinchani e42e42a2cc artifacts: initial support for artifacts/notaryv2 spec 
https://github.com/oras-project/artifacts-spec
https://github.com/notaryproject/notaryproject

Fixes issue #264

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
 2021-12-01 18:55:39 -08:00
Alexei Dodon e900b09cfb Fix data races in tests, closes #255 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-11-17 13:23:59 -08:00
Petu Eusebiu 9c568c0ee2 storage: add s3 backend support (without GC and dedupe) 
Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
 2021-11-15 08:09:00 -08:00
Alexei Dodon 8e4d828867 Implement an API for performance monitoring 
Signed-off-by: Alexei Dodon <adodon@cisco.com>
 2021-11-12 11:14:10 -08:00