Commit Graph

20 Commits

Author SHA1 Message Date
Piaras Hoban bc5fd1a357 feat(events): add events extension (#3045)
* feat: add events config

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* feat: implement event support with log sink

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* feat: integrate events and update tests

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* refactor: update event config

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* feat: implement http and nats sinks. remove log sink

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* refactor: events extension setup

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: cleanup tests to use nil event recorder

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: update events config example and add more logging

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* refactor: better use of build tags for minimal binary

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* fix: missing store param in evelated privileges tests

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* fix: regression in config decoding

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: update check logs script to enable cross-platform usage via GREP_BIN_PATH envvar

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: fix log lint issue for events

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: fix failing events disabled test

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* test: add blackbox tests for events

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: specify architecture when downloading binaries in Makefile

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: improve failure handling when no valid sinks are provided

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* test: fix data race in events test

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: cleanup event decoding

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* test: fix logging tests

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* test: make nats server test more reliable

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: go mod cleanup

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* test: add sleep when setting up nats client

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* fix: ensure event sink errors do not propogate

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* test: increase coverage for events

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* feat(events): Refactor events to be non-blocking from caller.

Signed-off-by: Asgeir Nilsen <asgeir.nilsen@bouvet.no>
Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: remove harded-coded linux

Co-authored-by: Andrei Aaron <andreifdaaron@gmail.com>
Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* feat(events): fail to start if incorrect event sink is configured

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* test: allow cli tests to return errors instead of panic

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

* chore: bump nats server to v2.11.3

Signed-off-by: Piaras Hoban <phoban01@gmail.com>

---------

Signed-off-by: Piaras Hoban <phoban01@gmail.com>
Signed-off-by: Asgeir Nilsen <asgeir.nilsen@bouvet.no>
Co-authored-by: Asgeir Nilsen <asgeir.nilsen@bouvet.no>
Co-authored-by: Andrei Aaron <andreifdaaron@gmail.com>
2025-05-02 12:30:06 -07:00
peusebiu 0e2aa81439 feat(sync): use regclient for sync extension (#2903)
* feat(sync): use regclient for sync extension

replaced containers/image package with regclient/regclient package

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* fix(sync): fixed converting innner docker list mediatype

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* feat(sync): added option to preserve digest

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* fix(sync): added coverage and various fixes

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* fix(metadb): fixed converting manifest list not setting platform and annotations

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* fix(sync): remove read lock on storage, not used concurrently

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* feat(sync): added cache for repo tags

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* fix(sync): fixed Makefile
removed opengpg tag

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

* fix(sync): add test for on demand referrer

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>

---------

Signed-off-by: Eusebiu Petu <petu.eusebiu@gmail.com>
2025-04-15 16:58:15 -07:00
Ramkumar Chinchani cb2af94b0b feat: add support for docker images (#2714)
* feat: add support for docker images

Issue #724

A new config section under "HTTP" called "Compat" is added which
currently takes a list of possible compatible legacy media-types.

https://github.com/opencontainers/image-spec/blob/main/media-types.md#compatibility-matrix

Only "docker2s2" (Docker Manifest V2 Schema V2) is currently supported.

Garbage collection also needs to be made aware of non-OCI compatible
layer types.
feat: add cve support for non-OCI compatible layer types

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* 

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* test: add more docker compat tests

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* feat: add additional validation checks for non-OCI images

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* ci: make "full" images docker-compatible

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-10-31 09:44:04 +02:00
Andrei Aaron a10c5fa7ab test: add more tests for GCing indexes referencing other indexes referencing manifests (#2716)
Looks like we didn't have many GC tests for retaining multiarch images.
I added more data to the existing image retention tests, besides the new GC tests.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-10-11 20:31:48 +03:00
Andrei Aaron 253aad3195 fix(gc): gc now removes blob uploads which have not changed within the gc delay interval (#2599)
See #2598

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-08-12 11:58:46 -07:00
Jan-Otto Kröpke fa4b69954d build(deps): bump all dependencies (#2532)
* build(deps): bump all dependencies

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

* build(deps): bump all dependencies

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): bump all dependencies

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): bump all dependencies

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

---------

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>
2024-08-02 14:23:53 -07:00
Jan-Otto Kröpke f618b1d4ef ci(deps): upgrade golangci-lint (#2556)
* ci(deps): upgrade golangci-lint

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

* build(deps): removed disabled linters

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>

* build(deps): go run github.com/daixiang0/gci@latest write .

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run golang.org/x/tools/cmd/goimports@latest -l -w .

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run github.com/bombsimon/wsl/v4/cmd...@latest -strict-append -test=true -fix ./...

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run github.com/catenacyber/perfsprint@latest -fix ./...

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): replace gomnd by mnd

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): make gqlgen

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build: Revert "build(deps): go run github.com/daixiang0/gci@latest write ."

This reverts commit 5bf8c42e1f.

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): go run github.com/daixiang0/gci@latest write -s 'standard' -s default -s 'prefix(zotregistry.dev/zot)' .

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* build(deps): make gqlgen

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: wsl issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: check-log issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: gci issues

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

* fix: tests

Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>

---------

Signed-off-by: Jan-Otto Kröpke <mail@jkroepke.de>
Signed-off-by: Jan-Otto Kröpke <joe@cloudeteer.de>
2024-07-29 10:32:51 -07:00
Andrei Aaron 26be383aae fix(digests): do not mandate sha256 as the only algorithm used for hashing blobs (#2075)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-07-19 09:56:31 -07:00
Ramkumar Chinchani 18235ca254 fix(oras)!: remove ORAS artifact references support (#2294)
* fix(oras)!: remove ORAS artifact references support

ORAS artifacts/references predated OCI dist-spec 1.1.0 which now has the
same functionality and likely to see wider adoption.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

* test: update to released official images

So that they are unlikely to be deleted.
*-rc images may be cleaned up over time.

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>

---------

Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
2024-03-06 12:16:42 -08:00
Andrei Aaron 6a83dd47c0 fix(scheduler): the session cleanup generator is reset too often (#2220)
This causes the "fair" scheduler to run it too often in the detriment of other generators.
The intention was to run it every 2 hours but the measurement unit for 7200 was not specified.

Add more logs, including showing a generator name, in order to troubleshoot this kind of issues easier in the future.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-02-01 09:15:53 -08:00
Andrei Aaron ce4924f841 refactor: rename go module from zotregistry.io/zot to zotregistry.dev/zot (#2187)
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2024-01-31 20:34:07 -08:00
LaurentiuNiculae 79e14027ee refactor(test): add lint rule for messages starting with the component (#2045)
Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-12-08 10:05:02 +02:00
Alexei Dodon 2e733b3f4f feat(metrics): add scheduler related metrics (#2076)
Signed-off-by: Alexei Dodon <adodon@cisco.com>
2023-12-04 14:13:50 -08:00
peusebiu 6222dae1f0 feat(scheduler): gracefully shutdown (#1951)
wait for workers to finish before exiting

should fix tests reporting they couldn't remove rootDir because it's being
written by tasks

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-24 00:40:10 -08:00
LaurentiuNiculae 4fb1e756c4 feat(startup): update logic for metadb update on startup, skip unmodified repos (#2024)
- MetaDB stores the time of the last update of a repo
- During startup we check if the layout has been updated after the last recorded change in the db
- If this is the case, the repo is parsed and updated in the DB otherwise it's skipped

Signed-off-by: Laurentiu Niculae <niculae.laurentiu1@gmail.com>
2023-11-16 10:39:27 -08:00
Andreea Lupu d5065513f5 feat: add support for oci1.1 cosign signatures(using referrers) (#1963)
- Cosign supports 2 types of signature formats:

	1. Using tag -> each new signature of the same manifest is
	added as a new layer of the signature manifest having that
	specific tag("{alghoritm}-{digest_of_signed_manifest}.sig")

	2. Using referrers -> each new signature of the same manifest is
	added as a new manifest

- For adding these cosign signature to metadb, we reserved index 0 of the
list of cosign signatures for tag-based signatures. When a new tag-based
signature is added for the same manifest, the element on first position
in its list of cosign signatures(in metadb) will be updated/overwritten.
When a new cosign signature(using referrers) will be added for the same
manifest this new signature will be appended to the list of cosign
signatures.

Signed-off-by: Andreea-Lupu <andreealupu1470@yahoo.com>
2023-11-06 14:09:39 -08:00
peusebiu 9074f8483b feat(retention): added image retention policies (#1866)
feat(metaDB): add more image statistics info

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-11-01 09:16:18 -07:00
Andrei Aaron ba6f347d8d refactor(pkg/test): split logic in pkg/test/common.go into multiple packages (#1861)
Which could be imported independently. See more details:
1. "zotregistry.io/zot/pkg/test/common" - currently used as
   tcommon "zotregistry.io/zot/pkg/test/common" - inside pkg/test
   test "zotregistry.io/zot/pkg/test/common" - in tests
   . "zotregistry.io/zot/pkg/test/common" - in tests
Decouple zb from code in test/pkg in order to keep the size small.

2. "zotregistry.io/zot/pkg/test/image-utils" - curently used as
   . "zotregistry.io/zot/pkg/test/image-utils"

3. "zotregistry.io/zot/pkg/test/deprecated" -  curently used as
   "zotregistry.io/zot/pkg/test/deprecated"
This one will bre replaced gradually by image-utils in the future.

4. "zotregistry.io/zot/pkg/test/signature" - (cosign + notation) use as
   "zotregistry.io/zot/pkg/test/signature"

5. "zotregistry.io/zot/pkg/test/auth" - (bearer + oidc)  curently used as
   authutils "zotregistry.io/zot/pkg/test/auth"

 6. "zotregistry.io/zot/pkg/test/oci-utils" -  curently used as
   ociutils "zotregistry.io/zot/pkg/test/oci-utils"

Some unused functions were removed, some were replaced, and in
a few cases specific funtions were moved to the files they were used in.

Added an interface for the StoreController, this reduces the number of imports
of the entire image store, decreasing binary size for tests.
If the zb code was still coupled with pkg/test, this would have reflected in zb size.

Signed-off-by: Andrei Aaron <aaaron@luxoft.com>
2023-09-27 11:34:48 -07:00
peusebiu c3801dc3d3 fix(dedupe): run dedupe only for repositories found at startup (#1844)
no need to run dedupe/restore blobs for images being pushed or synced while
running dedupe task, they are already deduped/restored inline.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-27 11:15:08 -07:00
peusebiu 1df743f173 fix(gc): sync repodb when gc'ing manifests (#1819)
fix(gc): fix cleaning deduped blobs because they have the modTime of
the original blobs, fixed by updating the modTime when hard linking
the blobs.
fix(gc): failing to parse rootDir at zot startup when using s3 storage
because there are no files under rootDir and we can not create empty dirs
on s3, fixed by creating an empty file under rootDir.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
2023-09-22 11:51:20 -07:00