1. Parse repos without metadata in ParseStorage
The timestamp check in ParseStorage was skipping repos that exist in
storage but don't have metadata. When GetRepoLastUpdated returns zero
time (no metadata), we should always parse the repo to create its
metadata. Check if metaLastUpdated is zero before comparing timestamps.
If zero, always parse regardless of storageLastUpdated.
2. Change the logic of how LastUpdated is computed in RepoSummary
It is not the latest tagged timestamp from the available images or
the last updated image created timestamp, based on whichever is the
latest.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
feat(meta): add TaggedTimestamp field and preserve during re-parsing
Add TaggedTimestamp field to track when image tags were created, exposed
through GraphQL API. Previously, when zot restarted and re-parsed storage,
ResetRepoReferences would clear all tags, causing timestamp information to
be lost and reset to the service restart time for existing images.
This change adds TaggedTimestamp support and modifies ResetRepoReferences to
selectively preserve tags that still exist in storage, maintaining their
TaggedTimestamp values. Tags that no longer exist in storage are removed as
before.
Changes:
- Add TaggedTimestamp field to GraphQL ImageSummary schema
- Update GraphQL conversion functions to populate TaggedTimestamp with
fallback to PushTimestamp when unavailable
- Updated ResetRepoReferences interface to accept tagsToKeep parameter
- Modified ParseRepo to collect tags from storage before resetting
- Updated all backend implementations (Redis, DynamoDB, BoltDB) to preserve
tags in tagsToKeep instead of clearing all tags
- Updated tests and mocks to match new signature
This ensures TaggedTimestamp accurately reflects when tags were originally
created, and exposes this information through the GraphQL API.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* ci: add a upgrade bats test
Fixes https://github.com/project-zot/zot/issues/3601
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* ci: first check existing images
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* ci: refactor into common test code
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
* refactor: Refactor upgrade tests to improve readability and maintainability
- Added section headers for release tests, upgrade process, and new tests in upgrade.bats and upgrade_minimal.bats.
- Replaced inline port retrieval with a function call to get_zot_port for consistency.
- Consolidated repeated test logic into dedicated functions (e.g., test_new_existing_pull_image, test_new_push_image).
- Removed unnecessary variable assignments and streamlined test cases for clarity.
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
---------
Signed-off-by: Ramkumar Chinchani <rchincha.dev@gmail.com>
Add support for configurable identity attributes in mTLS authentication,
allowing identity extraction from CommonName, Subject DN, Email SAN,
URI SAN, or DNSName SAN with fallback chain support. Includes regex
pattern matching for URI SANs (e.g., SPIFFE workload IDs).
- Add MTLSConfig with identity attributes, URISANPattern, and index fields
- Implement extractMTLSIdentity with fallback chain logic
- Move the mtls tests in the api package to pkg/api/mtls_test.go
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: prevent nil pointer dereference in RemoveImageFromRepoMeta
This commit fixes a critical bug where RemoveImageFromRepoMeta crashes with
a nil pointer dereference during retention policy execution and GC operations.
Root Cause:
The function was accessing blob metadata without checking if it exists first.
During GC/retention operations, the metadata database might have stale
references to blobs that no longer exist, causing runtime panics.
Changes:
- Added nil check for descriptorBlobInfo before accessing LastUpdated field
- Added nil check for blobInfo before dereferencing Size, Vendors, Platforms, and SubBlobs
- Made the function consistent with recalculateAggregateFields which already had these checks
Impact:
- Fixes crashes during retention policy execution
- Fixes crashes during GC manifest removal
- Fixes image deletion failures via API
- Eliminates need for dryRun: true workaround in retention config
The fix gracefully handles missing blob metadata by skipping those entries
instead of crashing.
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
* test: add comprehensive tests for RemoveImageFromRepoMeta nil checks
Add test coverage for the nil pointer dereference fixes in RemoveImageFromRepoMeta.
These tests ensure the function handles missing blob metadata gracefully during
GC and retention operations.
Test cases:
- Handle nil blob info for descriptor digest (line 280 check)
- Handle nil blob info in queue traversal (line 297 check)
- Verify correct behavior with valid blob info
- Handle empty tags edge case
- Skip tags with empty digest
Coverage: RemoveImageFromRepoMeta now has 100% test coverage
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
* test: fix RemoveImageFromRepoMeta tests to match actual usage
Address review feedback:
- Delete tag from repoMeta.Tags before calling RemoveImageFromRepoMeta
- Fix blob count expectations after tag removal
- Add assertion to verify tag was removed from metadata
- Update comments to clarify expected behavior
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
* test: add tag removal assertion to second test case
Add missing assertion to verify tag1 was removed from resultMeta.Tags
in the 'should handle nil blob info in queue traversal' test.
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
* refactor: improve nil blob handling documentation and test coverage
Address Copilot review feedback:
- Expand comment at line 278 to explain implications of skipping tags
with missing blob info, clarifying that metadata inconsistency is
acceptable in GC/cleanup scenarios
- Revise 'should handle nil blob info for descriptor digest' test to
cover more realistic scenario: remove tag1 while tag2 has missing
blob info, demonstrating graceful handling of data inconsistencies
in remaining tags during removal operations
All tests pass with 49 total assertions.
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
* fix: prevent nil pointer in GetCandidates when statistics missing
Add defensive check in pkg/retention/candidate.go to handle cases where
a tag exists in repoMeta.Tags but has no corresponding entry in
repoMeta.Statistics. This prevents incorrect retention decisions based
on zero-value timestamps.
Changes:
- Check statistics existence before creating candidates
- Skip tags with missing statistics (retained by GetRetainedTagsFromMetaDB)
- Improve performance from O(n*m) to O(n) by using direct map lookup
- Add comprehensive test coverage for missing statistics scenarios
This addresses the concern raised in PR #3658 about metadata
inconsistencies due to non-transactional writes to blob store and metaDB.
Related: #3658
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
* test: achieve 100% coverage for RemoveImageFromRepoMeta nil checks
Enhance test coverage for RemoveImageFromRepoMeta to address codecov failures
by adding comprehensive test cases that exercise all code paths including nil
pointer checks and continue statements.
Changes:
- Enhanced 'nil blob info for descriptor digest' test to verify processing
continues with other valid tags after skipping nil entries
- Enhanced 'nil blob info in queue traversal' test to handle mixed valid/nil
sub-blobs and verify correct processing continuation
- Added 'multiple nil blobs in deeply nested structure' test to cover complex
scenarios with multiple missing blobs at various nesting levels
- Enhanced 'skip tags with empty digest' test to verify processing continues
with valid tags after skipping empty digest entries
- Added 'combined edge cases' test to verify all edge cases work together:
empty digest, nil descriptor blob, and nil queue blob
Coverage Results:
- RemoveImageFromRepoMeta: 100.0% line coverage (was 87.50%)
- All 7 test scenarios pass with 75 total assertions
- All nil check code paths fully exercised
- All continue statement behaviors validated
Fixes codecov/patch failure on PR #3658 where 2 lines were missing coverage.
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
---------
Signed-off-by: Gianluca Boiano <morf3089@gmail.com>
- Refactored HTTP client from global cache to struct-based approach (global state was shared between tests, including what certificates to use)
- Enhanced pkg/test/tls to support ECDSA and ED25519 key types
- Replaced static certificate files with dynamic generation in golang tests
- Fixed test cleanup issues and improved resource management
This eliminates dependency on external cert generation scripts and
improves test maintainability.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* feat: support mTLS-only authn/authz with AccessControl and allow combining mTLS with other auth mechanisms
Signed-off-by: Ivan Arkhipov <me@endevir.ru>
* refactor: improve authentication logic and TLS certificate generation
- Fix mTLS authentication to use only leaf certificate instead of iterating
through all certificates in the chain
- Reject Authorization headers when corresponding auth method is disabled,
regardless of mTLS status (security improvement)
- Simplify authentication switch statement ordering and logic
- Move ErrUserDataNotFound error handling into sessionAuthn method
- Refactor TLS certificate generation to use Options pattern with
CertificateOptions struct for better extensibility
- Consolidate duplicate certificate generation code into helper functions
(generateCertificate, parseCA, initializeTemplate, applyOptions)
- Rename certificate generation functions for clarity:
- GenerateCertWithCN -> GenerateClientCert
- GenerateSelfSignedCertWithCN -> GenerateClientSelfSignedCert
- Add support for SAN settings including email addresses in certificates
- Update tests to reflect new authentication behavior and certificate API
This commit improves both the security posture (rejecting disabled auth
methods) and code maintainability (consolidated certificate generation).
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
* fix: guard against multiple Authorization headers
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
---------
Signed-off-by: Ivan Arkhipov <me@endevir.ru>
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Co-authored-by: Ivan Arkhipov <me@endevir.ru>
See: https://github.com/project-zot/zot/issues/3560#issuecomment-3594856118
What happens is:
- syncRef skips the image ("skipping image because it's already synced")
- syncReferrers doesn't sync anything
- CommitAll is still called even though nothing was synced
- The temp directory exists but is empty (no index.json, no blobs)
- CommitAll fails because index.json is missing
Let's ensure we properly check for errors, and skip the log messages if some of the cases.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
Replace MakeTempFile usage with MakeTempFilePath and MakeTempFileWithContent
helpers that automatically handle file lifecycle. This prevents resource
leaks by ensuring temporary files are properly closed.
Shoudld also make the tests easier to read.
Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>
copilot-swe-agent[bot]