github/zot
1
0
Fork 0
mirror of https://github.com/project-zot/zot.git synced 2026-06-15 20:07:55 +08:00
Code Issues Packages Projects Releases Wiki Activity

s3: added logic for deduping blobs

Browse Source 
Because s3 doesn't support hard links we store duplicated blobs
as empty files. When the original blob is deleted its content is
moved to the the next duplicated blob and so on.

Signed-off-by: Petu Eusebiu <peusebiu@cisco.com>
This commit is contained in:
Petu Eusebiu
2022-04-12 13:01:04 +03:00
committed by Ramkumar Chinchani
parent ad08c08986
commit 5e22acbbc4
11 changed files with 1027 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
examples/README.md
+6 -2
View File
@@ -308,13 +308,15 @@ zot also supports different storage drivers for each subpath.
### Specifying S3 credentials
There are multiple ways to specify S3 credentials:
- Config file:
```
"storage": {
"rootDirectory": "/tmp/zot", # local path used to store dedupe cache database
"dedupe": true,
"storageDriver": {
"name": "s3",
"rootdirectory": "/zot", # this is a prefix that is applied to all S3 keys to allow you to segment data in your bucket if necessary.
"region": "us-east-2",
"bucket": "zot-storage",
"secure": true,
@@ -324,6 +326,8 @@ There are multiple ways to specify S3 credentials:
}
```
There are multiple ways to specify S3 credentials besides config file:
- Environment variables:
SDK looks for credentials in the following environment variables:
examples/config-s3.json
+12 -4
View File
@@ -1,9 +1,11 @@
{
"distSpecVersion": "1.0.1-dev",
"storage": {
"rootDirectory": "/zot",
"rootDirectory": "/tmp/zot",
"dedupe": true,
"storageDriver": {
"name": "s3",
"rootdirectory": "/zot",
"region": "us-east-2",
"bucket": "zot-storage",
"secure": true,
@@ -11,9 +13,11 @@
},
"subPaths": {
"/a": {
"rootDirectory": "/zot-a",
"rootDirectory": "/tmp/zot1",
"dedupe": false,
"storageDriver": {
"name": "s3",
"rootdirectory": "/zot-a",
"region": "us-east-2",
"bucket": "zot-storage",
"secure": true,
@@ -21,9 +25,11 @@
}
},
"/b": {
"rootDirectory": "/zot-b",
"rootDirectory": "/tmp/zot2",
"dedupe": true,
"storageDriver": {
"name": "s3",
"rootdirectory": "/zot-b",
"region": "us-east-2",
"bucket": "zot-storage",
"secure": true,
@@ -31,9 +37,11 @@
}
},
"/c": {
"rootDirectory": "/zot-c",
"rootDirectory": "/tmp/zot3",
"dedupe": true,
"storageDriver": {
"name": "s3",
"rootdirectory": "/zot-c",
"region": "us-east-2",
"bucket": "zot-storage",
"secure": false,