Add kind-oidc.sh test and README

Co-authored-by: rchincha <45800463+rchincha@users.noreply.github.com>
2026-06-18 13:37:57 +08:00 · 2026-01-23 03:18:40 +00:00
parent a0dfe9fe3a
commit e6255a042b
2 changed files with 495 additions and 0 deletions
@@ -0,0 +1,161 @@
+# Kind Examples
+
+This directory contains scripts for running zot in Kubernetes using [kind](https://kind.sigs.k8s.io/) (Kubernetes IN Docker).
+
+## Scripts
+
+### kind-with-registry.sh
+
+A simple example that demonstrates:
+- Creating a kind cluster
+- Running a local registry (zot-minimal)
+- Configuring containerd to use the local registry
+- Deploying a sample application
+
+Usage:
+```bash
+./kind-with-registry.sh
+```
+
+### kind-ci.sh
+
+CI test that validates:
+- Kind cluster setup with local registry
+- Prometheus operator deployment
+- Zot extended deployment with metrics
+- Prometheus scraping of zot metrics
+
+Usage:
+```bash
+./kind-ci.sh
+```
+
+This script is used in the nightly CI workflow.
+
+### kind-oidc.sh
+
+Kubernetes OIDC authentication test that demonstrates:
+- Setting up a Dex OIDC provider
+- Creating a kind cluster with OIDC authentication configured for the Kubernetes API server
+- Deploying zot with OIDC authentication
+- Configuring RBAC for OIDC users
+
+This test validates that:
+1. The Kubernetes API server can authenticate users via OIDC
+2. Zot can be deployed and configured with OIDC authentication in Kubernetes
+3. The complete OIDC authentication flow works in a Kubernetes environment
+
+#### Prerequisites
+
+- Docker
+- kind (installed via `make` or available in `hack/tools/bin/kind`)
+- kubectl
+- openssl
+- skopeo
+
+#### Usage
+
+```bash
+./kind-oidc.sh
+```
+
+The script will:
+1. Generate TLS certificates for Dex
+2. Start a Dex OIDC provider in a Docker container
+3. Create a kind cluster with OIDC configuration
+4. Deploy zot with OIDC authentication enabled
+5. Configure RBAC for OIDC users
+6. Wait for user input before cleanup
+
+#### OIDC Configuration Details
+
+- **OIDC Provider**: Dex (https://dexidp.io/)
+- **Issuer URL**: `https://dex-server:10443/dex`
+- **Client ID**: `kubernetes`
+- **Test User**: `admin@example.com`
+- **Test Password**: `password`
+
+The Kubernetes API server is configured with the following OIDC flags:
+- `--oidc-issuer-url`: Points to the Dex server
+- `--oidc-client-id`: Set to "kubernetes"
+- `--oidc-username-claim`: Uses the "email" claim
+- `--oidc-groups-claim`: Uses the "groups" claim
+- `--oidc-ca-file`: CA certificate for TLS verification
+
+#### Testing OIDC Authentication
+
+To test OIDC authentication with kubectl, you can use [kubelogin](https://github.com/int128/kubelogin):
+
+```bash
+# Install kubelogin
+kubectl krew install oidc-login
+
+# Setup OIDC authentication
+kubectl oidc-login setup \
+  --oidc-issuer-url=https://localhost:10443/dex \
+  --oidc-client-id=kubernetes \
+  --oidc-client-secret=kubernetes-client-secret \
+  --certificate-authority=/tmp/kind-oidc/dex-ca.crt
+
+# Use OIDC authentication
+kubectl --user=oidc get nodes
+```
+
+#### Architecture
+
+```
+┌─────────────────────────────────────────────┐
+│  Host Machine                                │
+│                                              │
+│  ┌──────────────┐      ┌─────────────────┐  │
+│  │ Dex Server   │      │  Kind Cluster   │  │
+│  │ (Docker)     │◄─────┤  ┌───────────┐  │  │
+│  │              │      │  │ API Server│  │  │
+│  │ Port: 10443  │      │  │ + OIDC    │  │  │
+│  └──────────────┘      │  └───────────┘  │  │
+│                        │  ┌───────────┐  │  │
+│                        │  │ Zot Pod   │  │  │
+│                        │  │ + OIDC    │  │  │
+│                        │  └───────────┘  │  │
+│                        └─────────────────┘  │
+└─────────────────────────────────────────────┘
+```
+
+The Dex server and kind cluster are connected via Docker's "kind" network, allowing the Kubernetes API server to communicate with Dex for token validation.
+
+#### Cleanup
+
+The script automatically cleans up resources when it exits:
+- Deletes the kind cluster
+- Stops and removes the Dex container
+- Stops and removes the registry container
+
+#### References
+
+- [Kind documentation](https://kind.sigs.k8s.io/)
+- [Kubernetes OIDC authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens)
+- [Dex OIDC provider](https://dexidp.io/)
+- [kind-oidc example](https://github.com/int128/kind-oidc)
+
+## Common Issues
+
+### Certificate Errors
+
+If you encounter certificate errors, ensure that:
+- OpenSSL is installed and available
+- The certificates are generated correctly
+- The CA certificate is mounted to the correct path in the kind cluster
+
+### Network Issues
+
+If the Kubernetes API server cannot reach Dex:
+- Verify that Dex is running: `docker ps | grep dex-server`
+- Check that both containers are on the same network: `docker network inspect kind`
+- Verify DNS resolution in the control plane: `docker exec kind-oidc-control-plane cat /etc/hosts`
+
+### OIDC Token Validation Failures
+
+If OIDC authentication fails:
+- Check the API server logs: `docker logs kind-oidc-control-plane`
+- Verify the OIDC configuration: `kubectl --context kind-kind-oidc -n kube-system describe pod <api-server-pod>`
+- Ensure the issuer URL is accessible from within the cluster
@@ -0,0 +1,334 @@
+#!/bin/sh
+set -o errexit
+
+ROOT_DIR=$(git rev-parse --show-toplevel)
+KIND="${ROOT_DIR}"/hack/tools/bin/kind
+
+# Reference: https://github.com/int128/kind-oidc
+# This test validates Kubernetes OIDC authentication with zot registry
+
+# set no_proxy if applicable
+if [ ! -z "${no_proxy}" ]; then
+  echo "Updating no_proxy env var";
+  export no_proxy=${no_proxy},dex-server,kind-registry;
+  export NO_PROXY=${no_proxy};
+fi
+
+# Cleanup function
+cleanup() {
+  echo "Cleaning up..."
+  "${KIND}" delete cluster --name kind-oidc 2>/dev/null || true
+  docker stop dex-server 2>/dev/null || true
+  docker rm dex-server 2>/dev/null || true
+  docker stop kind-registry 2>/dev/null || true
+  docker rm kind-registry 2>/dev/null || true
+}
+
+# Set trap to cleanup on exit
+trap cleanup EXIT
+
+# Generate certificates for Dex
+echo "Generating certificates for Dex..."
+mkdir -p /tmp/kind-oidc
+cd /tmp/kind-oidc
+
+# Generate CA
+openssl genrsa -out dex-ca.key 2048
+openssl req -x509 -new -nodes -key dex-ca.key -days 365 -out dex-ca.crt -subj "/CN=dex-ca"
+
+# Generate server certificate
+openssl genrsa -out dex-server.key 2048
+openssl req -new -key dex-server.key -out dex-server.csr -subj "/CN=dex-server"
+openssl x509 -req -in dex-server.csr -CA dex-ca.crt -CAkey dex-ca.key -CAcreateserial -out dex-server.crt -days 365
+
+# Create Dex configuration
+cat > dex-config.yaml <<EOF
+issuer: https://dex-server:10443/dex
+
+storage:
+  type: memory
+
+web:
+  https: 0.0.0.0:10443
+  tlsCert: /dex-server.crt
+  tlsKey: /dex-server.key
+
+staticClients:
+- id: kubernetes
+  redirectURIs:
+  - http://localhost:8000
+  - http://localhost:18000
+  name: 'Kubernetes'
+  secret: kubernetes-client-secret
+
+enablePasswordDB: true
+
+staticPasswords:
+- email: admin@example.com
+  hash: "\$2a\$10\$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
+  username: admin
+  userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"
+EOF
+
+# Start Dex server
+echo "Starting Dex OIDC provider..."
+docker run -d --name dex-server \
+  -p 10443:10443 \
+  -v /tmp/kind-oidc/dex-config.yaml:/dex-config.yaml:ro \
+  -v /tmp/kind-oidc/dex-server.crt:/dex-server.crt:ro \
+  -v /tmp/kind-oidc/dex-server.key:/dex-server.key:ro \
+  ghcr.io/dexidp/dex:v2.41.1 \
+  serve /dex-config.yaml
+
+# Wait for Dex to start
+echo "Waiting for Dex to be ready..."
+for i in $(seq 1 30); do
+  if docker exec dex-server wget -qO- --no-check-certificate https://localhost:10443/dex/.well-known/openid-configuration > /dev/null 2>&1; then
+    echo "Dex is ready"
+    break
+  fi
+  echo "Waiting for Dex... ($i/30)"
+  sleep 2
+done
+
+# create registry container unless it already exists
+reg_name='kind-registry'
+reg_port='5001'
+if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then
+  docker run \
+    -d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}" \
+    ghcr.io/project-zot/zot-minimal-linux-amd64:latest
+fi
+
+CLUSTER_NAME=kind-oidc
+# Delete the cluster if it already exists
+"${KIND}" get clusters | grep ${CLUSTER_NAME} && "${KIND}" delete cluster --name ${CLUSTER_NAME}
+
+# Get Dex server IP address
+DEX_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' dex-server)
+echo "Dex server IP: ${DEX_IP}"
+
+# create a cluster with OIDC authentication enabled
+cat <<EOF | "${KIND}" create cluster --name ${CLUSTER_NAME} --config=-
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+kubeadmConfigPatches:
+- |
+  kind: ClusterConfiguration
+  apiVersion: kubeadm.k8s.io/v1beta3
+  apiServer:
+    extraArgs:
+      oidc-issuer-url: "https://dex-server:10443/dex"
+      oidc-client-id: "kubernetes"
+      oidc-ca-file: "/etc/kubernetes/pki/dex-ca.crt"
+      oidc-username-claim: "email"
+      oidc-groups-claim: "groups"
+    extraVolumes:
+    - name: dex-ca
+      hostPath: /tmp/kind-oidc/dex-ca.crt
+      mountPath: /etc/kubernetes/pki/dex-ca.crt
+      readOnly: true
+- |
+  kind: KubeletConfiguration
+  apiVersion: kubelet.config.k8s.io/v1beta1
+  cgroupDriver: systemd
+nodes:
+- role: control-plane
+  image: kindest/node:v1.28.7
+  extraMounts:
+  - hostPath: /tmp/kind-oidc/dex-ca.crt
+    containerPath: /etc/kubernetes/pki/dex-ca.crt
+    readOnly: true
+containerdConfigPatches:
+- |-
+  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
+    endpoint = ["http://${reg_name}:5000"]
+EOF
+
+# connect the registry to the cluster network if not already connected
+if [ "$(docker inspect -f='{{json .NetworkSettings.Networks.kind}}' "${reg_name}")" = 'null' ]; then
+  docker network connect "kind" "${reg_name}"
+fi
+
+# Connect Dex to kind network so API server can reach it
+docker network connect "kind" "dex-server"
+
+# Update /etc/hosts in the control plane to resolve dex-server
+echo "Configuring DNS for dex-server in cluster..."
+docker exec kind-oidc-control-plane sh -c "echo '${DEX_IP} dex-server' >> /etc/hosts"
+
+# Verify OIDC configuration
+echo "Verifying OIDC configuration in API server..."
+kubectl --context kind-kind-oidc get --raw /.well-known/openid-configuration 2>&1 || echo "OIDC discovery endpoint check (this may fail if not exposed)"
+
+# Document the local registry
+cat <<EOF | kubectl --context kind-kind-oidc apply -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: local-registry-hosting
+  namespace: kube-public
+data:
+  localRegistryHosting.v1: |
+    host: "localhost:${reg_port}"
+    help: "https://kind.sigs.k8s.io/docs/user/local-registry/"
+EOF
+
+# Create a cluster role binding for OIDC user
+echo "Creating RBAC for OIDC user..."
+kubectl --context kind-kind-oidc create clusterrolebinding oidc-cluster-admin \
+  --clusterrole=cluster-admin \
+  --user=admin@example.com || echo "ClusterRoleBinding already exists"
+
+# Build and deploy zot with OIDC configuration
+cd "${ROOT_DIR}"
+make oci-image
+
+# Copy the image to local registry
+COMMIT_HASH=$(git describe --always --tags --long)
+echo "Deploying zot-build:${COMMIT_HASH} image to local registry"
+skopeo copy --format=oci --dest-tls-verify=false oci:oci docker://localhost:5001/zot-build:${COMMIT_HASH}
+
+# Create ConfigMap with zot OIDC configuration
+kubectl --context kind-kind-oidc create configmap zot-oidc-config --from-literal=config.json='
+{
+  "distSpecVersion":"1.1.1",
+  "storage": {
+      "rootDirectory": "/var/lib/registry"
+  },
+  "http": {
+      "address": "0.0.0.0",
+      "port": "5000",
+      "auth": {
+          "openid": {
+              "providers": {
+                  "dex": {
+                      "issuer": "https://dex-server:10443/dex",
+                      "clientid": "kubernetes",
+                      "clientsecret": "kubernetes-client-secret",
+                      "scopes": ["openid", "email", "groups"]
+                  }
+              }
+          }
+      },
+      "accessControl": {
+          "repositories": {
+              "**": {
+                  "policies": [
+                      {
+                          "users": ["admin@example.com"],
+                          "actions": ["read", "create", "update", "delete"]
+                      }
+                  ],
+                  "defaultPolicy": []
+              }
+          }
+      }
+  },
+  "log": {
+      "level": "debug"
+  }
+}
+' --dry-run=client -o yaml | kubectl --context kind-kind-oidc apply -f -
+
+# Deploy zot
+kubectl --context kind-kind-oidc apply -f - <<EOF
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: zot-oidc
+  labels:
+    app: zot-oidc
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: zot-oidc
+  template:
+    metadata:
+      labels:
+        app: zot-oidc
+    spec:
+      containers:
+      - name: zot
+        image: localhost:5001/zot-build:${COMMIT_HASH}
+        imagePullPolicy: IfNotPresent
+        command: ["/usr/bin/zot"]
+        args: ["serve", "/etc/zot/config.json"]
+        ports:
+        - name: http
+          containerPort: 5000
+          protocol: TCP
+        volumeMounts:
+        - name: config
+          mountPath: /etc/zot
+          readOnly: true
+      volumes:
+      - name: config
+        configMap:
+          name: zot-oidc-config
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: zot-oidc
+spec:
+  selector:
+    app: zot-oidc
+  ports:
+  - protocol: TCP
+    port: 5000
+    targetPort: 5000
+EOF
+
+# Wait for deployment
+echo "Waiting for zot-oidc deployment to be ready..."
+kubectl --context kind-kind-oidc wait deployment -n default zot-oidc --for condition=Available=True --timeout=90s
+
+echo ""
+echo "=========================================="
+echo "Kind cluster with OIDC authentication is ready!"
+echo "=========================================="
+echo ""
+echo "Cluster name: kind-oidc"
+echo "Context: kind-kind-oidc"
+echo "Dex URL: https://dex-server:10443/dex (accessible from within cluster)"
+echo "Dex URL (external): https://localhost:10443/dex"
+echo "OIDC user: admin@example.com"
+echo "OIDC password: password"
+echo ""
+echo "To access the cluster with OIDC authentication, you would typically use a tool like kubelogin:"
+echo "  kubectl oidc-login setup --oidc-issuer-url=https://localhost:10443/dex \\"
+echo "    --oidc-client-id=kubernetes \\"
+echo "    --oidc-client-secret=kubernetes-client-secret \\"
+echo "    --certificate-authority=/tmp/kind-oidc/dex-ca.crt"
+echo ""
+echo "Zot registry is running at: http://localhost:5000 (via kubectl port-forward)"
+echo "To access: kubectl --context kind-kind-oidc port-forward svc/zot-oidc 5000:5000"
+echo ""
+
+# Validate the setup
+echo "Validating setup..."
+echo ""
+
+# Check pods
+echo "Cluster pods:"
+kubectl --context kind-kind-oidc get pods -A
+
+echo ""
+echo "Zot logs:"
+kubectl --context kind-kind-oidc logs -l app=zot-oidc --tail=50
+
+echo ""
+echo "=========================================="
+echo "Test completed successfully!"
+echo "=========================================="
+echo ""
+echo "Note: This test creates a kind cluster with OIDC authentication enabled."
+echo "The cluster will be deleted when the script exits."
+echo "Press Ctrl+C to cleanup and exit, or the cleanup will happen automatically."
+echo ""
+
+# Keep the script running so user can interact with the cluster
+echo "Press Enter to cleanup and exit..."
+read