diff --git a/examples/kind/README.md b/examples/kind/README.md new file mode 100644 index 00000000..49f85e54 --- /dev/null +++ b/examples/kind/README.md @@ -0,0 +1,161 @@ +# Kind Examples + +This directory contains scripts for running zot in Kubernetes using [kind](https://kind.sigs.k8s.io/) (Kubernetes IN Docker). + +## Scripts + +### kind-with-registry.sh + +A simple example that demonstrates: +- Creating a kind cluster +- Running a local registry (zot-minimal) +- Configuring containerd to use the local registry +- Deploying a sample application + +Usage: +```bash +./kind-with-registry.sh +``` + +### kind-ci.sh + +CI test that validates: +- Kind cluster setup with local registry +- Prometheus operator deployment +- Zot extended deployment with metrics +- Prometheus scraping of zot metrics + +Usage: +```bash +./kind-ci.sh +``` + +This script is used in the nightly CI workflow. + +### kind-oidc.sh + +Kubernetes OIDC authentication test that demonstrates: +- Setting up a Dex OIDC provider +- Creating a kind cluster with OIDC authentication configured for the Kubernetes API server +- Deploying zot with OIDC authentication +- Configuring RBAC for OIDC users + +This test validates that: +1. The Kubernetes API server can authenticate users via OIDC +2. Zot can be deployed and configured with OIDC authentication in Kubernetes +3. The complete OIDC authentication flow works in a Kubernetes environment + +#### Prerequisites + +- Docker +- kind (installed via `make` or available in `hack/tools/bin/kind`) +- kubectl +- openssl +- skopeo + +#### Usage + +```bash +./kind-oidc.sh +``` + +The script will: +1. Generate TLS certificates for Dex +2. Start a Dex OIDC provider in a Docker container +3. Create a kind cluster with OIDC configuration +4. Deploy zot with OIDC authentication enabled +5. Configure RBAC for OIDC users +6. Wait for user input before cleanup + +#### OIDC Configuration Details + +- **OIDC Provider**: Dex (https://dexidp.io/) +- **Issuer URL**: `https://dex-server:10443/dex` +- **Client ID**: `kubernetes` +- **Test User**: `admin@example.com` +- **Test Password**: `password` + +The Kubernetes API server is configured with the following OIDC flags: +- `--oidc-issuer-url`: Points to the Dex server +- `--oidc-client-id`: Set to "kubernetes" +- `--oidc-username-claim`: Uses the "email" claim +- `--oidc-groups-claim`: Uses the "groups" claim +- `--oidc-ca-file`: CA certificate for TLS verification + +#### Testing OIDC Authentication + +To test OIDC authentication with kubectl, you can use [kubelogin](https://github.com/int128/kubelogin): + +```bash +# Install kubelogin +kubectl krew install oidc-login + +# Setup OIDC authentication +kubectl oidc-login setup \ + --oidc-issuer-url=https://localhost:10443/dex \ + --oidc-client-id=kubernetes \ + --oidc-client-secret=kubernetes-client-secret \ + --certificate-authority=/tmp/kind-oidc/dex-ca.crt + +# Use OIDC authentication +kubectl --user=oidc get nodes +``` + +#### Architecture + +``` +┌─────────────────────────────────────────────┐ +│ Host Machine │ +│ │ +│ ┌──────────────┐ ┌─────────────────┐ │ +│ │ Dex Server │ │ Kind Cluster │ │ +│ │ (Docker) │◄─────┤ ┌───────────┐ │ │ +│ │ │ │ │ API Server│ │ │ +│ │ Port: 10443 │ │ │ + OIDC │ │ │ +│ └──────────────┘ │ └───────────┘ │ │ +│ │ ┌───────────┐ │ │ +│ │ │ Zot Pod │ │ │ +│ │ │ + OIDC │ │ │ +│ │ └───────────┘ │ │ +│ └─────────────────┘ │ +└─────────────────────────────────────────────┘ +``` + +The Dex server and kind cluster are connected via Docker's "kind" network, allowing the Kubernetes API server to communicate with Dex for token validation. + +#### Cleanup + +The script automatically cleans up resources when it exits: +- Deletes the kind cluster +- Stops and removes the Dex container +- Stops and removes the registry container + +#### References + +- [Kind documentation](https://kind.sigs.k8s.io/) +- [Kubernetes OIDC authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens) +- [Dex OIDC provider](https://dexidp.io/) +- [kind-oidc example](https://github.com/int128/kind-oidc) + +## Common Issues + +### Certificate Errors + +If you encounter certificate errors, ensure that: +- OpenSSL is installed and available +- The certificates are generated correctly +- The CA certificate is mounted to the correct path in the kind cluster + +### Network Issues + +If the Kubernetes API server cannot reach Dex: +- Verify that Dex is running: `docker ps | grep dex-server` +- Check that both containers are on the same network: `docker network inspect kind` +- Verify DNS resolution in the control plane: `docker exec kind-oidc-control-plane cat /etc/hosts` + +### OIDC Token Validation Failures + +If OIDC authentication fails: +- Check the API server logs: `docker logs kind-oidc-control-plane` +- Verify the OIDC configuration: `kubectl --context kind-kind-oidc -n kube-system describe pod ` +- Ensure the issuer URL is accessible from within the cluster diff --git a/examples/kind/kind-oidc.sh b/examples/kind/kind-oidc.sh new file mode 100755 index 00000000..67e56daa --- /dev/null +++ b/examples/kind/kind-oidc.sh @@ -0,0 +1,334 @@ +#!/bin/sh +set -o errexit + +ROOT_DIR=$(git rev-parse --show-toplevel) +KIND="${ROOT_DIR}"/hack/tools/bin/kind + +# Reference: https://github.com/int128/kind-oidc +# This test validates Kubernetes OIDC authentication with zot registry + +# set no_proxy if applicable +if [ ! -z "${no_proxy}" ]; then + echo "Updating no_proxy env var"; + export no_proxy=${no_proxy},dex-server,kind-registry; + export NO_PROXY=${no_proxy}; +fi + +# Cleanup function +cleanup() { + echo "Cleaning up..." + "${KIND}" delete cluster --name kind-oidc 2>/dev/null || true + docker stop dex-server 2>/dev/null || true + docker rm dex-server 2>/dev/null || true + docker stop kind-registry 2>/dev/null || true + docker rm kind-registry 2>/dev/null || true +} + +# Set trap to cleanup on exit +trap cleanup EXIT + +# Generate certificates for Dex +echo "Generating certificates for Dex..." +mkdir -p /tmp/kind-oidc +cd /tmp/kind-oidc + +# Generate CA +openssl genrsa -out dex-ca.key 2048 +openssl req -x509 -new -nodes -key dex-ca.key -days 365 -out dex-ca.crt -subj "/CN=dex-ca" + +# Generate server certificate +openssl genrsa -out dex-server.key 2048 +openssl req -new -key dex-server.key -out dex-server.csr -subj "/CN=dex-server" +openssl x509 -req -in dex-server.csr -CA dex-ca.crt -CAkey dex-ca.key -CAcreateserial -out dex-server.crt -days 365 + +# Create Dex configuration +cat > dex-config.yaml < /dev/null 2>&1; then + echo "Dex is ready" + break + fi + echo "Waiting for Dex... ($i/30)" + sleep 2 +done + +# create registry container unless it already exists +reg_name='kind-registry' +reg_port='5001' +if [ "$(docker inspect -f '{{.State.Running}}' "${reg_name}" 2>/dev/null || true)" != 'true' ]; then + docker run \ + -d --restart=always -p "127.0.0.1:${reg_port}:5000" --name "${reg_name}" \ + ghcr.io/project-zot/zot-minimal-linux-amd64:latest +fi + +CLUSTER_NAME=kind-oidc +# Delete the cluster if it already exists +"${KIND}" get clusters | grep ${CLUSTER_NAME} && "${KIND}" delete cluster --name ${CLUSTER_NAME} + +# Get Dex server IP address +DEX_IP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' dex-server) +echo "Dex server IP: ${DEX_IP}" + +# create a cluster with OIDC authentication enabled +cat <> /etc/hosts" + +# Verify OIDC configuration +echo "Verifying OIDC configuration in API server..." +kubectl --context kind-kind-oidc get --raw /.well-known/openid-configuration 2>&1 || echo "OIDC discovery endpoint check (this may fail if not exposed)" + +# Document the local registry +cat <