diff --git a/README.md b/README.md new file mode 100644 index 0000000..b56e0b3 --- /dev/null +++ b/README.md @@ -0,0 +1,194 @@ +# Kazan - Kubernetes cluster + +On prem self-hosted kubernetes cluster running on [microk8s](https://microk8s.io/docs). + +![cube gif](./assets/cube.gif) + +## Table of Contents + +- [Adding nodes](#adding-nodes) +- [Plugins](#plugins) + - [MetalLB](#metallb) + - [Helm](#helm) +- [Persistent Storage](#persistent-storage) +- [Ingress](#ingress) + - [Configuration](#configuration) + - [Deployment](#deployment) + +# Adding nodes + +## Provision and install microk8s +If this is first or additional node to be used for kubernetes we want to run base server configuration using playbook, replacing `KUBERNETES_HOST`: + +```bash +ansible-playbook plays/base_server_setup.yml -i schleppe.ini -l KUBERNETES_HOST +``` + +Install microk8s using snap: + +```bash +sudo snap install microk8s --classic --channel=1.27/stable +``` + +## Joining cluster + +From a existing node run: + +```bash +microk8s.add-node +``` + +Copy the output and join using command similar to the following on the new kubernetes node: + +```bash +microk8s join kazan.schleppe:25000/60e891acfc38556ea569cd15d5e025a1/91a7ab376757 +``` + +# Plugins + +## MetalLB + +MetallLB hooks into Kubernetes cluster and provides a network load-balancer implementation. + +```bash +microk8s.enable metallb:10.0.0.150-10.0.0.154 +``` + +## Helm + +Use helm charts to easier install managed systems and applications. + +```bash +microk8s.enable helm +``` + + +# Persistent storage + +To create a persistent volume claim (PVC) we need to first define a storage class. To not solve the issue of HA stoarge, a single NFS host is used. Setup expects a linux server service export path: `/srv/nfs`. + +## Setup/Deployment + +Define storage class pointing to NFS server: + +```yaml +kubectl apply -f storage-nfs/storageClass.yml +``` + +> [!NOTE] +> PVC resources will reference `spec.storageClassName: nfs-csi` + +# Ingress + +Traefik is configured as default ingress, enabling dynamic configuration & SSL generation. + + +## Configuration + +> [!IMPORTANT] +> Make sure that persistent storage is configured. Traefik is configured to expect `nfs-csi` storageClass already exists, follow steps above. + +### Disable nginx ingress + +Microk8s uses nginx as default ingress controller. We wish to replace this with traefik. Do disable plugin run: + +```bash +microk8s.disable ingress +``` + +### Configure cloudflare certificate resolver + +Add to `additionalArguments`: + +```yaml + - --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare + - --certificatesresolvers.cloudflare.acme.email=YOUR_EMAIL_HERE + - --certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1 + - --certificatesresolvers.cloudflare.acme.storage=/ssl-certs/acme-cloudflare.json +``` + +It is important that persistent storage is configured and a `acme-cloudflare.json` file is manually touched and it's permissions updated. From the nfs server run: + +```bash +cd /srv/nfs/PVC_VOLUME_NAME_HERE +sudo touch acme-cloudflare.json +sudo chown 65532:65532 acme-cloudflare.json +``` + +### Configure cloudflare credentials secret + +Add to `env`: + +```yaml + - name: CF_API_EMAIL + valueFrom: + secretKeyRef: + key: email + name: cloudflare-credentials + - name: CF_API_KEY + valueFrom: + secretKeyRef: + key: apiKey + name: cloudflare-credentials +``` + +Update following values from example secret file (`ingress-traefik/cloudflare-credentials.yml`): + +```yaml + email: YOUR_CLOUDFLARE_EMAIL + apiKey: YOUR_CLOUDFLARE_API_KEY +``` + +Create a scoped api key for only only domain or use global api key. + + +Create secret object: + +```bash +kubectl apply -f ingress-traefik/cloudflare-credentials.yml +``` + +### Configure persistent storage + +Define persistent storage: + +```yaml +persistence: + enabled: true + name: ssl-certs + accessMode: ReadWriteOnce + size: 1Gi + storageClass: nfs-csi + path: /ssl-certs +``` + +Define security context for all pods to not run as root, but as user & group `65532`. This solves NFS permissions in that we can ensure traefik has permissions to `acme-cloudflare.json` by setting file permissions to user & group: `65532`. + +```yaml +securityContext: + capabilities: + drop: [ALL] + readOnlyRootFilesystem: false + runAsGroup: 65532 + runAsNonRoot: true + runAsUser: 65532 + +podSecurityContext: + fsGroup: 65532 +``` + +## Deployment + +### 1. Add the Helm Repository and Update + +```bash +helm repo add traefik https://traefik.github.io/charts +helm repo update +``` + +### 2. Install Traefik with Helm + +```bash +helm install traefik traefik/traefik -n traefik --values=ingress-traefik/values.yml +``` + diff --git a/assets/cube.gif b/assets/cube.gif new file mode 100644 index 0000000..dc2be40 Binary files /dev/null and b/assets/cube.gif differ