Kazan - Kubernetes cluster

On prem self-hosted kubernetes cluster running on microk8s.

Table of Contents

• Adding nodes
• Plugins
  • MetalLB
  • Helm
• Persistent Storage
• Ingress
  • Configuration
  • Deployment

Adding nodes

Provision and install microk8s

If this is first or additional node to be used for kubernetes we want to run base server configuration using playbook, replacing KUBERNETES_HOST:

ansible-playbook plays/base_server_setup.yml -i schleppe.ini -l KUBERNETES_HOST

Install microk8s using snap:

sudo snap install microk8s --classic --channel=1.27/stable

Joining cluster

From a existing node run:

microk8s.add-node

Copy the output and join using command similar to the following on the new kubernetes node:

microk8s join kazan.schleppe:25000/60e891acfc38556ea569cd15d5e025a1/91a7ab376757

Plugins

MetalLB

MetallLB hooks into Kubernetes cluster and provides a network load-balancer implementation.

microk8s.enable metallb:10.0.0.150-10.0.0.154

Helm

Use helm charts to easier install managed systems and applications.

microk8s.enable helm

Persistent storage

To create a persistent volume claim (PVC) we need to first define a storage class. To not solve the issue of HA stoarge, a single NFS host is used. Setup expects a linux server service export path: /srv/nfs.

Setup/Deployment

Define storage class pointing to NFS server:

kubectl apply -f storage-nfs/storageClass.yml

Note

PVC resources will reference spec.storageClassName: nfs-csi

Ingress

Traefik is configured as default ingress, enabling dynamic configuration & SSL generation.

Configuration

Important

Make sure that persistent storage is configured. Traefik is configured to expect nfs-csi storageClass already exists, follow steps above.

Disable nginx ingress

Microk8s uses nginx as default ingress controller. We wish to replace this with traefik. Do disable plugin run:

microk8s.disable ingress

Configure cloudflare certificate resolver

Add to additionalArguments:

  - --certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare
  - --certificatesresolvers.cloudflare.acme.email=YOUR_EMAIL_HERE
  - --certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1
  - --certificatesresolvers.cloudflare.acme.storage=/ssl-certs/acme-cloudflare.json

It is important that persistent storage is configured and a acme-cloudflare.json file is manually touched and it's permissions updated. From the nfs server run:

cd /srv/nfs/PVC_VOLUME_NAME_HERE
sudo touch acme-cloudflare.json
sudo chown 65532:65532 acme-cloudflare.json

Configure cloudflare credentials secret

Add to env:

  - name: CF_API_EMAIL
    valueFrom:
      secretKeyRef:
        key: email
        name: cloudflare-credentials
  - name: CF_API_KEY
    valueFrom:
      secretKeyRef:
        key: apiKey
        name: cloudflare-credentials

Update following values from example secret file (ingress-traefik/cloudflare-credentials.yml):

  email: YOUR_CLOUDFLARE_EMAIL
  apiKey: YOUR_CLOUDFLARE_API_KEY

Create a scoped api key for only only domain or use global api key.

Create secret object:

kubectl apply -f ingress-traefik/cloudflare-credentials.yml

Configure persistent storage

Define persistent storage:

persistence:
  enabled: true
  name: ssl-certs
  accessMode: ReadWriteOnce
  size: 1Gi
  storageClass: nfs-csi
  path: /ssl-certs

Define security context for all pods to not run as root, but as user & group 65532. This solves NFS permissions in that we can ensure traefik has permissions to acme-cloudflare.json by setting file permissions to user & group: 65532.

securityContext:
  capabilities:
    drop: [ALL]
  readOnlyRootFilesystem: false
  runAsGroup: 65532
  runAsNonRoot: true
  runAsUser: 65532

podSecurityContext:
  fsGroup: 65532

Deployment

1. Add the Helm Repository and Update

helm repo add traefik https://traefik.github.io/charts
helm repo update

2. Install Traefik with Helm

helm install traefik traefik/traefik -n traefik --values=ingress-traefik/values.yml