Kubernetes Pitfalls

Ingress does not require an LB

For a while I assumed that ingress required some Load Balancing service to assign it an external IP. This is how it's done in most managed cloud k8s services.

But without a Load Balancer an ingress is still valuable because it's literally the only ingress to your service mesh. So you don't have to expose 50 different NodePort services to keep track of.

So even in my humble homelab I can make good use of an ingress controller.

Troubleshooting cert-manager

Not enough IPs

Following the official docs to setup kubeadm you end up using the option --cluster-cidr=10.0.12.0/24 which is enough for getting started but eventually you will run out of IPs to allocate and you might see an event say Node stage-app04 status is now: CIDRNotAvailable.

First of all you need to resolve this in your kubeadm command if you plan on setting up any new control nodes, but the quick workaround is to edit /etc/kubernetes/manifests/kube-controller-manager.yaml and change the cluster-cidr line like this;

spec:
  containers:
  - command:
    - kube-controller-manager
    ...
    - --cluster-cidr=10.0.12.0/22
    ...

This will use four subnets, 10.0.12.0/24, 10.0.13.0/24, 10.0.14.0/24 and 10.0.15.0/24. Then just restart kubelet and wait a while.

sudo systemctl restart kubelet

You can also verify the podCIDR used with this command;

kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

Fix permanently

I use Calico CNI so this only works for Calico users, other CNIs have their own setup.

Install calicoctl

curl -sLo ~/.local/bin/calicoctl https://github.com/projectcalico/calicoctl/releases/download/v3.18.1/calicoctl-linux-amd64

Create new ippool with larger CIDR

I believe it's a good idea to have the new CIDR cover the old subnet to avoid downtime.

Kustomize edit set image

The kustomize edit set image default="my-image:latest" command only works in an overlay if you make sure to use the image name. Even though the image name is used in base, it must be used in the overlay for the images list in the overlay kustomization.yaml file to be used.