

How to rotate all secrets / credentials

This is a disruptive procedure.

Delete all secrets & keypairs that kops is holding:

kops get secrets  | grep ^Secret | awk '{print $2}' | xargs -I {} kops delete secret secret {}

kops get secrets  | grep ^Keypair | awk '{print $2}' | xargs -I {} kops delete secret keypair {}

Now run kops update cluster and kops update cluster --yes to regenerate the secrets & keypairs.

We need to reboot every node (using a rolling-update). We have to use --cloudonly because our keypair no longer matches. We set the interval small because nodes will stop trusting each other during the process, so there is no point in going slowly.

kops rolling-update cluster --cloudonly --master-interval=10s --node-interval=10s --force --yes

Re-export kubecfg with new settings:

kops export kubecfg

Now the service account tokens will need to be regenerated inside the cluster:

kops toolbox dump and find a master IP

Then ssh admin@${IP} and run this to delete all the service account tokens:

# Delete all service account tokens in all namespaces
NS=`kubectl get namespaces -o 'jsonpath={.items[*].metadata.name}'`
for i in ${NS}; do kubectl get secrets --namespace=${i} --no-headers | grep "kubernetes.io/service-account-token" | awk '{print $1}' | xargs -I {} kubectl delete secret --namespace=$i {}; done

# Allow for new secrets to be created
sleep 60

# Bounce pods that we know use service account tokens - you will likely have to bounce more
kubectl delete pods -lk8s-app=dns-controller --namespace=kube-system
kubectl delete pods -lk8s-app=kube-dns --namespace=kube-system
kubectl delete pods -lk8s-app=kube-dns-autoscaler --namespace=kube-system
pkill -f kube-controller-manager