Context

When Jenkins Master restarts in Kubernetes due to an error or manual triggered restart, the currently running build slave pods are not cleaned up and they don't reconnect to the master.

Problem

Running build slaves during master restart are being kept in the Kubernetes cluster. They are consuming resources and are blocking the cluster scheduler to free / shutdown the nodes they are running on.

AT

• Create a Kubernetes cronjob which cleans up orphaned 'jenkins-slave' pods running for >3 hours.
• Send a notification to the SRE team if something has been cleaned.