Kubernetes Metrics Server TLS Scrape Issue

Issue Description

When deploying the Kubernetes Metrics Server, you may encounter the following error in the pod logs:

E0307 ... scraper.go:149] "Failed to scrape node" err="Get \"https://172.18.0.3:10250/metrics/resource\"..."

This occurs because the kubelet process on the Kubernetes nodes uses a self-signed TLS certificate by default for its HTTPS endpoint (port 10250). When the metrics-server attempts to scrape metrics, it cannot verify the self-signed certificate against the trusted cluster Certificate Authority (CA), resulting in a connection rejection.

Resolutions

1. The Quick/Insecure Fix (Development Only)

The fastest way to resolve this issue is to configure the Metrics Server to skip TLS certificate verification. This is common in local environments (Minikube, kind, k3s) but is vulnerable to Man-in-the-Middle (MITM) attacks and should not be used in production. How to apply: Add the --kubelet-insecure-tls flag to the metrics-server deployment container arguments. Via kubectl edit:

kubectl edit deployment metrics-server -n kube-system

Add the flag to the args section:

      containers:
      - args:
        - --kubelet-insecure-tls # Add this line
        # ... other args

Via Helm:

helm upgrade metrics-server metrics-server/metrics-server -n kube-system --set args={--kubelet-insecure-tls}

2. The Secure Fix (Production Recommended)

The most secure approach is to configure the kubelet on all nodes to request a serving certificate signed by the central Kubernetes cluster CA using TLS bootstrapping. Since the Metrics Server inherently trusts the cluster CA, it will successfully verify the kubelet’s new certificate.

Step-by-step implementation:
Step 1: Enable serverTLSBootstrap on the Kubelet
Modify the kubelet configuration file (e.g., /var/lib/kubelet/config.yaml or via the kube-system ConfigMap) to include:

serverTLSBootstrap: true

Then restart the kubelet service on each node:

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Step 2: Approve the Certificate Signing Requests (CSR)
Once restarted, the kubelet will generate a CSR.

View pending CSRs:
```
kubectl get csr
```
Approve the CSR manually:
```
kubectl certificate approve <csr-name>
```

(Note: For production environments, it is recommended to automate this process using an auto-approver controller like kubelet-rubber-stamp or by configuring specific RBAC policies.)

Step 3: Remove the Insecure Flag
Ensure you remove the --kubelet-insecure-tls flag from the Metrics Server deployment to enforce strict TLS verification.

# Using kubectl edit
kubectl edit deployment metrics-server -n kube-system
# Or using Helm
helm upgrade metrics-server metrics-server/metrics-server -n kube-system --set args={}