Experiencing kube-apiserver response times exceeding 3 seconds is a critical performance issue that can impact cluster stability and reliability. This is often caused by high request loads, resource contention, etcd problems, or misconfigured admission controllers.
Here is a systematic approach to diagnosing and resolving kube-apiserver latency.
- Monitor API server metrics
Use a monitoring tool like Prometheus and Grafana to examine the API server's metrics. This is the first step to narrowing down the source of the problem.
Request duration: Look at the apiserver_request_duration_seconds metric, segmented by verb (GET, LIST, POST), resource, group, and component.
High GET/LIST latency indicates potential issues with the underlying etcd storage or the volume of objects being requested.
High POST/PUT latency points to possible delays from admission webhooks or general write performance bottlenecks.
In-flight requests: Check apiserver_current_inflight_requests. A high number can indicate the API server is overloaded and struggling to keep up with the incoming request rate.
Request throttling: Look for apiserver_flowcontrol_rejected_requests_total. A high or non-zero value indicates that API Priority and Fairness (APF) is throttling requests, suggesting resource bottlenecks.
API server logs: Check the kube-apiserver pod logs for any network-related errors, connection issues, or webhook failures.
- Identify the source of high API server load
An overloaded API server is one of the most common causes of high latency.
Find noisy clients: Use kube-audit logs to identify which user agents, service accounts, or pods are making a high volume of requests. Managed Kubernetes services like AKS offer built-in diagnostics to identify noisy clients making excessive LIST calls.
Inspect API Priority and Fairness (APF): Review the APF metrics, such as apiserver_flowcontrol_current_inqueue_request, to see if a particular request queue has a backlog.
Identify inefficient requests: Check for clients making frequent, unoptimized LIST requests. Instead of polling, applications should use "watch" features, which are more efficient.
-
Troubleshoot etcd performance
The API server relies on etcd for all cluster state data. etcd latency directly impacts API server performance.
Monitor etcd metrics: Check the etcd_request_duration_seconds metric to measure the latency of read and write requests to the database.
Check database size: A large number of objects in etcd can cause performance degradation. Check the etcd_db_total_size_in_bytes or apiserver_storage_db_total_size_in_bytes metric to monitor size. The etcd database has a default size limit of 4 GB.
Defragment etcd: If the etcd database is fragmented, use etcdctl defrag to clean up storage.
Clean up old resources: Identify and remove old, unused objects, such as completed jobs, to free up etcd space. For example: -
Investigate Admission Controller overhead
Admission controllers can add latency, especially with multiple validating or mutating webhooks.
Check admission webhook latency: Monitor the apiserver_admission_webhook_admission_duration_seconds metric to identify any webhooks causing delays.
Look for deadlocks: Check logs for errors related to webhook communication failures, such as failed calling webhook or timeout errors.
Tune webhooks: Optimize or disable any slow or unnecessary webhooks. In some cases, you may be able to use built-in ValidatingAdmissionPolicy instead of external webhooks.
- Check cluster resources and network
API server resources: Ensure the kube-apiserver pod has adequate CPU and memory requests and limits configured. A lack of resources will directly impact performance.
etcd cluster resources: For self-hosted etcd, ensure the nodes have sufficient resources, including fast SSD storage.
Network latency: Poor network connectivity between the API server and its clients, or between the API server and etcd, can introduce significant latency.
Test connectivity from the kube-apiserver pod to the etcd endpoints.
Test network latency from a client machine to the kube-apiserver.
Inspect CNI plugins for network issues.
- Address inefficient API calls
Some API calls can be inherently slow, especially in large clusters.
Unoptimized LIST requests: Large clusters with thousands of objects can cause LIST operations to become very slow as the API server retrieves and filters objects in memory. Kubernetes has implemented API Streaming to improve memory usage for large lists, but some calls can still be intensive.
Large objects: A large average object size (e.g., in ConfigMaps or Secrets) can put pressure on both the API server and etcd. Consider splitting large objects or moving data into a different storage backend.
Monitoring for Kubernetes API server performance lags