121. 排查 etcd 时间同步问题

etcd is a critical component in Kubernetes environments, serving as the distributed reliable key-value store that holds the cluster's state. Its proper functioning is highly dependent on accurate time synchronisation across all etcd nodes. Even a small clock drift can lead to significant issues, impacting cluster stability and performance.
etcd 是 Kubernetes 环境中的关键组件，作为分布式且可靠的键值存储，保存集群的状态。 其正常工作高度依赖于所有 etcd 节点间的时间同步准确。即使是小的时钟漂移也可能引发重大问题，影响集群的稳定性和性能。

Time synchronisation issues in etcd can manifest in various ways, often leading to a cascade of errors throughout your cluster. Common symptoms include:
etcd 中的时间同步问题可能以多种方式表现出来，常常导致整个集群中出现连锁错误。常见症状包括：

• High Clock Drift errors in etcd logs:You'll see warnings like"prober found high clock drift".
  etcd 日志中的高时钟漂移错误：你会看到类似"prober found high clock drift"的警告。
• Slow etcd requests:Messages such as"apply request took too long"or"waiting for ReadIndex response took too long"indicate that etcd operations are being delayed, often due to inconsistencies between nodes. While these kinds of messages can also indicate other etcd issues, they will commonly be included when a time synchronisation error is present.
  慢速 etcd 请求：诸如“应用请求耗时过长”等消息，或"waiting for ReadIndex response took too long"表明 etcd 操作被延迟，通常是由于节点间的不一致。虽然这类消息也可能提示其他 etcd 问题，但当存在时间同步错误时，通常会包含这些消息。
• Kubernetes API Server timeouts:Thekube-apiservermight struggle to connect to etcd, resulting inhttp: Handler timeouterrors.
  Kubernetes API 服务器超时：kube-apiserver可能难以连接 etcd，导致http： Handler 超时错误。
• Unhealthy etcd nodes:You'll see your etcd nodes as unhealthy or flapping (losing and regaining connection).
  不健康的 etcd 节点：你会看到你的 etcd 节点不健康或颤动（连接断开和恢复）。
• Raft misalignment:Inconsistent Raft indexes among etcd members.
  筏子错位：ETCHD 成员间的筏子指数不一致。

In order to solve a time synchronisation issue, you can follow these steps:
为了解决时间同步问题，你可以按照以下步骤操作：

Check the current time and synchronisation status on all your etcd nodes:
检查所有 etcd 节点当前的时间和同步状态：

• Check current time: 查看当前时间：

  date +"%T.%N"

  Execute this command simultaneously on all etcd nodes to quickly spot any significant differences. Even differences of a few seconds can be critical for etcd.
  同时对所有 etcd 节点执行此命令，以快速发现任何显著差异。即使是几秒钟的差异，对 ETC 来说也可能至关重要。

• Check the status of your NTP client.E.g., forchrony:
  检查你的 NTP 客户状态。例如，对于时间性：

  timedatectl chronyc sources list chronyc tracking

Based on your verification, identify potential clock misalignments, and fix the issue:
根据你的验证，识别潜在的时钟错位，并解决问题：

• Review your NTP configuration:Examine/etc/chrony.conf(for chrony) or/etc/ntp.conf(for ntpd) on each etcd node. Ensure that the configured NTP servers are correct and reachable. It's recommended to use reliable and accessible NTP sources.
  检查你的 NTP 配置：在每个 etcd 节点上检查/etc/chrony.conf（chrony）或/etc/ntp.conf（用于 ntpd）。确保配置好的 NTP 服务器正确且可访问。建议使用可靠且可访问的 NTP 资源。
• Check Firewall Rules:Verify that UDP port 123 is open in your firewall configuration on all etcd nodes to allow NTP traffic to and from your configured time servers.
  检查防火墙规则：确认所有 ETC 节点的防火墙配置中 UDP 123 端口是否开放，以便允许 NTP 流量往返你配置的时服务器。

Once you've corrected the configuration, force the NTP client to resynchronize the time.
一旦你纠正了配置，强制 NTP 客户端重新同步时间。

sudo systemctl restart chronyd

Important:When restarting the time synchronisation service, it's generally safer to do itone etcd node at a time. While a temporary clock adjustment might occur, restarting the service on all nodes simultaneously could introduce further instability if a significant time jump occurs across the entire etcd cluster at once.
重要提示：重启时间同步服务时，通常一次处理一个 etcd 节点更安全。虽然可能会进行临时时钟调整，但如果整个 ETCHD 集群同时发生显著时间跳跃，所有节点同时重启服务可能会带来进一步的不稳定性。

After ensuring that time synchronisation is correct on all etcd nodes, verify the health of your etcd cluster and monitor the etcd logs for any recurring "high clock drift" or "took too long" errors. These should disappear once the time synchronisation is stable.
在确保所有 etcd 节点的时间同步正确后，检查你的 etcd 集群健康状况，并监控 etcd 日志中是否有反复出现的“高时钟漂移”或“耗时过长”错误。一旦时间同步稳定，这些问题应该会消失。

The primary cause of time synchronisation issues on etcd nodes is often a misconfigured or non-functional NTP (Network Time Protocol) client, such aschronyorntpd. Specific problems can include:
etcd 节点时间同步问题的主要原因通常是配置错误或无法正常工作的 NTP（网络时间协议）客户端，如chrony或ntpd。具体问题可能包括：

• Incorrect NTP Server Configuration:Thechrony.conforntp.conffile might point to incorrect or unreachable NTP servers.
  NTP 服务器配置错误：chrony.conf或ntp.conf文件可能指向错误或无法访问的 NTP 服务器。
• Firewall Rules:Necessary firewall rules (e.g., UDP port 123 for NTP) might be missing, preventing the nodes from reaching the configured time servers.
  防火墙规则：必要的防火墙规则（例如 NTP 的 UDP 端口 123）可能缺失，导致节点无法到达配置的时服务器。
• Network Connectivity Issues:General network problems can also prevent NTP synchronisation.
  网络连接问题：一般网络问题也可能阻碍 NTP 同步。

An RKE or RKE2 cluster with multiple etcd nodes.
一个带有多个 etcd 节点的 RKE 或 RKE2 集群。