CANN/hcomm集群心跳监控模块

ClusterMonitor Cluster Heartbeat Detection Function Description

【免费下载链接】hcommHCOMM（Huawei Communication）是HCCL的通信基础库，提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcomm

Function Description

ClusterMonitoris the cluster-level heartbeat monitoring module of HCCL (Huawei Collective Communication Library) under thecoll_communicator_mgr/dfxpath, mainly used forcontinuous detection of reachability and liveness of all participating nodes and abnormal propagation after the establishment of a collective communication domain (Communicator).

Core capabilities include:

1. Node UID identification: Each Rank is encoded as a uniqueClusterUIDType(up to 2048 bytes) vianetInstId + localId, used for cross-layer (device / server / pod / superPod) node identification.
2. Dual Ring link establishment: Within eachnetLayerplane, nodes are sorted bylocalIdornetInstIdto form a Ring topology; each Rank establishes heartbeat socket connections with the "left" and "right" neighbors on the ring (only one neighbor when there are only 2 nodes).
3. Async link establishment: Each remote UID starts an independent threadCreateLinkWithRemotePonitforSocketCreate+ status polling, controlled byHCCL_CONNECT_TIMEOUT.
4. Periodic heartbeat send/receive: The backgroundMonitorThreadtraverses all sockets atBROADCAST_INTERVALintervals, firstSocketSendNbto send heartbeats thenSocketRecvNbto receive heartbeats;lostNumaccumulates everyHEARTBEAT_COUNTcycles, and whenHCCL_LOST_THRESHOLD(30s) is reached, the status is determined asLOST.
5. Abnormal status propagation: NodeLOSTorCQE_ERRstatus is broadcast to other neighbors via the Ring link (SetStatus→errRankQueue_→ProcessExceptionEvent→SendFrame).
6. Error information reporting: Through theGetCqeErrInfoFromTaskExceptioncallback registered via__attribute__((constructor)), CQE errors from AICPU/CCU tasks are collected intocqeErrInfo_, and triggered viaSetStatus(..., CLUSTER_MONITOR_CQE_ERR, true)to broadcast; the query interfaceGetErrStatusVecFromCluserMonitorformats error descriptions by priority (CQE_ERR > LOST) and returns them to the upper layer.

The module belongs to the DFX (Design For X) category and is a key component of HCCL for providing "network disconnection / peer CoreDump" observability at the cluster level.

Directory Description

cluster_monitor/ ├── CMakeLists.txt # Build script, only adds cluster_monitor.cc to the hcomm target ├── cluster_monitor.h # Class and data structure definitions (ClusterMonitor, Frame, SockCtx, UID, etc.) └── cluster_monitor.cc # Module implementation, including heartbeat main thread, async link establishment, frame send/receive, exception handling

Directory features:

• Located incoll_communicator_mgr/dfx/, belongs to DFX monitoring functionality;
• Module autonomous: singleton (GetInstance(u32 deviceId)), does not depend on other files in the directory;
• External dependencies:ring_buffer,reference_map,hcclCommSocket,hccl_communicator,hcclCommDfx,coll_comm,log,comm_addr_logger, etc.

Flow Description (Mermaid Sequence Diagram)

Overall Registration / Link Establishment / Heartbeat / Deregistration Main Flow

CQE Exception Reporting and Broadcast Sequence

Data Description (Mermaid Class Diagram)

Interface Description

Public API (External)

Internal / Private Interfaces

Cross-module Callback Registration (__attribute__((constructor)))

ClusterMonitorCallBackInit() { RegisterGetAicpuCqeErrInfoCallBackHcomm(GetCqeErrInfoFromTaskException); RegisterGetCcuCqeErrInfoCallBackHcomm(GetCqeErrInfoFromTaskException); RegisterAicpuGetErrStatusVecCallBack(GetErrStatusVecFromCluserMonitor); RegisterCcuGetErrStatusVecCallBack(GetErrStatusVecFromCluserMonitor); }

The module registers two callbacks with the AICPU/CCU framework at load time:Exception Entry(error reporting) andError Query(error formatting export), which are the only coupling points between the module and the upper-layer Task exception system.

Usage Limitations (Supported Scenarios and Constraint Specifications)

Supported Scenarios

1. Multi-Rank communicator: rankSize >= 2 communicators; when rankSize == 1,RegisterToClusterMonitordirectly returnsHCCL_SUCCESSwith aWARNINGlog and no link establishment.
2. Multi-plane topology: Supports dual Ring link establishment fornetLayer = 0(within same server/device, sorted bylocalId) andnetLayer >= 1(cross-server/pod/superPod, sorted bynetInstId).
3. Cross-communicator shared socket: Viahccl::ReferenceMapcounting, only one socket is created when multiple communicators connect to the same remote node; reference count--on deregistration, truly destroyed only when it reaches zero.
4. Ring fault propagation: Single nodeLOST/CQE_ERRstatus can spread to other nodes along the Ring link, enabling querying cluster-wide exceptions from any node.
5. CQE error capture: ReceivesTaskexceptions via AICPU/CCU callbacks, formatted as readable logs with local / remoteinstanceId / localId / Eid.
6. Toggle switch: The environment variableHCCL_DFS_CONFIG.cluster_heartbeatcan disable heartbeat registration / CQE error capture chain.
7. HCCL v2 communicator: Depends onHcclCommunicator::GetRankGraphV2andRankGraph, only effective whenCommunicatorV2exists; otherwise fails withCHK_PTR_NULLinGetRemEndpointDescs.

Constraint Specifications

【免费下载链接】hcommHCOMM（Huawei Communication）是HCCL的通信基础库，提供通信域以及通信资源的管理能力。项目地址: https://gitcode.com/cann/hcomm