在国产OpenEuler 24.03上，手把手教你搭建Hadoop 3.3.4三节点集群（含一键管理脚本）
在国产OpenEuler 24.03上构建高可用Hadoop 3.3.4集群：自动化部署与智能运维实战
当企业级大数据平台遇上国产操作系统，会碰撞出怎样的火花？OpenEuler作为国产Linux发行版的领军者，其24.03 LTS版本在稳定性与安全性上的突破，使其成为构建自主可控大数据基础设施的理想选择。本文将带您深入探索如何在OpenEuler 24.03上部署生产级Hadoop 3.3.4集群，并通过创新的自动化运维方案，实现从"手工操作"到"智能管理"的质变飞跃。
1. 环境规划与系统调优
1.1 硬件资源配置策略
构建高可用Hadoop集群的首要步骤是科学规划硬件资源。我们建议采用以下配置方案：
| 节点类型 | CPU核心数 | 内存容量 | 磁盘配置 | 网络带宽 |
|---|---|---|---|---|
| Master节点 | 8核 | 32GB | 500GB SSD系统盘 | 10Gbps |
| 2TB HDFS数据盘×2 | ||||
| Worker节点 | 16核 | 64GB | 500GB SSD系统盘 | 10Gbps |
| 4TB HDFS数据盘×4 |
提示：在OpenEuler环境下，建议使用XFS文件系统格式化的数据盘，其在大文件处理性能上优于ext4
1.2 OpenEuler系统深度优化
OpenEuler 24.03作为专为企业级场景设计的操作系统，需要进行针对性调优：
# 禁用不必要的服务（以node1为例） sudo systemctl disable firewalld sudo systemctl stop firewalld sudo systemctl disable NetworkManager-wait-online.service # 优化内核参数 echo "vm.swappiness = 10" | sudo tee -a /etc/sysctl.conf echo "net.ipv6.conf.all.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf echo "net.core.somaxconn = 32768" | sudo tee -a /etc/sysctl.conf sudo sysctl -p # 调整文件描述符限制 echo "hadoop - nofile 65536" | sudo tee -a /etc/security/limits.conf echo "hadoop - nproc 32768" | sudo tee -a /etc/security/limits.conf1.3 集群拓扑设计与服务分布
我们采用三节点黄金架构，实现服务高可用与负载均衡：
节点规划： - node1 (10.90.100.101): • NameNode • JournalNode • ZKFC • DataNode • NodeManager - node2 (10.90.100.102): • ResourceManager • NameNode (Standby) • JournalNode • ZKFC • DataNode • NodeManager - node3 (10.90.100.103): • JournalNode • DataNode • NodeManager • HistoryServer这种设计确保了关键服务(NameNode, ResourceManager)的HA特性，同时通过JournalNode实现元数据同步，避免单点故障。
2. 基础环境自动化配置
2.1 智能主机初始化脚本
传统的手工配置方式效率低下且容易出错，我们开发了全自动初始化脚本init_env.sh：
#!/bin/bash # 定义集群节点IP和主机名 declare -A NODES=( ["node1"]="10.90.100.101" ["node2"]="10.90.100.102" ["node3"]="10.90.100.103" ) # 基础环境配置函数 function init_node { # 设置主机名 hostnamectl set-hostname $1 echo "$2 $1" >> /etc/hosts # 配置静态IP cat > /etc/sysconfig/network-scripts/ifcfg-ens33 <<EOF TYPE=Ethernet BOOTPROTO=static NAME=ens33 DEVICE=ens33 ONBOOT=yes IPADDR=$2 NETMASK=255.255.255.0 GATEWAY=10.90.100.1 DNS1=114.114.114.114 EOF # 创建Hadoop专用用户 useradd -m -s /bin/bash hadoop echo "hadoop ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers # 安装基础工具 dnf install -y rsync net-tools lrzsz telnet vim } # 主执行逻辑 current_node=$(hostname -s) init_node $current_node ${NODES[$current_node]} systemctl restart network2.2 免密登录的进阶实现
传统的ssh-copy-id方式在集群规模大时效率低下，我们采用更高效的密钥分发方案：
# 在node1上生成密钥对 su - hadoop ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa # 编写分布式密钥分发脚本 cat > ~/distribute_key.sh <<'EOF' #!/bin/bash for node in node1 node2 node3; do ssh $node "mkdir -p ~/.ssh && chmod 700 ~/.ssh" scp ~/.ssh/id_rsa.pub hadoop@$node:~/.ssh/authorized_keys ssh $node "chmod 600 ~/.ssh/authorized_keys" done EOF # 执行分发 chmod +x ~/distribute_key.sh ./distribute_key.sh2.3 增强版文件分发系统
基于rsync的智能分发脚本xsync增加了断点续传和增量同步功能：
#!/bin/bash # 参数校验 if [ $# -lt 1 ]; then echo "Usage: $0 <file_or_dir> [nodes]" exit 1 fi # 默认集群节点 NODES=${2:-"node1 node2 node3"} # 获取绝对路径 FILE=$(readlink -f "$1") DIR=$(dirname "$FILE") BASE=$(basename "$FILE") # 多节点并行分发 for host in $NODES; do echo "===== Syncing to $host =====" rsync -avz --partial --progress --delete \ -e "ssh -o StrictHostKeyChecking=no" \ "$FILE" hadoop@$host:"$DIR/" & done wait echo "All files synchronized successfully"3. Hadoop集群高级部署
3.1 JDK与Hadoop自动化安装
我们采用源码编译方式获取针对OpenEuler优化的JDK和Hadoop：
# JDK11自动安装脚本 JDK_URL="https://repo.huaweicloud.com/openjdk/11.0.2/openjdk-11.0.2_linux-x64_bin.tar.gz" wget -O /tmp/jdk11.tar.gz $JDK_URL tar -xzf /tmp/jdk11.tar.gz -C /opt ln -s /opt/jdk-11.0.2 /opt/jdk # 配置环境变量 cat >> /etc/profile <<EOF export JAVA_HOME=/opt/jdk export PATH=\$JAVA_HOME/bin:\$PATH EOF source /etc/profile # Hadoop编译安装（针对OpenEuler优化） HADOOP_SRC_URL="https://archive.apache.org/dist/hadoop/core/hadoop-3.3.4/hadoop-3.3.4-src.tar.gz" wget -O /tmp/hadoop-src.tar.gz $HADOOP_SRC_URL tar -xzf /tmp/hadoop-src.tar.gz -C /opt cd /opt/hadoop-3.3.4-src # 安装编译依赖 dnf install -y gcc-c++ cmake autoconf automake libtool zlib-devel openssl-devel # 开始编译 mvn package -Pdist,native -DskipTests -Dtar3.2 高可用配置模板
Hadoop高可用配置是生产环境的核心，我们提供经过验证的配置模板：
core-site.xml关键配置：
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://mycluster</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>node1:2181,node2:2181,node3:2181</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/data/hadoop/tmp</value> </property> </configuration>hdfs-site.xml高可用配置：
<property> <name>dfs.nameservices</name> <value>mycluster</value> </property> <property> <name>dfs.ha.namenodes.mycluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn1</name> <value>node1:8020</value> </property> <property> <name>dfs.namenode.http-address.mycluster.nn1</name> <value>node1:9870</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://node1:8485;node2:8485;node3:8485/mycluster</value> </property>3.3 智能资源管理配置
YARN资源管理需要根据实际硬件进行精细化调整：
<!-- yarn-site.xml优化配置 --> <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>57344</value> <!-- 56GB of 64GB --> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>57344</value> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>14</value> <!-- 16 cores - 2 for system --> </property> <property> <name>yarn.scheduler.maximum-allocation-vcores</name> <value>14</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property>4. 智能运维与监控体系
4.1 全能集群管理脚本
我们开发的hdp_manager.sh脚本集成了集群管理、日志收集、性能监控等多项功能：
#!/bin/bash # 定义颜色代码 RED='\033[0;31m' GREEN='\033[0;32m' NC='\033[0m' # No Color # 集群节点定义 NODES=("node1" "node2" "node3") MASTER_NODE="node1" RESOURCE_NODE="node2" case $1 in "start") echo -e "${GREEN}Starting Hadoop Cluster...${NC}" # 启动Zookeeper集群 for node in "${NODES[@]}"; do ssh $node "/opt/zookeeper/bin/zkServer.sh start" done # 启动JournalNode for node in "${NODES[@]}"; do ssh $node "/opt/hadoop/bin/hdfs --daemon start journalnode" done # 格式化ZKFC并启动NameNode ssh $MASTER_NODE "/opt/hadoop/bin/hdfs zkfc -formatZK" ssh $MASTER_NODE "/opt/hadoop/sbin/start-dfs.sh" ssh $RESOURCE_NODE "/opt/hadoop/sbin/start-yarn.sh" ;; "stop") echo -e "${RED}Stopping Hadoop Cluster...${NC}" ssh $RESOURCE_NODE "/opt/hadoop/sbin/stop-yarn.sh" ssh $MASTER_NODE "/opt/hadoop/sbin/stop-dfs.sh" # 停止JournalNode for node in "${NODES[@]}"; do ssh $node "/opt/hadoop/bin/hdfs --daemon stop journalnode" done # 停止Zookeeper for node in "${NODES[@]}"; do ssh $node "/opt/zookeeper/bin/zkServer.sh stop" done ;; "status") echo -e "${GREEN}Cluster Status:${NC}" # 检查Zookeeper状态 for node in "${NODES[@]}"; do echo -n "Zookeeper@$node: " ssh $node "/opt/zookeeper/bin/zkServer.sh status" | grep Mode done # 检查HDFS服务 echo -e "\nHDFS Services:" for node in "${NODES[@]}"; do echo "$node:" ssh $node "jps | grep -E 'NameNode|DataNode|JournalNode|DFSZKFailoverController'" done # 检查YARN服务 echo -e "\nYARN Services:" for node in "${NODES[@]}"; do echo "$node:" ssh $node "jps | grep -E 'ResourceManager|NodeManager'" done ;; *) echo "Usage: $0 {start|stop|status}" exit 1 ;; esac4.2 实时监控告警系统
集成Prometheus+Grafana的监控方案配置：
# 在所有节点安装Node Exporter for node in node1 node2 node3; do ssh $node "wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz" ssh $node "tar xzf node_exporter-*.tar.gz && cd node_exporter-* && nohup ./node_exporter &" done # 配置Prometheus抓取规则 cat > prometheus.yml <<EOF global: scrape_interval: 15s scrape_configs: - job_name: 'hadoop' static_configs: - targets: ['node1:9100', 'node2:9100', 'node3:9100'] - job_name: 'hdfs' static_configs: - targets: ['node1:9870', 'node2:9870'] - job_name: 'yarn' static_configs: - targets: ['node2:8088'] EOF4.3 日志智能分析方案
ELK日志收集系统的快速部署：
# 在日志分析节点(node3)上部署ELK sudo dnf install -y java-11-openjdk wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.0-linux-x86_64.tar.gz tar xzf elasticsearch-*.tar.gz cd elasticsearch-*/bin ./elasticsearch -d # 安装Logstash wget https://artifacts.elastic.co/downloads/logstash/logstash-7.17.0-linux-x86_64.tar.gz tar xzf logstash-*.tar.gz # 配置Hadoop日志收集 cat > logstash-hadoop.conf <<EOF input { file { path => "/opt/hadoop/logs/*.log" start_position => "beginning" } } filter { grok { match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{DATA:component}: %{GREEDYDATA:message}" } } } output { elasticsearch { hosts => ["localhost:9200"] index => "hadoop-logs-%{+YYYY.MM.dd}" } } EOF # 启动Logstash nohup ./logstash -f logstash-hadoop.conf &