Minio分布式集群+nginx+keepalived部署

0.准备

minio官方文档：https://www.minio.org.cn/docs/minio/linux/

6台服务器：四台用于minio分布式集群部署，2台用于nginx+keepalived高可用，keep alived VIP =192.168.10.17

一、minio集群配置、四个节点步骤一致

1.minio下载

https://dl.minio.org.cn/server/minio/release/linux-amd64/minio

2.目录规划

（1）创建不存在的目录，【/data】在服务器中需要单独挂载一块盘

mkdir -p /usr/local/minio/config mkdir -p /data/minio/data mkdir -p /data/minio/logs

（2）上传minio二进制文件

cd /usr/local/minio/ chmod 755 minio

3.集群启动脚本

需要保证脚本中服务器http链接的顺序·一致

minio-01

vi /usr/local/minio/minio-start.sh #!/bin/bash export MINIO_ACCESS_KEY=admin export MINIO_SECRET_KEY=password /usr/local/minio/minio server --config-dir /usr/local/minio/config \ --address "192.168.10.11:9001" --console-address 192.168.10.11:9000 \ http://192.168.10.11/data/minio/data \ http://192.168.10.12/data/minio/data \ http://192.168.10.13/data/minio/data \ http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

minio-02

vi /usr/local/minio/minio-start.sh #!/bin/bash export MINIO_ACCESS_KEY=admin export MINIO_SECRET_KEY=password /usr/local/minio/minio server --config-dir /usr/local/minio/config \ --address "192.168.10.12:9001" --console-address 192.168.10.12:9000 \ http://192.168.10.11/data/minio/data \ http://192.168.10.12/data/minio/data \ http://192.168.10.13/data/minio/data \ http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

minio-03

vi /usr/local/minio/minio-start.sh #!/bin/bash export MINIO_ACCESS_KEY=admin export MINIO_SECRET_KEY=password /usr/local/minio/minio server --config-dir /usr/local/minio/config \ --address "192.168.10.13:9001" --console-address 192.168.10.13:9000 \ http://192.168.10.11/data/minio/data \ http://192.168.10.12/data/minio/data \ http://192.168.10.13/data/minio/data \ http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

minio-04

vi /usr/local/minio/minio-start.sh #!/bin/bash export MINIO_ACCESS_KEY=admin export MINIO_SECRET_KEY=password /usr/local/minio/minio server --config-dir /usr/local/minio/config \ --address "192.168.10.14:9001" --console-address 192.168.10.14:9000 \ http://192.168.10.11/data/minio/data \ http://192.168.10.12/data/minio/data \ http://192.168.10.13/data/minio/data \ http://192.168.10.14/data/minio/data > /data/minio/logs/minio.logs

4.配置开机启动服务

（1）修改启动文件

vi /etc/systemd/system/minio.service [Unit] Description=Minio service Documentation=https://docs.minio.io/ [Service] WorkingDirectory=/usr/local/minio/ ExecStart=/usr/local/minio/minio-run.sh Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target

（2）脚本增加可执行权限

systemctl daemon-reload systemctl enable minio systemctl start minio systemctl status minio

5.访问验证

http://192.168.10.11:9000

二、nginx配置，双节点步骤一致

1.nginx下载链接

https://nginx.org/en/download.html

2.安装依赖包

yum install -y gcc pcre pcre-devel zlib zlib-devel openssl openssl-devel

3.创建用户组

useradd -M -s /sbin/nologin nginx

4.解压、编译安装

tar -xzvf nginx-1.23.4.tar.gz cd nginx-1.23.4 ./configure --prefix=/usr/local/nginx --user=nginx --group=nginx --with-http_stub_status_module --with-http_ssl_module --with-http_mp4_module --with-http_realip_module --with-pcre --with-http_gunzip_module --with-http_gzip_static_module --with-stream

5.增加软连接、开机启动文件

ln -s sbin/nginx /usr/local/sbin/ vi /lib/systemd/system/nginx.service [Unit] Description=nginx service After=network.target [Service] Type=forking ExecStart=/usr/local/nginx/sbin/nginx ExecReload=/usr/local/nginx/sbin/nginx -s reload ExecStop=/usr/local/nginx/sbin/nginx -s quit PrivateTmp=true [Install] WantedBy=multi-user.target chmod 754 /lib/systemd/system/nginx.service

6.配置conf文件

cd /usr/local/nginx/conf vi nginx.conf worker_processes 4; events { worker_connections 1024; } http { tcp_nodelay on; types_hash_max_size 2048; sendfile on; tcp_nopush on; server_tokens off; keepalive_timeout 60; keepalive_requests 2048; gzip on; gzip_static on; gzip_min_length 1000; gzip_buffers 4 8k; gzip_comp_level 7; gzip_http_version 1.1; gzip_vary on; gzip_types text/plain application/x-javascript text/css application/xml image/jpeg image/gif image/png; proxy_max_temp_file_size 0; proxy_buffer_size 64k; proxy_buffers 4 64k; proxy_connect_timeout 500ms; proxy_headers_hash_max_size 1024; proxy_headers_hash_bucket_size 128; client_header_buffer_size 16k; large_client_header_buffers 4 32k; client_max_body_size 80m; client_body_buffer_size 60m; server_names_hash_max_size 1024; server_names_hash_bucket_size 1024; underscores_in_headers on; upstream minio_server { least_conn; server 192.168.10.11:9000; server 192.168.10.12:9000; server 192.168.10.13:9000; server 192.168.10.14:9000; } upstream minio_console { least_conn; server 192.168.10.11:9001; server 192.168.10.12:9001; server 192.168.10.13:9001; server 192.168.10.14:9001; } server { listen 9000; listen [::]:9000; server_name minio_server; # Allow special characters in headers ignore_invalid_headers off; # Allow any size file to be uploaded. # Set to a value such as 1000m; to restrict file size to a specific value client_max_body_size 0; # Disable buffering proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; # Default is HTTP/1, keepalive is only enabled in HTTP/1.1 proxy_http_version 1.1; proxy_set_header Connection ""; chunked_transfer_encoding off; proxy_pass http://minio_s3; # This uses the upstream directive definition to load balance } } server { listen 9001; listen [::]:9001; #server_name console.example.net; server_name minio_console; # Allow special characters in headers ignore_invalid_headers off; # Allow any size file to be uploaded. # Set to a value such as 1000m; to restrict file size to a specific value client_max_body_size 0; # Disable buffering proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-NginX-Proxy true; # This is necessary to pass the correct IP to be hashed real_ip_header X-Real-IP; proxy_connect_timeout 300; # To support websocket proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; chunked_transfer_encoding off; proxy_pass http://minio_console/; # This uses the upstream directive definition to load balance } } }

7.启动nginx

systemctl enable nginx systemctl start nginx systemctl status nginx

三、keepalived配置，双节点步骤一致

1.下载链接

https://www.keepalived.org/download.html

2.安装依赖包

yum -y install libnl libnl-devel libnfnetlink-devel openssl-devel gcc

3.解压&编译安装

tar -zxvf keepalived-2.3.1.tar.gz cd keepalived-2.3.1 ./configure --prefix=/usr/local/keepalived make && make install

4.配置开机启动

vi /usr/lib/systemd/system/keepalived.service [Unit] Description=LVS and VRRP High Availability Monitor After=network-online.target syslog.target Wants=network-online.target Documentation=man:keepalived(8) Documentation=man:keepalived.conf(5) Documentation=man:genhash(1) Documentation=https://keepalived.org [Service] Type=forking PIDFile=/run/keepalived.pid KillMode=process EnvironmentFile=-/usr/local/keepalived/etc/sysconfig/keepalived ExecStart=/usr/local/keepalived/sbin/keepalived $KEEPALIVED_OPTIONS ExecReload=/bin/kill -HUP $MAINPID [Install] WantedBy=multi-user.target

5.配置keepalived高可用、检查脚本

vi /usr/local/keepalived/etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_NGINX } vrrp_script chk_nginx { script "/usr/local/keepalived/etc/keepalived/check_nginx.sh" interval 2 weight -30 rise 2 fall 2 } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 150 priority 100 # 备节点权重小 ## 192.168.10.15配置100，192.168.10.16配置80 advert_int 3 #nopreempt authentication { auth_type PASS auth_pass 1111 } track_script { chk_nginx } virtual_ipaddress { 192.168.10.17 # vip } }

vi /usr/local/keepalived/etc/keepalived/check_nginx.sh #!/bin/bash nginx_status=$(systemctl is-active nginx) if [ "$nginx_status" = "active" ]; then exit 0 else pkill keepalived fi

6.访问验证

配置完成后使用VIP可以访问MINIO的控制台界面，实现自动负载均衡

http://192.168.10.17:9000