当前位置：首页 > news >正文

【ELK】分布式日志平台搭建全攻略 - 详解

news 2026/5/12 23:09:42

>ELK 是一个由 Elasticsearch、Logstash 和 Kibana 组成的开源日志收集、存储、搜索和可视化分析平台。

一、环境准备

1.1 创建目录

1.2 创建配置文件

二、系统集成

2.1 FileBeat

2.2 项目集成

2.3 日志查看

一、环境准备

1.1 创建目录

elk架构图

创建目录结构

mkdir -p /opt/elk/{elasticsearch/{data,logs,plugins,config},logstash/{config,pipeline},kibana/config,filebeat/{config,data}}

设置权限

chomd -R 777 elasticsearch
chmod -R 777 logstash
chmod -R 777 kibana
chmod -R 777 filebeat

1.2 创建配置文件

Logstash配置：

vim logstash/config/logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["http://elasticsearch:9200"]
xpack.monitoring.elasticsearch.username: "elastic"
xpack.monitoring.elasticsearch.password: "mH0awV4RrkN2"
# 日志级别
log.level: info

vim logstash/pipeline/logstash.conf

input {beats {port => 5045ssl => false}tcp {port => 5044codec => json}
}
filter {# RuoYi 应用日志（优先级最高）if [app_name] {mutate {add_field => { "[@metadata][target_index]" => "ruoyi-logs-%{+YYYY.MM.dd}" }}}# 系统日志else if [fields][log_type] == "system" {grok {match => { "message" => "%{SYSLOGLINE}" }}date {match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]target => "@timestamp"}mutate {add_field => { "[@metadata][target_index]" => "system-log-%{+YYYY.MM.dd}" }}}# Docker 容器日志else if [container] {# 尝试解析 JSON 消息if [message] =~ /^\{.*\}$/ {json {source => "message"skip_on_invalid_json => true}}mutate {add_field => { "[@metadata][target_index]" => "docker-log-%{+YYYY.MM.dd}" }}}# 其他未分类日志else {mutate {add_field => { "[@metadata][target_index]" => "logstash-%{+YYYY.MM.dd}" }}}# 清理不需要的字段mutate {remove_field => ["agent", "ecs", "input"]}
}
output {elasticsearch {hosts => ["elasticsearch:9200"]user => "elastic"password => "mH0awV4RrkN2"index => "%{[@metadata][target_index]}"}# 调试输出（生产环境建议关闭）# stdout {#   codec => rubydebug# }
}

Kibana配置

vim kibana/config/kibana.yml

server.name: kibana
server.host: "0.0.0.0"
server.port: 5601
elasticsearch.hosts: ["http://elasticsearch:9200"]
# 中文界面
i18n.locale: "zh-CN"
# 监控配置
monitoring.ui.container.elasticsearch.enabled: true

filebeat配置

这里我加上了系统和docker的运行日志，是为了给读者扩展的，各位读者可以参考修改，让elk不仅是只会接受服务的日志，还能接受nginx日志，mysql慢日志等。

vim /filebeat/config/filebeat.yml

filebeat.inputs:# 收集系统日志- type: logenabled: truepaths:- /var/log/messages- /var/log/syslogtags: ["system"]fields:log_type: system# 收集 Docker 容器日志- type: containerenabled: truepaths:- '/var/lib/docker/containers/*/*.log'processors:- add_docker_metadata:host: "unix:///var/run/docker.sock"
# 输出到 Logstash
output.logstash:hosts: ["logstash:5045"]
# 或者直接输出到 ES（二选一）
#output.elasticsearch:
#  hosts: ["elasticsearch:9200"]
#  username: "elastic"
#  password: "mH0awV4RrkN2"
#  index: "filebeat-%{+yyyy.MM.dd}"
# Kibana 配置
setup.kibana:host: "kibana:5601"username: "elastic"password: "mH0awV4RrkN2"
# 日志级别
logging.level: info
logging.to_files: true
logging.files:path: /usr/share/filebeat/logsname: filebeatkeepfiles: 7permissions: 0644
# 启用监控
monitoring.enabled: true
monitoring.elasticsearch:hosts: ["elasticsearch:9200"]username: "elastic"password: "mH0awV4RrkN2"filebeat.inputs:# 收集系统日志- type: logenabled: truepaths:- /var/log/messages- /var/log/syslogtags: ["system"]fields:log_type: system# 收集 Docker 容器日志- type: containerenabled: truepaths:- '/var/lib/docker/containers/*/*.log'processors:- add_docker_metadata:host: "unix:///var/run/docker.sock"
# 输出到 Logstash
output.logstash:hosts: ["logstash:5045"]
# 或者直接输出到 ES（二选一）
#output.elasticsearch:
#  hosts: ["elasticsearch:9200"]
#  username: "elastic"
#  password: "mH0awV4RrkN2"
#  index: "filebeat-%{+yyyy.MM.dd}"
# Kibana 配置
setup.kibana:host: "kibana:5601"username: "elastic"password: "mH0awV4RrkN2"
# 日志级别
logging.level: info
logging.to_files: true
logging.files:path: /usr/share/filebeat/logsname: filebeatkeepfiles: 7permissions: 0644
# 启用监控
monitoring.enabled: true
monitoring.elasticsearch:hosts: ["elasticsearch:9200"]username: "elastic"password: "mH0awV4RrkN2"

compose配置

vim docker-compose.yml

version: '3.8'
services:elasticsearch:image: docker.elastic.co/elasticsearch/elasticsearch:7.12.1container_name: elasticsearchenvironment:- node.name=es-node-1- cluster.name=elk-cluster- discovery.type=single-node- bootstrap.memory_lock=true- "ES_JAVA_OPTS=-Xms512M -Xmx1g"- http.cors.enabled=true- http.cors.allow-origin=*# === 安全认证配置 ===- xpack.security.enabled=true- xpack.security.http.ssl.enabled=false  # 禁用 HTTP SSL- xpack.security.transport.ssl.enabled=false  # 禁用内部通信 SSL# 设置 elastic 用户的初始密码（重要！）- ELASTIC_PASSWORD=mH0awV4RrkN2ulimits:memlock:soft: -1hard: -1nofile:soft: 65536hard: 65536volumes:- ./elasticsearch/data:/usr/share/elasticsearch/data- ./elasticsearch/logs:/usr/share/elasticsearch/logs- ./elasticsearch/plugins:/usr/share/elasticsearch/pluginsports:- "9200:9200"- "9300:9300"networks:- elk-networkrestart: unless-stoppedhealthcheck:# 健康检查需要认证test: ["CMD-SHELL", "curl -u elastic:mH0awV4RrkN2 -f http://localhost:9200/_cluster/health || exit 1"]interval: 30stimeout: 10sretries: 5logstash:image: docker.elastic.co/logstash/logstash:7.12.1container_name: logstashenvironment:- "LS_JAVA_OPTS=-Xms512m -Xmx512m"# 使用 elastic 超级用户（后续可改为 logstash_system）- ELASTICSEARCH_USERNAME=elastic- ELASTICSEARCH_PASSWORD=mH0awV4RrkN2volumes:- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml- ./logstash/pipeline:/usr/share/logstash/pipelineports:- "5044:5044"  # TCP输入- "5045:5045"  # Beats输入- "9600:9600"  # Logstash APInetworks:- elk-networkdepends_on:elasticsearch:condition: service_healthyrestart: unless-stoppedkibana:image: docker.elastic.co/kibana/kibana:7.12.1container_name: kibanaenvironment:- ELASTICSEARCH_HOSTS=http://elasticsearch:9200- I18N_LOCALE=zh-CN# 使用 elastic 超级用户（后续可改为 kibana_system）- ELASTICSEARCH_USERNAME=elastic- ELASTICSEARCH_PASSWORD=mH0awV4RrkN2volumes:- ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.ymlports:- "5601:5601"networks:- elk-networkdepends_on:elasticsearch:condition: service_healthyrestart: unless-stoppedhealthcheck:test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]interval: 30stimeout: 10sretries: 5filebeat:image: docker.elastic.co/beats/filebeat:7.12.1container_name: filebeatuser: rootvolumes:- ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro- ./filebeat/data:/usr/share/filebeat/data# 挂载宿主机日志目录（根据实际需求调整）- /var/log:/var/log:ro# 如果需要收集 Docker 容器日志- /var/lib/docker/containers:/var/lib/docker/containers:ro- /var/run/docker.sock:/var/run/docker.sock:rocommand: filebeat -e -strict.perms=falsenetworks:- elk-networkdepends_on:- elasticsearch- logstashrestart: unless-stopped
networks:elk-network:driver: bridge
volumes:elasticsearch-data:driver: local

启动所有服务

docker-compose up -d

浏览器访问

http://127/.0.0.1:5601

二、系统集成

2.1 FileBeat

规范的流程是先通过filebeat给logstash插入到es，但是笔者实在不想折腾了，我这里就直接省去filebeat这一流程。

正常部署方式为：在每台应用服务器上安装Filebeat，配置相应的日志收集路径，指向中心化的ELK服务器地址，启动Filebeat服务

2.2 项目集成

依赖引入

net.logstash.logbacklogstash-logback-encoder7.2

本地日志文件配置，在resouces目录下创建logback-elk.xml


${log.pattern}${log.path}/info.log${log.path}/info.%d{yyyy-MM-dd}.log60${log.pattern}INFOACCEPTDENY${log.path}/error.log${log.path}/error.%d{yyyy-MM-dd}.log60${log.pattern}ERRORACCEPTDENY${LOGSTASH_HOST}50001000163845000{"app_name":"${APP_NAME}"}truetruefalse5 minutes0512