Elastic_stack_kibana_apache_nginx日志

配置 Elastic和kibana监控apache/nginx 日志

elastic 监控 apache 日志

• elastic stack # 开源日志平台 由elasticsearch ,logstash，kibana，beats四个组件组成
• kibana # 前端web展示日志分析和内容
• kafka # 消息队列
• filebeat # 将系统日志可视化，把数据发送给 logstash和elasticsearch
• logstash # 收集分析存储工具
• elasticsearch # 搜索分析存储的数据,基于JSON分布式搜索

拉es镜像

docker pull elasticsearch:6.7.0

创建es配置文件和挂载目录

cd  /
mkdir -p mnt/elasticsearch
cd  mnt/elasticsearch
mkdir config
mkdir matser
mkdir slave
chmod 777 master
chmod 777 slave

调高jvm线程

vim /etc/sysctl.conf
# 添加这个
vm.max_map_count=262144 
# 保存后执行这个命令
sysctl -p

matser.yml

cluster.name: elasticsearch-cluster
node.name: master
network.bind_host: 0.0.0.0
network.publish_host: `you ip`
http.port: 9200
transport.tcp.port: 9300
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: true 
node.data: true  
discovery.zen.ping.unicast.hosts: ["`you ip`:9300","`you ip`:9301"]

slave.yml

cluster.name: elasticsearch-cluster
node.name: slave
network.bind_host: 0.0.0.0
network.publish_host: `you ip`
http.port: 9202
transport.tcp.port: 9302
http.cors.enabled: true
http.cors.allow-origin: "*"
node.master: false
node.data: true  
discovery.zen.ping.unicast.hosts: ["`you ip`:9300","`you ip`:9301"]

启动master

docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d -p 9200:9200 -p 9300:9300 -v /data/dev/es/mnt/elasticsearch/config/master.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /data/dev/es/mnt/elasticsearch/master:/usr/share/elasticsearch/data --name es-master elasticsearch:6.7.0

查看是否有返回数据 curl you ip:9200

启动slave

 docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d -p 9201:9201 -p 9301:9301 -v /data/dev/es/mnt/elasticsearch/config/slave.yml:/usr/share/elasticsearch/config/elasticsearch.yml -v /data/dev/es/mnt/elasticsearch/slave:/usr/share/elasticsearch/data --name es-slave elasticsearch:6.7.0

启动kibana

docker pull kibana:6.7.0

docker run --link es-master:elasticsearch -p 5601:5601 --name kibana -d kibana:6.7.0

---

docker-compose.yml 版本

version: '2.2'
services:
  es01:
    image: elasticsearch:7.5.1
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  es02:
    image: elasticsearch:7.5.1
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01
      - cluster.initial_master_nodes=es01,es02
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic
  kibana:
    image: kibana:7.5.1
    container_name: kibana
    restart: always
    ports:
      - "5601:5601"
    environment:
      I18N_LOCALE: zh-CN #汉化
    networks:
      - elastic
    links:
      - es01:elasticsearch
volumes:
  data01:
    driver: local
  data02:
    driver: local
networks:
  elastic:
    driver: bridge

---

安装 Filebeat 查看 apache 日志 linux

1. 下载并安装 Filebeat

   curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-amd64.deb
   sudo dpkg -i filebeat-7.5.1-amd64.deb
   

2. 编辑 /etc/filebeat/filebeat.yml

   output.elasticsearch:
   hosts: ["<es_url>"]
   username: "elastic"
   password: "<password>"
   setup.kibana:
   host: "<kibana_url>"
   
   

3. 启动和配置apache模块

   sudo filebeat modules enable apache
   
   

在 /etc/filebeat/modules.d/apache.yml 文件中修改设置。

1. 启动 Filebeat

   sudo service filebeat start
   

确认已从 Filebeat apache 模块成功收到数据

安装Filebeat 查看 nginx 日志

1. 下载并安装 Filebeat

   curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.5.1-amd64.deb
   sudo dpkg -i filebeat-7.5.1-amd64.deb
   
   

2. 修改 /etc/filebeat/filebeat.yml 以设置连接信息：

   output.elasticsearch:
   hosts: ["<es_url>"]
   username: "elastic"
   password: "<password>"
   setup.kibana:
   host: "<kibana_url>"
   
   

3. 启用和配置 nginx 模块

   sudo filebeat modules enable nginx
   在 /etc/filebeat/modules.d/nginx.yml 文件中修改设置。
   
   
   

4. 启动 Filebeat

   sudo service filebeat start
   

确认已从 Filebeat nginx 模块成功收到数据