博客
关于我
强烈建议你试试无所不能的chatGPT,快点击我
ELK实战之logstash收集日志写入redis
阅读量:5878 次
发布时间:2019-06-19

本文共 4710 字,大约阅读时间需要 15 分钟。

一、部署redis

1、下载redis

[root@linux-node2 ~]# wget http://download.redis.io/releases/redis-4.0.6.tar.gz[root@linux-node2 ~]# tar -zxvf redis-4.0.6.tar.gz[root@linux-node2 ~]# mv redis-4.0.6 /usr/loca/src[root@linux-node2 ~]# cd /usr/local/src/redis-4.0.6[root@linux-node2 redis-4.0.6]# make[root@linux-node2 redis-4.0.6]# ln -sv /usr/local/src/redis-4.0.6 /usr/local/redis[root@linux-node2 redis-4.0.6]# cd /usr/local/redis

2、配置redis

[root@linux-node2 redis]# vim redis.conf bind 192.168.56.12daemonize yessave ""requirepass 123456    #开启认证[root@linux-node2 redis]# cp /usr/local/src/redis-4.0.6/src/redis-server /usr/bin/[root@linux-node2 redis]# cp /usr/local/src/redis-4.0.6/src/redis-cli /usr/bin/[root@linux-node2 redis]# redis-server /usr/local/redis/redis.conf 26617:C 02 Jan 10:35:26.801 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo26617:C 02 Jan 10:35:26.801 # Redis version=4.0.6, bits=64, commit=00000000, modified=26617:C 02 Jan 10:35:26.801 # Configuration loaded

3、测试redis

[root@linux-node2 ~]# netstat -tulnp |grep 6379tcp        0      0 192.168.56.12:6379      0.0.0.0:*               LISTEN      26618/redis-server  [root@linux-node2 redis]# redis-cli -h 192.168.56.12192.168.56.12:6379> KEYS *(error) NOAUTH Authentication required.192.168.56.12:6379> auth 123456OK192.168.56.12:6379> KEYS *(empty list or set)192.168.56.12:6379> quit

二、配置logstash将日志写入redis

1、配置logstash的system.conf

[root@linux-node1 conf.d]# vim system.confinput {  file {        path => "/var/log/messages"        type => "systemlog"        start_position => "beginning"        stat_interval => "2"  }}output {  if [type] == "systemlog" {        redis {                data_type => "list"                host => "192.168.56.12"                db => "1"                port => "6379"                password => "123456"                key => "systemlog"        }  }}

2、检测配置语法

[root@linux-node1 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/syOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase CThreads=NWARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.propertiConfiguration OK[root@linux-node1 conf.d]# systemctl restart logstash

3、写入messages日志测试

[root@linux-node1 conf.d]# cat /etc/hosts >> /var/log/messages[root@linux-node1 conf.d]# echo "helloword" >> /var/log/messages

4、登陆redis中查看

[root@linux-node2 ~]# redis-cli -h 192.168.56.12192.168.56.12:6379> KEYS *(error) NOAUTH Authentication required.192.168.56.12:6379> AUTH 123456OK192.168.56.12:6379> 192.168.56.12:6379> select 1OK192.168.56.12:6379[1]> KEYS *1) "systemlog"192.168.56.12:6379[1]> LLEN systemlog      #查看key的长度(integer) 248192.168.56.12:6379[1]> LLEN systemlog(integer) 249192.168.56.12:6379[1]> LPOP systemlog    #展示一条记录会减少一条"{\"@version\":\"1\",\"host\":\"linux-node1\",\"path\":\"/var/log/messages\",\"@timestamp\":\"2018-01-02T03:04:40.424Z\",\"type\":\"systemlog\",\"tags\":[\"_geoip_lookup_failure\"]}"192.168.56.12:6379[1]> LLEN systemlog(integer) 248

三、配置logstash从reids中取出数据到elasticsearch

1、使用linux-node2上的logstash从redis取数据

[root@linux-node2 conf.d]# vim redis-es.conf input {    redis {        data_type => "list"        host => "192.168.56.12"        db => "1"        port => "6379"        key => "systemlog"        password => "123456"    }}output {    elasticsearch {        hosts => ["192.168.56.11:9200"]        index => "redis-systemlog-%{+YYYY.MM.dd}"    }}[root@linux-node2 conf.d]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/redis-es.conf -tOpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=NWARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaultsCould not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the consoleConfiguration OK[root@linux-node2 conf.d]# systemctl restart logstash

2、从linux-node1上写入数据查看

[root@linux-node1 conf.d]# cat /etc/passwd >> /var/log/messages[root@linux-node2 ~]# redis-cli -h 192.168.56.12192.168.56.12:6379> KEYS *(error) NOAUTH Authentication required.192.168.56.12:6379> AUTH 123456OK192.168.56.12:6379> select 1OK192.168.56.12:6379[1]> KEYS *1) "systemlog"192.168.56.12:6379[1]> LLEN systemlog      #查看数据长度为38(integer) 38192.168.56.12:6379[1]> LLEN systemlog      #配置成功logstash从redis中取完数据,redis长度变成0(integer) 0

3、head插件和Kibana添加索引查看

ELK实战之logstash收集日志写入redis

ELK实战之logstash收集日志写入redis
ELK实战之logstash收集日志写入redis

转载于:https://blog.51cto.com/jinlong/2056563

你可能感兴趣的文章
python实现链表
查看>>
java查找string1和string2是不是含有相同的字母种类和数量(string1是否是string2的重新组合)...
查看>>
Android TabActivity使用方法
查看>>
Eclipse的 window-->preferences里面没有Android选项
查看>>
《麦田里的守望者》--[美]杰罗姆·大卫·塞林格
查看>>
央行下属的上海资信网络金融征信系统(NFCS)签约机构数量突破800家
查看>>
[转] Lazy evaluation
查看>>
常用查找算法总结
查看>>
被神话的大数据——从大数据(big data)到深度数据(deep data)思维转变
查看>>
修改校准申请遇到的问题
查看>>
Linux 进程中 Stop, Park, Freeze【转】
查看>>
文件缓存
查看>>
远程协助
查看>>
Scrum实施日记 - 一切从零开始
查看>>
关于存储过程实例
查看>>
配置错误定义了重复的“system.web.extensions/scripting/scriptResourceHandler” 解决办法...
查看>>
AIX 7.1 install python
查看>>
PHP盛宴——经常使用函数集锦
查看>>
重写 Ext.form.field 扩展功能
查看>>
Linux下的搜索查找命令的详解(locate)
查看>>