[Redis - Sentinel 설정 & Failover test] part 2

이전 글 redis install & replication 에 계속해서 진행하도록 하겠습니다.

https://jhdatabase.tistory.com/5?category=946638

[Redis 1. install & replication]

Redis(레디스; REmote DIctionary System)은 요즘 각광 받고 있는 In-memory Data(key-value) Store입니다. Redis Cluster는 최대 1000개의 노드로 구성이 가능하며, session clustering , cache server 용도로..

jhdatabase.tistory.com

서버 구성 정보

Server	Public IP	Private IP	비고
Redis1	115.85.182.24	192.168.100.51	192.168.100.54 (vip)
Redis2	49.50.164.159	192.168.100.52
Sentinel	118.67.132.167	192.168.100.53

Sentinel 구성

## pid, log 쌓일 디렉토리 생성
[root@redis1 ~]# mkdir -p /var/log/redis
[root@redis1 ~]# mkdir -p /var/run/redis

## sentinel config 추가
[root@redis1 ~]# vi /root/redis-3.2.8/sentinel.conf
daemonize yes
pidfile "/var/run/redis/sentinel.pid"
logfile "/var/log/redis/sentinel.log"

port 26379 //sentinel-port
bind 0.0.0.0

sentinel monitor mymaster 192.168.100.51 6379 2 //Failover시 Sentinel 이 자동으로 값들을 수정 Mster IP 작성 / Port / 쿼럼 수

sentinel down-after-milliseconds mymaster 5000

sentinel failover-timeout mymaster 90000       // Failover 완료되어야 하는 시간, 데이터 양에 따라 값을 증가

sentinel auth-pass mymaster test

protected-mode no    //주석 해제

## 기동

[root@redis1 redis]# redis-sentinel /root/redis-3.2.8/sentinel.conf

[root@redis1 redis]# tail -f /var/log/redis/sentinel.log

|    `-._`-._        _.-'_.-'    |

  `-._    `-._`-.__.-'_.-'    _.-'

      `-._    `-.__.-'    _.-'

          `-._        _.-'

              `-.__.-'

82232:X 19 Jan 15:19:11.059 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

82232:X 19 Jan 15:19:11.061 # Sentinel ID is 9b6a7f12634ea9cd69b5a6552e1fb1b1742b204a

82232:X 19 Jan 15:19:11.061 # +monitor master mymaster 192.168.100.51 6379 quorum 2

82232:X 19 Jan 15:19:11.061 * +slave slave 192.168.100.52:6379 192.168.100.52 6379 @ mymaster 192.168.100.51 6379

Failover Test

[root@redis1 redis]# vi /etc/init.d/redis_6379
CLIEXEC="/usr/local/bin/redis-cli -a 1234"

<Master redis shutdown>
[root@redis1 redis]# redis-cli

127.0.0.1:6379> auth test
OK
127.0.0.1:6379> shutdown
not connected>

<Slave sentinel log>
[root@redis2 ~]# tail -f /var/log/redis/sentinel.log
21229:X 19 Jan 15:50:25.068 # -sdown sentinel 9b6a7f12634ea9cd69b5a6552e1fb1b1742b204a 192.168.100.51 26379 @ mymaster 192.168.100.51 6379
21229:X 19 Jan 15:53:13.648 # +sdown master mymaster 192.168.100.51 6379
21229:X 19 Jan 15:53:13.699 # +new-epoch 1
21229:X 19 Jan 15:53:13.701 # +vote-for-leader 9b6a7f12634ea9cd69b5a6552e1fb1b1742b204a 1
21229:X 19 Jan 15:53:13.707 # +odown master mymaster 192.168.100.51 6379 #quorum 3/2
21229:X 19 Jan 15:53:13.707 # Next failover delay: I will not start a failover before Wed Jan 19 15:56:14 2022
21229:X 19 Jan 15:53:14.801 # +config-update-from sentinel 9b6a7f12634ea9cd69b5a6552e1fb1b1742b204a 192.168.100.51 26379 @ mymaster 192.168.100.51 6379
21229:X 19 Jan 15:53:14.801 # +switch-master mymaster 192.168.100.51 6379 192.168.100.52 6379
21229:X 19 Jan 15:53:14.801 * +slave slave 192.168.100.51:6379 192.168.100.51 6379 @ mymaster 192.168.100.52 6379
21229:X 19 Jan 15:53:19.846 # +sdown slave 192.168.100.51:6379 192.168.100.51 6379 @ mymaster 192.168.100.52 6379

<Sentinel server log>
22366:X 19 Jan 15:53:14.801 # +switch-master mymaster 192.168.100.51 6379 192.168.100.52 6379
22366:X 19 Jan 15:53:14.801 * +slave slave 192.168.100.51:6379 192.168.100.51 6379 @ mymaster 192.168.100.52 6379
22366:X 19 Jan 15:53:19.834 # +sdown slave 192.168.100.51:6379 192.168.100.51 6379 @ mymaster 192.168.100.52 6379

■ Slave가 Master 로 변하는것 확인

<Redis2>
[root@redis2 ~]# redis-cli
127.0.0.1:6379> auth test
OK
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

## Redis1번에 redis2를 Master로 보도록 다시 설정 후 start
[root@redis1 redis]# vi /etc/redis/6379.conf
slaveof 192.168.100.52 6379

[root@redis1 redis]# /etc/init.d/redis_6379 start
Starting Redis server...

127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.100.52
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:12978
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

## Redis2번에서 다시 replication 상태 확인
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.100.51,port=6379,state=online,offset=2469,lag=0
master_repl_offset:2469
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:2468

VIP 설정(ALL)

REDIS 장애 조치를 수행할 때 VIP를 새로운 기본 Redis 서버로 드리프트 설정

■ VIP 생성

[root@redis1 ~]# cd /etc/sysconfig/network-scripts/

[root@redis1 network-scripts]# vi ifcfg-eth1:1
DEVICE=eth1:1
BOOTPROTO=static
IPADDR=192.168.100.54
NETMASK=255.255.255.0

## 모든서버에 만들어 놓긴하나 ifup은 현재 redis master 서버에만 띄어놔야함.
[root@redis1 network-scripts]# ifup ifcfg-eth1:1

[root@redis1 network-scripts]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.41.219.90  netmask 255.255.254.0  broadcast 10.41.219.255
        ether f2:20:cd:e8:e2:4e  txqueuelen 1000  (Ethernet)
        RX packets 1494866  bytes 858583062 (818.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 255428  bytes 154136979 (146.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.51  netmask 255.255.255.0  broadcast 192.168.100.255
        ether f2:d7:cf:cf:d0:50  txqueuelen 1000  (Ethernet)
        RX packets 139373  bytes 12476244 (11.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 131495  bytes 15705993 (14.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.54  netmask 255.255.255.0  broadcast 192.168.100.255
        ether f2:d7:cf:cf:d0:50  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 48512  bytes 5870860 (5.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 48512  bytes 5870860 (5.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

■ failover 스크립트 생성 & 경로 기입

[root@redis1 redis]# mkdir failover

[root@redis1 redis]# cd failover/

[root@redis1 failover]# vi failover.sh

#!/bin/sh
_DEBUG="on"
DEBUGFILE=/tmp/sentinel_failover.log

VIP='192.168.100.54'
MASTERIP=${6}
MASK='24'
IFACE='eth1:1'
MYIP='192.168.100.52'

DEBUG () {
        if [ "$_DEBUG" = "on" ]; then
                $@
        fi
}
set -e

DEBUG date >> ${DEBUGFILE}
DEBUG echo $@ >> ${DEBUGFILE}

DEBUG echo "Master: ${MASTERIP} My IP: ${MYIP}" >> ${DEBUGFILE}

if [ ${MASTERIP} = ${MYIP} ]; then
##IPUP
                sudo /sbin/ip addr add ${VIP}/${MASK} dev ${IFACE}
                DEBUG echo "sudo /sbin/ip addr add ${VIP}/${MASK} dev ${IFACE}" >> ${DEBUGFILE}
        exit 0
else
##IPDOWN
                sudo /sbin/ip addr del ${VIP}/${MASK} dev ${IFACE}
                DEBUG echo "sudo /sbin/ip addr del ${VIP}/${MASK} dev ${IFACE}" >> ${DEBUGFILE}
        exit 0
fi
exit 1

## failover 스크립트 경로 기입
[root@redis1 failover]# vi /root/redis-3.2.8/sentinel.conf
sentinel client-reconfig-script mymaster /root/failover/failover.sh

## Sentinel 재시작 -> kill로 죽이고 다시 시작함
[root@redis1 ~]# ps -ef | grep redis
root      84315      1  0 15:50 ?        00:00:05 redis-sentinel 0.0.0.0:26379 [sentinel]
root      84870      1  0 15:58 ?        00:00:02 /usr/local/bin/redis-server 0.0.0.0:6379
root      87361  60443  0 16:36 pts/0    00:00:00 grep --color=auto redis

[root@redis1 ~]# kill -9 84315

[root@redis1 ~]# redis-sentinel /root/redis-3.2.8/sentinel.conf

## Sentinel 접속 -> port 26379

[root@redis1 ~]# redis-cli -p 26379

127.0.0.1:26379> info sentinel

# Sentinel

sentinel_masters:1

sentinel_tilt:0

sentinel_running_scripts:0

sentinel_scripts_queue_length:0

sentinel_simulate_failure_flags:0

master0:name=mymaster,status=ok,address=192.168.100.52:6379,slaves=1,sentinels=3

127.0.0.1:26379> sentinel master mymaster
1) "name"
2) "mymaster"
3) "ip"
4) "192.168.100.52"
5) "port"
6) "6379"
7) "runid"
8) "556a60a39d4f03d118b454de29f63bd7444a9342"
9) "flags"
10) "master"
11) "link-pending-commands"
12) "0"
13) "link-refcount"
14) "1"
15) "last-ping-sent"
16) "0"
17) "last-ok-ping-reply"
18) "393"
19) "last-ping-reply"
20) "393"
21) "down-after-milliseconds"
22) "5000"
23) "info-refresh"
24) "1048"
25) "role-reported"
26) "master"
27) "role-reported-time"
28) "823909"
29) "config-epoch"
30) "1"
31) "num-slaves"
32) "1"
33) "num-other-sentinels"
34) "2"
35) "quorum"
36) "2"
37) "failover-timeout"
38) "180000"
39) "parallel-syncs"
40) "1"
41) "client-reconfig-script"
42) "/root/failover/failover.sh" => 해당 스크립트가 잘 적용됐는지 확인

■ VIP Failover 테스트

현재 redis2번에 vip 띄어놓은 상태(redis2 -> Master)

[root@redis2 network-scripts]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.41.181.58  netmask 255.255.254.0  broadcast 10.41.181.255
        ether f2:20:cd:f1:fb:b5  txqueuelen 1000  (Ethernet)
        RX packets 66780  bytes 67897391 (64.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16017  bytes 7671872 (7.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.52  netmask 255.255.255.0  broadcast 192.168.100.255
        ether f2:d7:cf:d3:f4:fa  txqueuelen 1000  (Ethernet)
        RX packets 142269  bytes 12821480 (12.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 137434  bytes 16205127 (15.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.54  netmask 255.255.255.0  broadcast 192.168.100.255
        ether f2:d7:cf:d3:f4:fa  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 45892  bytes 5540026 (5.2 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 45892  bytes 5540026 (5.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@redis2 ~]# redis-cli -a test info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.100.51,port=6379,state=online,offset=771956,lag=1
master_repl_offset:772099
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:772098

## shutdown
[root@redis2 ~]# redis-cli
127.0.0.1:6379> auth test
OK
127.0.0.1:6379> shutdown

[root@redis2 ~]# cat /tmp/sentinel_failover.log
Wed Jan 19 17:01:41 KST 2022
mymaster leader start 192.168.100.52 6379 192.168.100.51 6379
Master: 192.168.100.51 My IP: 192.168.100.52
sudo /sbin/ip addr del 192.168.100.54/24 dev eth1:1

##VIP DOWN
[root@redis2 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.41.181.58  netmask 255.255.254.0  broadcast 10.41.181.255
        ether f2:20:cd:f1:fb:b5  txqueuelen 1000  (Ethernet)
        RX packets 67600  bytes 67964115 (64.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 16254  bytes 7766680 (7.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.100.52  netmask 255.255.255.0  broadcast 192.168.100.255
        ether f2:d7:cf:d3:f4:fa  txqueuelen 1000  (Ethernet)
        RX packets 147421  bytes 13312039 (12.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 142392  bytes 16783392 (16.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 47470  bytes 5716137 (5.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 47470  bytes 5716137 (5.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

<Redis2>
## Role 이 Slave로 바뀜
[root@redis2 ~]# /etc/init.d/redis_6379  start

[root@redis2 ~]# redis-cli
127.0.0.1:6379> auth test
OK

127.0.0.1:6379> info replication
# Replication
role:slave
master_host:192.168.100.51
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:32981
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

<Redis1>
[root@redis1 ~]# redis-cli -h 192.168.100.54 -a test
192.168.100.54:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=192.168.100.52,port=6379,state=online,offset=118926,lag=1
master_repl_offset:118926
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:118925

정상적으로 failover까지 되는것을 확인 하였습니다.

저작자표시

'Others > Opensource Tool' 카테고리의 다른 글

[Ansible - AWX설치] (0)	2022.04.24
[pt-query-digest - Mysql slowquery 분석 ] (0)	2022.03.12
[Ansible - MySQL 설치 & 배포] (0)	2022.02.05
[Pacemaker & corosync 구성 Vip failover] (0)	2022.01.20
[Redis - install & replication] part 1 (0)	2022.01.20

서버 구성 정보

Sentinel 구성

Failover Test

VIP 설정(ALL)

'Others > Opensource Tool' 카테고리의 다른 글

티스토리툴바