由于在生產(chǎn)環(huán)境使用了mysqlcluster,需要實現(xiàn)高可用負載均衡,這里提供了keepalived+haproxy來實現(xiàn).
keepalived主要功能是實現(xiàn)真實機器的故障隔離及負載均衡器間的失敗切換.可在第3,4,5層交換.它通過VRRPv2(Virtual Router Redundancy Protocol) stack實現(xiàn)的.
Layer3:Keepalived會定期向服務(wù)器群中的服務(wù)器.發(fā)送一個ICMP的數(shù)據(jù)包（既我們平時用的Ping程序）,如果發(fā)現(xiàn)某臺服務(wù)的IP地址沒有激活，Keepalived便報告這臺服務(wù)器失效，并將它從服務(wù)器群中剔除，這種情況的典型例子是某臺服務(wù)器被非法關(guān)機。Layer3的方式是以服務(wù)器的IP地址是否有效作為服務(wù)器工作正常與否的標準。
Layer4:主要以TCP端口的狀態(tài)來決定服務(wù)器工作正常與否。如web server的服務(wù)端口一般是80，如果Keepalived檢測到80端口沒有啟動，則Keepalived將把這臺服務(wù)器從服務(wù)器群中剔除。
Layer5:在網(wǎng)絡(luò)上占用的帶寬也要大一些。Keepalived將根據(jù)用戶的設(shè)定檢查服務(wù)器程序的運行是否正常，如果與用戶的設(shè)定不相符，則Keepalived將把服務(wù)器從服務(wù)器群中剔除。
Software Design
keepalived啟動后會有單個進程
8352 ? Ss 0:00 /usr/sbin/keepalived 8353 ? S 0:00 \_ /usr/sbin/keepalived 8356 ? S 0:01 \_ /usr/sbin/keepalived
父進程:內(nèi)存管理,子進程管理等等
子進程:VRRP子進程
子進程:Healthchecking 子進程
實例
2臺mysqlcluster 10.1.6.203 master 10.1.6.205 backup
vip 10.1.6.173
目的訪問10.1.6.173 3366端口 分別輪詢通過haproxy轉(zhuǎn)發(fā)到10.1.6.203 3306 和10.1.6.205 3306
mysqlcluster搭建參照之前博客,這里在2臺機上安裝keepalived
root@10.1.6.203:~# apt-get install keepalived
root@10.1.6.203:~# cat /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
script "killall -0 haproxy" # verify the pid existance
interval 2 # check every 2 seconds
weight -2 # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface eth1 # interface to monitor
state MASTER
virtual_router_id 51 # Assign one ID for this route
priority 101 # 101 on master, 100 on backup
nopreempt
debug
virtual_ipaddress {
10.1.6.173
}
track_script { #注意大括號空格
chk_haproxy
}
notify_master /etc/keepalived/scripts/start_haproxy.sh #表示當切換到master狀態(tài)時,要執(zhí)行的腳本
notify_fault /etc/keepalived/scripts/stop_keepalived.sh #故障時執(zhí)行的腳本
notify_stop /etc/keepalived/scripts/stop_haproxy.sh #keepalived停止運行前運行notify_stop指定的腳本 }
VRRPD配置包括三個類:
這里使用了 VRRP實例, VRRP腳本
注意配置選項:
stat:指定instance(Initial)的初始狀態(tài)，就是說在配置好后，這臺服務(wù)器的初始狀態(tài)就是這里指定的，但這里指定的不算，還是得要通過競選通過優(yōu)先級來確定，里如果這里設(shè)置為master，但如若他的優(yōu)先級不及另外一臺，那么這臺在發(fā)送通告時，會發(fā)送自己的優(yōu)先級，另外一臺發(fā)現(xiàn)優(yōu)先級不如自己的高，那么他會就回搶占為master
vrrp_script chk_haproxy {
script "killall -0 haproxy" # verify the pid existance
interval 2 # check every 2 seconds 腳本執(zhí)行間隔
weight -2 # add 2 points of prio if OK 腳本結(jié)果導(dǎo)致的優(yōu)先級變更：2表示優(yōu)先級+2；-2則表示優(yōu)先級-2
}
然后在實例(vrrp_instance)里面引用，有點類似腳本里面的函數(shù)引用一樣：先定義，后引用函數(shù)名
track_script {
chk_haproxy
}
注意：VRRP腳本(vrrp_script)和VRRP實例(vrrp_instance)屬于同一個級別
root@10.1.6.203:scripts# cat start_haproxy.sh #!/bin/bash sleep 5 get=`ip addr |grep 10.1.6.173 |wc -l` echo $get >> /etc/keepalived/scripts/start_ha.log if [ $get -eq 1 ] then echo "`date +%c` success to get vip" >> /etc/keepalived/scripts/start_ha.log /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg else echo "`date +%c` can not get vip" >> /etc/keepalived/scripts/start_ha.log fi root@10.1.6.203:scripts# cat stop_keepalived.sh #!/bin/bash pid=`pidof keepalived` if [ $pid == "" ] then echo "`date +%c` no keepalived process id" >> /etc/keepalived/scripts/stop_keep.log else echo "`date +%c` will stop keepalived " >> /etc/keepalived/scripts/stop_keep.log /etc/init.d/keepalived stop fi /etc/init.d/keepalived stop root@10.1.6.203:scripts# cat stop_haproxy.sh #!/bin/bash pid=`pidof haproxy` echo "`date +%c` stop haproxy" >> /etc/keepalived/scripts/stop_ha.log kill -9 $pid
同理配置10.1.6.205
root@10.1.6.205:~# cat /etc/keepalived/keepalived.conf
vrrp_script chk_haproxy {
script "killall -0 haproxy" # verify the pid existance
interval 2 # check every 2 seconds
weight 2 # add 2 points of prio if OK
}
vrrp_instance VI_1 {
interface eth1 # interface to monitor
state BACKUP
virtual_router_id 51 # Assign one ID for this route
priority 100 # 101 on master, 100 on backup
virtual_ipaddress {
10.1.6.173
}
track_script {
chk_haproxy
}
notify_master /etc/keepalived/scripts/start_haproxy.sh
notify_fault /etc/keepalived/scripts/stop_keepalived.sh
notify_stop /etc/keepalived/scripts/stop_haproxy.sh
}
HAProxy
下面再介紹下haproxy
HAProxy是一款基于TCP(第四層)和HTTP(第七層)應(yīng)用的代理軟件,它也可作為負載均衡器.可以支持數(shù)以萬計的并發(fā)連接.同時可以保護服務(wù)器不暴露到網(wǎng)絡(luò)上,通過端口映射.它還自帶監(jiān)控服務(wù)器狀態(tài)的頁面.
安裝haproxy
wget -O/tmp/haproxy-1.4.22.tar.gz http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.22.tar.gz tar xvfz /tmp/haproxy-1.4.22.tar.gz -C /tmp/ cd /tmp/haproxy-1.4.22 make TARGET=linux26 make install
haproxy需要對每一個mysqlcluster服務(wù)器進行健康檢查
1.在2臺主機分別配置haproxy.cfg
root@10.1.6.203:scripts# cat /etc/haproxy/haproxy.cfg global maxconn 51200 #默認最大連接數(shù) #uid 99 #gid 99 daemon #以后臺形式運行haproxy #quiet nbproc 1 #進程數(shù)量(可以設(shè)置多個進程提高性能) pidfile /etc/haproxy/haproxy.pid #haproxy的pid存放路徑,啟動進程的用戶必須有權(quán)限訪問此文件 defaults mode tcp #所處理的類別 (#7層 http;4層tcp ) option redispatch #serverId對應(yīng)的服務(wù)器掛掉后,強制定向到其他健康的服務(wù)器 option abortonclose #當服務(wù)器負載很高的時候，自動結(jié)束掉當前隊列處理比較久的連接 timeout connect 5000s #連接超時 timeout client 50000s #客戶端超時 timeout server 50000s #服務(wù)器超時 log 127.0.0.1 local0 #錯誤日志記錄 balance roundrobin #默認的負載均衡的方式,輪詢方式 listen proxy bind 10.1.6.173:3366 #監(jiān)聽端口 mode tcp #http的7層模式 option httpchk #心跳檢測的文件 server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 #服務(wù)器定義,check inter 12000是檢測心跳頻率 rise 3是3次正確認為服務(wù)器可用， fall 3是3次失敗認為服務(wù)器不可用，weight代表權(quán)重 server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 listen haproxy_stats mode http bind 10.1.6.173:8888 option httplog stats refresh 5s stats uri /status #網(wǎng)站健康檢測URL，用來檢測HAProxy管理的網(wǎng)站是否可以用，正常返回200，不正常返回503 stats realm Haproxy Manager stats auth admin:p@a1SZs24 #賬號密碼 root@10.1.6.205:~$ cat /etc/haproxy/haproxy.cfg global maxconn 51200 #uid 99 #gid 99 daemon #quiet nbproc 1 pidfile /etc/haproxy/haproxy.pid defaults mode tcp option redispatch option abortonclose timeout connect 5000s timeout client 50000s timeout server 50000s log 127.0.0.1 local0 balance roundrobin listen proxy bind 10.1.6.173:3366 mode tcp option httpchk server db1 10.1.6.203:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 server db2 10.1.6.205:3306 weight 1 check port 9222 inter 12000 rise 3 fall 3 listen haproxy_stats mode http bind 10.1.6.173:8888 option httplog stats refresh 5s stats uri /status stats realm Haproxy Manager stats auth admin:p@a1SZs24
2.安裝xinetd
root@10.1.6.203:~# apt-get install xinetd
3.在每個節(jié)點添加xinetd服務(wù)腳本和mysqlchk端口號
root@10.1.6.203:~# vim /etc/xinetd.d/mysqlchk
# default: on
# description: mysqlchk
service mysqlchk #需要在servive定義
{
flags = REUSE
socket_type = stream
port = 9222
wait = no
user = nobody
server = /opt/mysqlchk
log_on_failure += USERID
disable = no
per_source = UNLIMITED
bind = 10.1.6.173
}
root@10.1.6.203:~# vim /etc/services
mysqlchk 9222/tcp # mysqlchk
4.編寫mysqlchk監(jiān)控服務(wù)腳本
root@10.1.6.203:~# ls -l /opt/mysqlchk
-rwxr--r-- 1 nobody root 1994 2013-09-17 11:27 /opt/mysqlchk
root@10.1.6.203:~# cat /opt/mysqlchk
#!/bin/bash
#
# This script checks if a mysql server is healthy running on localhost. It will
# return:
# "HTTP/1.x 200 OK\r" (if mysql is running smoothly)
# - OR -
# "HTTP/1.x 500 Internal Server Error\r" (else)
#
# The purpose of this script is make haproxy capable of monitoring mysql properly
#
MYSQL_HOST="localhost"
MYSQL_SOCKET="/var/run/mysqld/mysqld.sock"
MYSQL_USERNAME="mysqlchkusr" #該賬戶密碼需要在mysql里添加
MYSQL_PASSWORD="secret"
MYSQL_OPTS="-N -q -A"
TMP_FILE="/dev/shm/mysqlchk.$$.out"
ERR_FILE="/dev/shm/mysqlchk.$$.err"
FORCE_FAIL="/dev/shm/proxyoff"
MYSQL_BIN="/opt/mysqlcluster/mysql-cluster-gpl-7.2.6-linux2.6-x86_64/bin/mysql"
CHECK_QUERY="select 1"
preflight_check()
{
for I in "$TMP_FILE" "$ERR_FILE"; do
if [ -f "$I" ]; then
if [ ! -w $I ]; then
echo -e "HTTP/1.1 503 Service Unavailable\r\n"
echo -e "Content-Type: Content-Type: text/plain\r\n"
echo -e "\r\n"
echo -e "Cannot write to $I\r\n"
echo -e "\r\n"
exit 1
fi
fi
done
}
return_ok()
{
echo -e "HTTP/1.1 200 OK\r\n"
echo -e "Content-Type: text/html\r\n"
echo -e "Content-Length: 43\r\n"
echo -e "\r\n"
echo -e "MySQL is running.\r\n"
echo -e "\r\n"
rm $ERR_FILE $TMP_FILE
exit 0
}
return_fail()
{
echo -e "HTTP/1.1 503 Service Unavailable\r\n"
echo -e "Content-Type: text/html\r\n"
echo -e "Content-Length: 42\r\n"
echo -e "\r\n"
echo -e "MySQL is *down*.\r\n"
sed -e 's/\n$/\r\n/' $ERR_FILE
echo -e "\r\n"
rm $ERR_FILE $TMP_FILE
exit 1
}
preflight_check
if [ -f "$FORCE_FAIL" ]; then
echo "$FORCE_FAIL found" > $ERR_FILE
return_fail;
fi
$MYSQL_BIN $MYSQL_OPTS --host=$MYSQL_HOST --socket=$MYSQL_SOCKET --user=$MYSQL_USERNAME --password=$MYSQL_PASSWORD -e "$CHECK_QUERY" > $TMP_FILE 2> $ERR_FILE
if [ $? -ne 0 ]; then
return_fail;
fi
return_ok;
測試
2個節(jié)點開啟keepalived(主節(jié)點會獲得vip,自動拉起haproxy),xinetd
