165 lines
No EOL
39 KiB
HTML
165 lines
No EOL
39 KiB
HTML
<!DOCTYPE html>
|
||
<html>
|
||
<head>
|
||
<meta charset="utf-8">
|
||
<meta name="tool" content="leanote-desktop-app">
|
||
<title>Codis安装部署全架构</title>
|
||
<style>
|
||
|
||
*{font-family:"lucida grande","lucida sans unicode",lucida,helvetica,"Hiragino Sans GB","Microsoft YaHei","WenQuanYi Micro Hei",sans-serif;}
|
||
|
||
body {
|
||
margin: 0;
|
||
}
|
||
|
||
/*公用文字样式*/
|
||
h1{font-size:30px}h2{font-size:24px}h3{font-size:18px}h4{font-size:14px}
|
||
.note-container{
|
||
width:850px;
|
||
margin:auto;
|
||
padding: 10px 20px;
|
||
box-shadow: 1px 1px 10px #eee;
|
||
}
|
||
#title {
|
||
margin: 0;
|
||
}
|
||
table {
|
||
margin-bottom: 16px;
|
||
border-collapse: collapse;
|
||
}
|
||
table th, table td {
|
||
padding: 6px 13px;
|
||
border: 1px solid #ddd;
|
||
}
|
||
table th {
|
||
font-weight: bold;
|
||
}
|
||
|
||
table tr {
|
||
background-color: none;
|
||
border-top: 1px solid #ccc;
|
||
}
|
||
table tr:nth-child(2n) {
|
||
background-color: rgb(247, 247, 249);
|
||
}
|
||
.mce-item-table, .mce-item-table td, .mce-item-table th, .mce-item-table caption {
|
||
border: 1px solid #ddd;
|
||
border-collapse: collapse;
|
||
padding: 6px 13px;
|
||
}
|
||
blockquote {
|
||
border-left-width:10px;
|
||
background-color:rgba(128,128,128,0.05);
|
||
border-top-right-radius:5px;
|
||
border-bottom-right-radius:5px;
|
||
padding:15px 20px;
|
||
border-left:5px solid rgba(128,128,128,0.075);
|
||
}
|
||
blockquote p {
|
||
margin-bottom:1.1em;
|
||
font-size:1em;
|
||
line-height:1.45
|
||
}
|
||
blockquote ul:last-child,blockquote ol:last-child {
|
||
margin-bottom:0
|
||
}
|
||
pre {
|
||
padding: 18px;
|
||
background-color: #f7f7f9;
|
||
border: 1px solid #e1e1e8;
|
||
border-radius: 3px;
|
||
display: block;
|
||
}
|
||
code {
|
||
padding: 2px 4px;
|
||
font-size: 90%;
|
||
color: #c7254e;
|
||
white-space: nowrap;
|
||
background-color: #f9f2f4;
|
||
border-radius: 4px;
|
||
}
|
||
.footnote {
|
||
vertical-align: top;
|
||
position: relative;
|
||
top: -0.5em;
|
||
font-size: .8em;
|
||
}
|
||
|
||
hr {
|
||
margin:2em 0
|
||
}
|
||
img {
|
||
max-width:100%
|
||
}
|
||
pre {
|
||
word-break:break-word
|
||
}
|
||
p,pre,pre.prettyprint,blockquote {
|
||
margin:0 0 1.1em
|
||
}
|
||
hr {
|
||
margin:2em 0
|
||
}
|
||
img {
|
||
max-width:100%
|
||
}
|
||
.sequence-diagram,.flow-chart {
|
||
text-align:center;
|
||
margin-bottom:1.1em
|
||
}
|
||
.sequence-diagram text,.flow-chart text {
|
||
font-size:15px !important;
|
||
font-family:"Source Sans Pro",sans-serif !important
|
||
}
|
||
.sequence-diagram [fill="#ffffff"],.flow-chart [fill="#ffffff"] {
|
||
fill:#f6f6f6
|
||
}
|
||
.sequence-diagram [stroke="#000000"],.flow-chart [stroke="#000000"] {
|
||
stroke:#3f3f3f
|
||
}
|
||
.sequence-diagram text[stroke="#000000"],.flow-chart text[stroke="#000000"] {
|
||
stroke:none
|
||
}
|
||
.sequence-diagram [fill="#000"],.flow-chart [fill="#000"],.sequence-diagram [fill="#000000"],.flow-chart [fill="#000000"],.sequence-diagram [fill="black"],.flow-chart [fill="black"] {
|
||
fill:#3f3f3f
|
||
}
|
||
ul,ol {
|
||
margin-bottom:1.1em
|
||
}
|
||
ul ul,ol ul,ul ol,ol ol {
|
||
margin-bottom:1.1em
|
||
}
|
||
kbd {
|
||
padding:.1em .6em;
|
||
border:1px solid rgba(63,63,63,0.25);
|
||
-webkit-box-shadow:0 1px 0 rgba(63,63,63,0.25);
|
||
box-shadow:0 1px 0 rgba(63,63,63,0.25);
|
||
font-size:.7em;
|
||
font-family:sans-serif;
|
||
background-color:#fff;
|
||
color:#333;
|
||
border-radius:3px;
|
||
display:inline-block;
|
||
margin:0 .1em;
|
||
white-space:nowrap
|
||
}
|
||
.toc ul {
|
||
list-style-type:none;
|
||
margin-bottom:15px
|
||
}
|
||
</style>
|
||
<!-- 该css供自定义样式 -->
|
||
<link href="../leanote-html.css" rel="stylesheet">
|
||
</head>
|
||
|
||
<body>
|
||
|
||
<div class="note-container">
|
||
<h1 class="title" id="leanote-title">Codis安装部署全架构</h1>
|
||
<div class="content-html" id="leanote-content"><div>Codis安装部署全架构</div><div><br></div><div>Codis 是一个分布式 Redis 解决方案, 对于上层的应用来说, 连接到 Codis Proxy 和连接原生的 Redis Server 没有明显的区别 (不支持的命令列表), 上层应用可以像使用单机的 Redis 一样使用, Codis 底层会处理请求的转发, 不停机的数据迁移等工作, 所有后边的一切事情, 对于前面的客户端来说是透明的, 可以简单的认为后边连接的是一个内存无限大的 Redis 服务。</div><div><br></div><div><br></div><div>Codis 由四部分组成:</div><div><br></div><div><br></div><div>Codis Proxy (codis-proxy)</div><div>Codis Manager (codis-config)</div><div>Codis Redis (codis-server)</div><div>ZooKeeper</div><div><br></div><div><br></div><div>codis-proxy 是客户端连接的 Redis 代理服务, codis-proxy 本身实现了 Redis 协议, 表现得和一个原生的 Redis 没什么区别 (就像 Twemproxy), 对于一个业务来说, 可以部署多个 codis-proxy, codis-proxy 本身是无状态的。</div><div><br></div><div><br></div><div>codis-config 是 Codis 的管理工具, 支持包括, 添加/删除 Redis 节点, 添加/删除 Proxy 节点, 发起数据迁移等操作. codis-config 本身还自带了一个 http server, 会启动一个 dashboard，用户可以直接在浏览器上观察 Codis 集群的运行状态。</div><div><br></div><div><br></div><div>codis-server 是 Codis 项目维护的一个 Redis 分支, 基于 2.8.13 开发, 加入了 slot 的支持和原子的数据迁移指令。Codis 上层的 codis-proxy 和 codis-config 只能和这个版本的 Redis 交互才能正常运行。</div><div><br></div><div><br></div><div>Codis 依赖 ZooKeeper 来存放数据路由表和 codis-proxy 节点的元信息, codis-config 发起的命令都会通过 ZooKeeper 同步到各个存活的 codis-proxy。</div><div><br></div><div><br></div><div>Codis 支持按照 Namespace 区分不同的产品, 拥有不同的 product name 的产品, 各项配置都不会冲突。</div><div><br></div><div><br></div><div>Codis架构图</div><div><br></div><div><img id="__LEANOTE_D_IMG_1512010880057" src="Codis安装部署全架构_files/5a1f7480c464842ba8000001.png" alt="" data-mce-src="Codis安装部署全架构_files/5a1f7480c464842ba8000001.png"></div><div><br></div><div>前期规划</div><div><br></div><div><br></div><div>机器与应用列表：</div><div>操作系统：CentOS6.5</div><div><br></div><div><br></div><div>1，操作系统：CentOS6.5</div><div>192.168.88.106 codis-server1</div><div>192.168.88.107 codis-server2</div><div>192.168.88.108 codis-server3</div><div>192.168.88.111 codis-ha1</div><div>192.168.88.112 codis-ha2</div><div>192.168.88.113 zookeeper-1（codis-proxy-1）</div><div>192.168.88.114 zookeeper-2（codis-proxy-2）</div><div>192.168.88.115 zookeeper-3（codis-proxy-3）</div><div>2，硬件配置：</div><div>ha mem：8G cpu：4 disk：100G</div><div>zookeeper mem：16G cpu：8 disk：300G</div><div>codis-server mem：16G cpu：8 disk：200G</div><div><br></div><div><br></div><div>3，架构应用</div><div>1）， HA（192.168.88.111、192.168.88.112、VIP 192.168.88.159）</div><div>hostname:codisha-1 apps：keepalived master,haproxy prots:19000</div><div>hostname:codisha-2 apps：keepalived slave ,haproxy，codis-config prots:19000,18087</div><div><br></div><div><br></div><div>2），zookeeper+codis-proxy</div><div>hostname:zookeeper-1 apps: zookeeper1, codis_proxy_1 prots:2811,19000</div><div>hostname:zookeeper-2 apps: zookeeper2, codis_proxy_2 prots:2811,19000</div><div>hostname:zookeeper-3 apps: zookeeper3, codis_proxy_3 prots:2811,19000</div><div><br></div><div><br></div><div>3），codis-server</div><div>codis-server(192.168.88.106、192.168.88.107、192.168.88.108)</div><div>hostname: codis-server1 apps: codis_server_master,slave, ports:6379,6380，6389,6390hostname: codis-server2 apps: codis_server_master,slave ports:6379,6380，</div><div><br></div><div><br></div><div>6389,6390</div><div>hostname: codis-server3 apps: codis_server_master,slave ports:6379,6380，6389,6390</div><div><br></div><div><br></div><div><br></div><div><br></div><div>部署安装详细步骤</div><div><br></div><div><br></div><div>一，安装zookeeper</div><div><br></div><div><br></div><div>1，配置hosts文件 (所有机器上配置)</div><div>vim /etc/hosts</div><div>192.168.88.106 codis-server1</div><div>192.168.88.107 codis-server2</div><div>192.168.88.108 codis-server3</div><div>192.168.88.111 codis-ha1</div><div>192.168.88.112 codis-ha2</div><div>192.168.88.113 zookeeper-1(codis-proxy-1)</div><div>192.168.88.114 zookeeper-2(codis-proxy-2)</div><div>192.168.88.115 zookeeper-3(codis-proxy-3)</div><div><br></div><div><br></div><div><br></div><div><br></div><div>2，安装java环境</div><div>ZooKeeper 要求 JAVA 的环境才能运行，并且需要 JAVA6 以上的版本，可以</div><div><br></div><div><br></div><div>从 SUN 官网上下载，并对 JAVA 环境变量进行设置。</div><div>yum -y install java-1.7.0-openjdk-devel</div><div>java -version</div><div>java version "1.7.0_75"</div><div>OpenJDK Runtime Environment (rhel-2.5.4.0.el6_6-x86_64 u75-b13)</div><div>OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)</div><div><br></div><div><br></div><div><br></div><div><br></div><div>3，安装zookeeper</div><div>wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz</div><div>tar zxvf zookeeper-3.4.6.tar.gz</div><div>mv zookeeper-3.4.6 /usr/local/zookeeper</div><div>mkdir -p /data/zookeeper/{data,logs}</div><div>配置zoo.cfg</div><div>vim /usr/local/zookeeper/conf/zoo.cfg </div><div>tickTime=2000</div><div>initLimit=5</div><div>syncLimit=2</div><div>dataDir=/data/zookeeper/data</div><div>#dataLogDir=/data/zookeeper/logs</div><div>clientPort=2181</div><div>server.1=zookeeper-1:2888:3888</div><div>server.2=zookeeper-2:2888:3888</div><div>server.3=zookeeper-3:2888:3888</div><div>上述配置内容说明，可以参考</div><div>http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_RunningReplicatedZooKeeper</div><div><br></div><div><br></div><div><br></div><div><br></div><div>4，设置myid</div><div>在我们配置的dataDir指定的目录下面，创建一个myid文件，里面内容为一个数字，用来标识当前主机，conf/zoo.cfg文件中配置的server.X中X为什么数字，则myid文件中就输入这个数字</div><div>[root@zookeeper-1 ~]# echo 1 > /data/zookeeper/data/myid</div><div>[root@zookeeper-2 ~]# echo 2 > /data/zookeeper/data/myid</div><div>[root@zookeeper-3 ~]# echo 3 > /data/zookeeper/data/myid</div><div><br></div><div><br></div><div><br></div><div><br></div><div>5，启动zookeeper</div><div>启动顺序zookeeper-1>zookeeper-2>zookeeper-3</div><div>[root@zookeeper-1 zookeeper]# zkServer.sh start </div><div>JMX enabled by default</div><div>Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg</div><div>Starting zookeeper ... STARTED</div><div>[root@zookeeper-1 zookeeper]# zkServer.sh status</div><div>JMX enabled by default</div><div>Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg</div><div>Mode: leader</div><div>可以看到最先开始启动的是leader，其他两个是follower</div><div><br></div><div><br></div><div>设置开机启动</div><div>vim /etc/rc.local</div><div>/usr/local/zookeeper/bin/zkServer.sh start</div><div><br></div><div><br></div><div>设置环境变量</div><div>vim /etc/profile</div><div>export ZOOKEEPERPATH=/usr/local/zookeeper</div><div>export GOROOT=/usr/local/go</div><div>export CODISPATH=/usr/local/codis</div><div>export PATH=$PATH:$GOROOT/bin:$ZOOKEEPERPATH/bin:$CODISPATH/bin</div><div><br></div><div><br></div><div>source /etc/profile</div><div><br></div><div><br></div><div>二，安装codis集群</div><div><br></div><div><br></div><div><br></div><div><br></div><div>1，安装go</div><div>设置环境变量</div><div>vim /etc/profile</div><div>export GOROOT=/usr/local/go</div><div>export CODISPATH=/usr/local/codis</div><div>export PATH=$PATH:$GOROOT/bin:$CODISPATH/bin</div><div>source /etc/profile</div><div>下载安装go</div><div>cd /usr/local/</div><div>wget http://golangtc.com/static/go/go1.3.3.linux-amd64.tar.gz</div><div>tar -zxvf go1.3.3.linux-amd64.tar.gz go version</div><div>go version go1.3.3 linux/amd64</div><div><br></div><div><br></div><div><br></div><div><br></div><div>2，安装依赖环境</div><div>yum groupinstall "Development Tools"</div><div><br></div><div><br></div><div>3，安装codis</div><div>yum install -y git</div><div>go get github.com/wandoulabs/codis #这个需要几分钟下载共30M文件</div><div>package github.com/wandoulabs/codis</div><div>imports github.com/wandoulabs/codis</div><div>imports github.com/wandoulabs/codis: no buildable Go source </div><div><br></div><div><br></div><div>files in /usr/local/codis/src/github.com/wandoulabs/codis</div><div>cd $GOPATH/src/github.com/wandoulabs/codis</div><div>[root@localhost codis]# pwd</div><div>/usr/local/codis/src/github.com/wandoulabs/codis</div><div>#执行编译测试脚本，编译go和reids。 </div><div>./bootstrap.sh #这个需要十几分钟共下载50M文件</div><div>make gotest</div><div># 将编译好后，把bin目录和一些脚本复制过去/usr/local/codis目录下：</div><div>mkdir -p /usr/local/codis/{logs,conf,scripts}</div><div>mkdir -p /data/codis_server/{logs,conf,data}</div><div>cp -rf bin /usr/local/codis/</div><div>cp sample/config.ini /usr/local/codis/conf/</div><div>cp sample/redis_conf/6381.conf /data/codis_server/conf/</div><div>cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/*.sh</div><div>/usr/local/codis/scripts/</div><div>cp -rf /usr/local/codis/src/github.com/wandoulabs/codis/sample/usage.md </div><div>/usr/local/codis/scripts/</div><div>cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.13/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.13</div><div>cp /usr/local/codis/src/github.com/wandoulabs/codis/extern/redis-2.8.21/src/redis-cli /usr/local/codis/bin/redis-cli-2.8.21</div><div>ln -s /usr/local/codis/bin/redis-cli-2.8.21 /usr/local/codis/bin/redis-cli</div><div><br></div><div><br></div><div>4. 配置codis_proxy ( zookeeper-1、zookeeper-2、zookeeper-3 机器上配置)</div><div><br></div><div><br></div><div>配置codis_proxy_1 ( zookeeper-1 机器上配置)</div><div>cd /usr/local/codis/conf</div><div>vim config.ini </div><div>zk=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181</div><div>product=jerrymin-codis</div><div>proxy_id=codis_proxy_1</div><div>net_timeout=50</div><div>dashboard_addr=192.168.88.112:18087</div><div>coordinator=zookeeper</div><div><br></div><div><br></div><div>配置codis_proxy_2 ( zookeeper-1 机器上配置)</div><div>cd /usr/local/codis/conf</div><div>vim config.ini </div><div>zk=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181</div><div>product=jerrymin-codis</div><div>proxy_id=codis_proxy_2</div><div>net_timeout=50</div><div>dashboard_addr=192.168.88.112:18087</div><div>coordinator=zookeeper</div><div><br></div><div><br></div><div>配置codis_proxy_3 ( zookeeper-1 机器上配置)</div><div>cd /usr/local/codis/conf</div><div>vim config.ini </div><div>zk=zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181</div><div>product=jerrymin-codis</div><div>proxy_id=codis_proxy_3</div><div>net_timeout=50</div><div>dashboard_addr=192.168.88.112:18087</div><div>coordinator=zookeeper</div><div><br></div><div><br></div><div><br></div><div><br></div><div>5. 修改配置文件，启动codis-server服务（codis-server1、codis-server2、codis-server3 机器上配置）</div><div>cd /data/codis_server/conf/</div><div>grep -Ev "^#|^$" 6379.conf.bak >6379.conf</div><div>vim 6379.conf</div><div>修改如下参数: (生产环境，参数适当进行调整)</div><div>daemonize yes</div><div>timeout 300</div><div>pidfile /var/run/redis_6379.pid</div><div>port 6379</div><div>logfile "/data/codis_server/logs/codis_6379.log"</div><div>save 900 1</div><div>save 300 10</div><div>save 60 10000</div><div>dbfilename 6379.rdb</div><div>dir /data/codis_server/data</div><div>appendfilename "6379_appendonly.aof"</div><div>appendfsync everysec</div><div><br></div><div><br></div><div>具体配置文件如下：</div><div>daemonize yes</div><div>pidfile /var/run/redis_6379.pid</div><div>port 6379</div><div>tcp-backlog 511</div><div>timeout 300</div><div>tcp-keepalive 0</div><div>loglevel notice</div><div>logfile "/data/codis_server/logs/redis_6379.log"</div><div>databases 16</div><div>stop-writes-on-bgsave-error yes</div><div>rdbcompression yes</div><div>rdbchecksum yes</div><div>dbfilename 6379.rdb</div><div>dir /data/codis_server/data</div><div>slave-serve-stale-data yes</div><div>repl-disable-tcp-nodelay no</div><div>slave-priority 100</div><div>maxclients 10000</div><div>maxmemory 3gb</div><div>maxmemory-policy allkeys-lru</div><div>appendonly yes</div><div>appendfilename "6379_appendonly.aof"</div><div>appendfsync everysec</div><div>no-appendfsync-on-rewrite no</div><div>auto-aof-rewrite-percentage 100</div><div>auto-aof-rewrite-min-size 64mb</div><div>lua-time-limit 5000</div><div>slowlog-log-slower-than 10000</div><div>slowlog-max-len 128</div><div>latency-monitor-threshold 0</div><div>notify-keyspace-events ""</div><div>hash-max-ziplist-entries 512</div><div>hash-max-ziplist-value 64</div><div>list-max-ziplist-entries 512</div><div>list-max-ziplist-value 64</div><div>set-max-intset-entries 512</div><div>zset-max-ziplist-entries 128</div><div>zset-max-ziplist-value 64</div><div>hll-sparse-max-bytes 3000</div><div>activerehashing yes</div><div>client-output-buffer-limit normal 0 0 0</div><div>client-output-buffer-limit slave 256mb 64mb 60</div><div>client-output-buffer-limit pubsub 32mb 8mb 60</div><div>hz 10</div><div>aof-rewrite-incremental-fsync yes</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>复制6380、6389、6390配置文件</div><div>cp 6379.conf 6380.conf</div><div>cp 6379.conf 6389.conf</div><div>cp 6379.conf 6390.conf</div><div>sed -i 's/6379/6380/g' 6380.conf</div><div>sed -i 's/6379/6389/g' 6380.conf</div><div>sed -i 's/6379/6390/g' 6380.conf</div><div><br></div><div><br></div><div>添加内核参数</div><div>echo "vm.overcommit_memory = 1" >> /etc/sysctl.conf</div><div>sysctl -p</div><div>内核参数说明如下： overcommit_memory文件指定了内核针对内存分配的策略</div><div><br></div><div><br></div><div>，其值可以是0、1、2。 0， 表示内核将检查是否有足够的可用内存供应用进程使用；如果有足够的可</div><div>用内存，内存申请允许；否则，内存申请失败，并把错误返回给应用进程。 </div><div>1， 表示内核允许分配所有的物理内存，而不管当前的内存状态如何。</div><div>2， 表示内核允许分配超过所有物理内存和交换空间总和的内存</div><div><br></div><div><br></div><div>启动codis-server服务</div><div>/usr/local/codis/bin/codis-server /data/codis_server/conf/6379.conf </div><div>/usr/local/codis/bin/codis-server /data/codis_server/conf/6380.conf </div><div>/usr/local/codis/bin/codis-server /data/codis_server/conf/6389.conf </div><div>/usr/local/codis/bin/codis-server /data/codis_server/conf/6390.conf </div><div>[root@codis-server1 ~]# /usr/local/codis/bin/codis-server</div><div>/data/codis_server/conf/6379.conf </div><div>[root@codis-server1 ~]# /usr/local/codis/bin/codis-server </div><div>/data/codis_server/conf/6380.conf </div><div>[root@codis-server1 ~]# /usr/local/codis/bin/codis-server </div><div>/data/codis_server/conf/6389.conf </div><div>[root@codis-server1 ~]# /usr/local/codis/bin/codis-server </div><div>/data/codis_server/conf/6390.conf </div><div>[root@codis-server1 ~]# ps aux |grep codis</div><div>root 7473 0.0 0.0 137388 9540 ? Ssl 09:48 0:00 </div><div>/usr/local/codis/bin/codis-server *:6379 </div><div>root 7478 0.0 0.0 137388 9524 ? Ssl 09:48 0:00 </div><div>/usr/local/codis/bin/codis-server *:6380 </div><div>root 7482 0.0 0.0 137388 9516 ? Ssl 09:48 0:00 </div><div>/usr/local/codis/bin/codis-server *:6389 </div><div>root 7486 0.0 0.0 137388 9524 ? Ssl 09:48 0:00 </div><div>/usr/local/codis/bin/codis-server *:6390 </div><div>root 7490 0.0 0.0 103252 856 pts/0 S+ 09:49 0:00 </div><div>grep --color=auto codis</div><div>[root@codis-server1 ~]# netstat -tulpn|grep codis</div><div>tcp 0 0 0.0.0.0:6379 0.0.0.0:* </div><div> LISTEN 7473/codis-server * </div><div>tcp 0 0 0.0.0.0:6380 0.0.0.0:* </div><div> LISTEN 7478/codis-server * </div><div>tcp 0 0 0.0.0.0:6389 0.0.0.0:* </div><div> LISTEN 7482/codis-server * </div><div>tcp 0 0 0.0.0.0:6390 0.0.0.0:* </div><div> LISTEN 7486/codis-server * </div><div>tcp 0 0 :::6379 :::* </div><div> LISTEN 7473/codis-server * </div><div>tcp 0 0 :::6380 :::* </div><div> LISTEN 7478/codis-server * </div><div>tcp 0 0 :::6389 :::* </div><div> LISTEN 7482/codis-server * </div><div>tcp 0 0 :::6390 :::* </div><div> LISTEN 7486/codis-server * </div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>6. 查看一下启动流程：</div><div>cat /usr/local/codis/scripts/usage.md</div><div>0. start zookeeper //启动zookeeper服务</div><div>1. change config items in config.ini //修改codis配置文件</div><div>2. ./start_dashboard.sh //启动 dashboard</div><div>3. ./start_redis.sh //启动redis实例</div><div>4. ./add_group.sh //添加redis组，一个redis组只能有一个master</div><div>5. ./initslot.sh //初始化槽</div><div>6. ./start_proxy.sh //启动proxy</div><div>7. ./set_proxy_online.sh //上线proxy项目</div><div>8. open browser to http://localhost:18087/admin //访问web</div><div>这只是一个参考，有些顺序不是必须的，但启动dashboard前，必须启动zookeeper服务，这是必须的，后面有很多操作，都可以在web页面完成，例如添加/删除组，添加/删除redis实例等</div><div><br></div><div><br></div><div><br></div><div><br></div><div>7. 修改脚本，启动 dashboard。( 只需在一台机器上启动即可。codis-ha2上启动 ，后续大部分操作都可以在面板上操作)</div><div>cat /usr/local/codis/scripts/start_dashboard.sh</div><div>#!/bin/sh</div><div>CODIS_HOME=/usr/local/codis </div><div>nohup $CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini -L </div><div>$CODIS_HOME/logs/dashboard.log dashboard --addr=:18087 --http-log=$CODIS_HOME/logs/requests.log &>/dev/null &</div><div><br></div><div><br></div><div>启动dashboard</div><div>[root@codis-ha2 scripts]# ls -lh start_dashboard.sh </div><div>-rwxr-xr-x 1 root root 218 Jun 24 22:04 start_dashboard.sh</div><div>[root@codis-ha2 scripts]# ./start_dashboard.sh </div><div>[root@codis-ha2 scripts]# ps aux |grep codis-config</div><div>root 2435 0.0 0.1 216444 11044 pts/1 Sl 10:06 0:00 </div><div>/usr/local/codis/bin/codis-config -c </div><div>/usr/local/codis/conf/config.ini -L </div><div>/usr/local/codis/logs/dashboard.log dashboard --addr=:18087 --http-</div><div>log=/usr/local/codis/logs/requests.log</div><div>root 2441 0.0 0.0 103252 840 pts/1 S+ 10:06 0:00 </div><div>grep --color=auto codis-config</div><div>[root@codis-ha2 scripts]# netstat -tulpn |grep codis</div><div>tcp 0 0 :::10086 :::* </div><div> LISTEN 2435/codis-config </div><div>tcp 0 0 :::18087 :::* </div><div> LISTEN 2435/codis-config </div><div><br></div><div><br></div><div>访问dashboard</div><div>http://192.168.88.112:18087/admin/</div><div><br></div><div><br></div><div><br></div><div><br></div><div>8. 添加redis组 </div><div>通过管理页面添加组ID，为组添加主从实例，一个组里只能有一个redis-master：</div><div>http://192.168.88.112:18087/admin/（最好用Firefox浏览器或者谷歌浏览器）</div><div>登录http://192.168.88.112:18087/admin/,添加2个组，组里面有2个实例，</div><div><br></div><div><br></div><div>一个主一个从，默认每个组里面第一个实例是主</div><div>group_1</div><div>192.168.88.106:6379 master</div><div>192.168.88.107:6380 slave</div><div>group_2</div><div>192.168.88.106:6389 master</div><div>192.168.88.108:6390 slave</div><div><br></div><div><br></div><div>group_3</div><div>192.168.88.107:6379 master</div><div>192.168.88.106:6380 slave</div><div>group_4</div><div>192.168.88.107:6389 master</div><div>192.168.88.108:6380 slave</div><div><br></div><div><br></div><div>group_5</div><div>192.168.88.108:6379 master</div><div>192.168.88.106:6390 slave</div><div>group_6</div><div>192.168.88.108:6389 master</div><div>192.168.88.107:6390 slave</div><div><br></div><div><br></div><div>9. 修改脚本，初始化槽 ( 在codis-ha2机器上配置，初始化solt是在group设置好之后 )</div><div>[root@codis-ha2 scripts]# cat initslot.sh </div><div>#!/bin/sh</div><div>CODIS_HOME=/usr/local/codis</div><div>echo "slots initializing..."</div><div>$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini slot </div><div><br></div><div><br></div><div>init -f</div><div>echo "done"</div><div>echo "set slot ranges to server groups..."</div><div>$CODIS_HOME/bin/codis-config -c config.ini slot range-set 0 170 1 online</div><div>$CODIS_HOME/bin/codis-config -c config.ini slot range-set 171 341 2 online</div><div>$CODIS_HOME/bin/codis-config -c config.ini slot range-set 342 512 3 online</div><div>$CODIS_HOME/bin/codis-config -c config.ini slot range-set 513 683 4 online</div><div>$CODIS_HOME/bin/codis-config -c config.ini slot range-set 684 853 5 online</div><div>$CODIS_HOME/bin/codis-config -c config.ini slot range-set 854 1023 6 online</div><div>echo "done"</div><div><br></div><div><br></div><div><br></div><div><br></div><div>10.测试一下redis-master和redis-slave是否正常同步数据了：</div><div>这个需要用redis-cli测试：</div><div>[root@codis-server1 ~]# redis-cli -h 192.168.88.106 -p 6379</div><div>192.168.88.106:6379> ping</div><div>PONG</div><div>192.168.88.106:6379> set name jerrymin</div><div>OK</div><div>192.168.88.106:6379> get name</div><div>"jerrymin"</div><div>192.168.88.106:6379> quit</div><div>[root@codis-server1 ~]# redis-cli -h 192.168.88.107 -p 6380</div><div>192.168.88.107:6380> get name</div><div>"jerrymin"</div><div>192.168.88.107:6380> set name foo</div><div>(error) READONLY You can't write against a read only slave.</div><div>192.168.88.107:6380> quit</div><div><br></div><div><br></div><div><br></div><div><br></div><div>11. 修改start_proxy.sh，启动codis-proxy服务 ( 在zookeeper-1、zookeeper-2、zookeeper-3上配置)</div><div>zookeeper-1上（其他上面就是codis_proxy_2、codis_proxy_3）</div><div>查看start_proxy脚本</div><div>[root@zookeeper-1 scripts]# cat start_proxy.sh </div><div>#!/bin/sh</div><div>CODIS_HOME=/usr/local/codis</div><div>echo "shut down codis_proxy_1..."</div><div>$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy </div><div>offline codis_proxy_1</div><div>echo "done"</div><div>echo "start new codis_proxy_1..."</div><div>nohup $CODIS_HOME/bin/codis-proxy --log-level info -c </div><div>$CODIS_HOME/conf/config.ini -L $CODIS_HOME/logs/codis_proxy_1.log </div><div>--cpu=4 --addr=0.0.0.0:19000 --http-addr=0.0.0.0:11000 &</div><div>echo "done"</div><div>echo "sleep 3s"</div><div>sleep 3</div><div>tail -n 30 $CODIS_HOME/logs/codis_proxy_1.log.0</div><div><br></div><div><br></div><div>查看proxy_online脚本</div><div>[root@zookeeper-1 scripts]# cat set_proxy_online.sh </div><div>#!/bin/sh</div><div>CODIS_HOME=/usr/local/codis</div><div>echo "set codis_proxy_1 online"</div><div>$CODIS_HOME/bin/codis-config -c $CODIS_HOME/conf/config.ini proxy </div><div>online codis_proxy_1</div><div>echo "done"</div><div><br></div><div><br></div><div>启动proxy</div><div>[root@zookeeper-1 scripts]# ./start_proxy.sh </div><div>shut down codis_proxy_1...</div><div>{</div><div> "msg": "OK",</div><div> "ret": 0</div><div>}</div><div>done</div><div>start new codis_proxy_1...</div><div>done</div><div>sleep 3s</div><div>nohup: appending output to `nohup.out'</div><div>2015/07/24 11:06:13 [INFO] set log level to %!s(log.LogLevel=7)</div><div>2015/07/24 11:06:13 [INFO] running on 0.0.0.0:19000</div><div>2015/07/24 11:06:13 [INFO] start proxy with config: &</div><div>{proxyId:codis_proxy_1 productName:jerrymin-codis zkAddr:zookeeper-</div><div>1:2181,zookeeper-2:2181,zookeeper-3:2181 fact: netTimeout:50 </div><div>proto:tcp provider:zookeeper}</div><div>2015/07/24 11:06:13 [INFO] proxy info = {Id:codis_proxy_1 </div><div>Addr:zookeeper-1:19000 LastEvent: LastEventTs:0 State:offline </div><div>Description: DebugVarAddr:zookeeper-1:11000 Pid:8097 StartAt:2015-</div><div>07-24 11:06:13.4791833 +0800 CST}</div><div>2015/07/24 11:06:13 [WARN] wait to be online: codis_proxy_1</div><div><br></div><div><br></div><div>上线proxy</div><div>[root@zookeeper-1 scripts]# ./set_proxy_online.sh </div><div>set codis_proxy_1 online</div><div>{</div><div> "msg": "OK",</div><div> "ret": 0</div><div>}</div><div>done</div><div><br></div><div><br></div><div><br></div><div><br></div><div>三，配置HA</div><div><br></div><div><br></div><div>1，codis-ha1和codis-ha2上操作</div><div>安装keeplived和ipvsadm</div><div>设置VIP 192.168.88.159</div><div>codis-ha1 为LVS主</div><div>codis-ha2 为LVS备</div><div>如下是备的配置，主的配置与此类似</div><div>[root@codis-ha2 haproxy]# cat /etc/keepalived/keepalived.conf</div><div>! Configuration File for keepalived</div><div>global_defs {</div><div> notification_email {</div><div> local@localhost</div><div> }</div><div> notification_email_from localhost@localhost</div><div> smtp_server 127.0.0.1</div><div> smtp_connect_timeout 30</div><div> router_id LVS_DEVEL_112</div><div>}</div><div>vrrp_script chk_haproxy_port {</div><div> script "/etc/keepalived/check_haproxy.sh"</div><div> interval 2</div><div> weight 2</div><div>}</div><div>vrrp_instance VI_1 {</div><div> state BACKUP</div><div> interface eth0</div><div> virtual_router_id 159</div><div> priority 50</div><div> advert_int 3</div><div> authentication {</div><div> auth_type PASS</div><div> auth_pass jerrymin</div><div> }</div><div> virtual_ipaddress {</div><div> 192.168.88.159</div><div> }</div><div> track_script {</div><div> chk_haproxy_port</div><div> }</div><div>}</div><div>检测脚本，保证keepalived切换成主时haproxy在工作</div><div>[root@codis-ha2 keepalived]# cat check_haproxy.sh </div><div>#!/bin/bash</div><div>A=`ps -C haproxy --no-header |wc -l`</div><div>if [ $A -eq 0 ];then</div><div>/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg</div><div>sleep 3</div><div>if [ `ps -C haproxy --no-header |wc -l` -eq 0 ];then</div><div>/etc/init.d/keepalived stop</div><div>fi</div><div>fi</div><div><br></div><div><br></div><div><br></div><div><br></div><div>2，安装配置haproxy</div><div>[root@codis-ha1 ~]#tar zxvf haproxy-1.5.11.tar.gz </div><div>[root@codis-ha1 ~]#cd haproxy-1.5.11</div><div>[root@codis-ha1 haproxy-1.5.11]#make TARGET=linux26 PREFIX=/usr/local/haproxy </div><div>[root@codis-ha1 haproxy-1.5.11]#make install PREFIX=/usr/local/haproxy</div><div><br></div><div><br></div><div>3，配置haproxy.cfg</div><div>global </div><div> maxconn 40000 </div><div>daemon</div><div> user root</div><div> group root </div><div> nbproc 4 </div><div> log 127.0.0.1 local3 </div><div> spread-checks 2 </div><div>defaults </div><div> timeout server 3s </div><div> timeout connect 3s </div><div> timeout client 60s </div><div> timeout http-request 3s </div><div> timeout queue 3s</div><div>frontend codis-proxy</div><div> bind :19000 </div><div> default_backend codis-proxy-19000</div><div><br></div><div><br></div><div>frontend web_haproxy_status </div><div> bind :8080 </div><div> default_backend web_status</div><div><br></div><div><br></div><div><br></div><div><br></div><div>backend codis-proxy-19000</div><div> mode tcp </div><div> option tcpka </div><div> balance roundrobin </div><div> server msvr1 192.168.88.113:19000 check inter 1s rise 5 fall 1 </div><div> server msvr2 192.168.88.114:19000 check inter 1s rise 5 fall 1 </div><div> server msvr3 192.168.88.115:19000 check inter 1s rise 5 fall 1 </div><div> timeout server 9s </div><div><br></div><div><br></div><div>backend web_status </div><div> mode http </div><div> stats enable </div><div> stats refresh 5s </div><div> stats uri /status </div><div> stats realm Haproxy\ statistics </div><div> stats auth jerrymin:jerrymin@2015</div><div><br></div><div><br></div><div>4.启动服务</div><div>启动服务：</div><div>[root@codis-ha1 haproxy]# /usr/local/haproxy/sbin/haproxy -f </div><div><br></div><div><br></div><div>/usr/local/haproxy/haproxy.cfg</div><div>[root@codis-ha1 haproxy]# ps aux |grep haproxy</div><div>root 3147 0.0 0.0 18668 3020 ? Ss 14:42 0:00 </div><div>/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg</div><div>root 3148 0.0 0.0 18668 2888 ? Ss 14:42 0:00 </div><div>/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg</div><div>root 3149 0.0 0.0 18668 2888 ? Ss 14:42 0:00 </div><div>/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg</div><div>root 3150 0.0 0.0 18668 3020 ? Ss 14:42 0:00 </div><div>/usr/local/haproxy/sbin/haproxy -f /usr/local/haproxy/haproxy.cfg</div><div>[root@codis-ha1 haproxy]# ipvsadm -ln</div><div>IP Virtual Server version 1.2.1 (size=4096)</div><div>Prot LocalAddress:Port Scheduler Flags</div><div> -> RemoteAddress:Port Forward Weight ActiveConn </div><div><br></div><div><br></div><div>InActConn</div><div>TCP 192.168.88.159:19000 wrr</div><div> -> 192.168.88.111:19000 Local 1 0 0 </div><div><br></div><div><br></div><div><br></div></div>
|
||
</div>
|
||
|
||
<!-- 该js供其它处理 -->
|
||
<script src="../leanote-html.js"></script>
|
||
</body>
|
||
</html> |