发表于   |   分类于   |     |   阅读次数

openresty模块

1
https://github.com/openresty/lua-nginx-module#readme

其中redis模块叫做lua-resty-redis,可以在上述页面进行搜索

1.一个完整的demo

使用起来不困难，也不必深究：redis的API、luajit的基本语法。

需要注意的是在openresty中已经有了类库，也就是下面require “resty.redis”

阅读全文 »

发表于   |   分类于   |     |   阅读次数

提示，有关ngx_lua的语法可以到如下网站搜索

1
https://www.nginx.com/resources/wiki/modules/lua/?highlight=luas

1.一个完整的程序

1
2
3
4
5
6
7
local args = ngx.req.get_uri_args()
local salt = args.salt
if not salt then
    ngx.exit(ngx.HTTP_BAD_REQUEST)
end
local str = ngx.md5(ngx.time() .. salt)
ngx.say(str)
阅读全文 »

发表于   |   分类于   |     |   阅读次数

一、相关资源

1.openresty官网

2.openresty视频教程

3.ngx_lua文档

4.openresty最佳实践

5.相关类库说明

6.【开源访谈】OpenResty 作者章亦春访谈实录

附图一张：

二、快速安装

1. 下载

从https://openresty.org/cn/download.html下载最新的版本，例如·

1
2
3
4
cd /opt/soft/
wget https://openresty.org/download/openresty-1.9.15.1.tar.gz '--no-check-certificate'
tar -xvf openresty-1.9.15.1.tar.gz
ln -s openresty-1.9.15.1 openresty

2. 准备

(1) yum相关的依赖

1
yum install readline-devel pcre-devel openssl-devel gcc

(2) 编译安装

1
2
3
4
5
6
7
8
9
10
cd /opt/soft/openresty
./configure \
   --prefix=/opt/openresty
   --without-http_redis2_module \
   --with-cc-opt="-I/usr/local/opt/openssl/include/ -I/usr/local/opt/pcre/include/" \
   --with-ld-opt="-L/usr/local/opt/openssl/lib/ -L/usr/local/opt/pcre/lib/" \
   -j8
   --with-luajit
make
make install

3.启动

启动方法和nginx是一样的:

1
/opt/soft/openresty/nginx/sbin/nginx -c /opt/soft/openresty/nginx/conf/nginx.conf

4.验证

1
curl 127.0.0.1

结果:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

三、快速使用

1.在nginx.conf中添加ngx_lua代码

1
2
3
4
5
location /hello {
    content_by_lua '
        ngx.say("<p>hello openresty!!</p>")
    ';
}

2.验证并重启nginx

1
2
/opt/soft/openresty/nginx/sbin/nginx -t /opt/soft/openresty/nginx/conf/nginx.conf
/opt/soft/openresty/nginx/sbin/nginx -s reload

3.访问验证

1
2
[@bx-14-182 conf]# curl 127.0.0.1/hello
<p>hello openresty!!</p>

提示，有关ngx_lua的语法可以到如下网站搜索

1
https://www.nginx.com/resources/wiki/modules/lua/?highlight=luas

4.lua脚本独立

(1) lua独立

1
2
3
location /hello {
    content_by_lua_file /usr/local/openresty/script/hello.lua;
}

其中content_by_lua_file是通过上面的网站查到的，这个网址应该经常用到。

(2) 脚本

1
2
[@bx-14-182 script]# cat /usr/local/openresty/script/hello.lua
ngx.say("<p>hello openresty content_by_lua_file.</p>")

(3) 缓存：
先修改脚本内容发现，返回内容不变，需要添加缓存选项(如果是生产可能会影响性能)
默认开启, 有多个使用范围

1
2
3
syntax: lua_code_cache on | off
default: lua_code_cache on
context: http, server, location, location if

例如：

1
2
[@bx-14-182 script]# ../nginx/sbin/nginx -t
nginx: [alert] lua_code_cache is off; this will hurt performance in /usr/local/openresty/nginx/conf/nginx.conf:48

发表于   |   分类于   |     |   阅读次数

主要引入如下模块

1
2
import redis
import rediscluster

一.删除指定模式的key

1.单个节点

使用scan + pipeline + del(每次扫描100个) 

1
2
3
4
5
6
7
8
9
10
11
12
13
def delInstancePatternKeys(host, port, pattern, password=None):
    client = redis.StrictRedis(host=host, port=port, password=password)
    pipeline = client.pipeline(transaction=False)
    cursor = 0
    while True:
        currentCursor, keys = client.scan(cursor=cursor, match=pattern, count=100)
        if len(keys) > 0:
            for key in keys:
                pipeline.delete(key)
            pipeline.execute()
        cursor = currentCursor
        if cursor == 0:
            break

例如删除127.0.0.1:6380中以video开头的key，可以执行如下

1
delInstancePatternKeys("127.0.0.1",6380,"video*")
阅读全文 »

发表于   |   分类于   |     |   阅读次数

一.获取redis-py-cluster

redis-py并没有支持Redis Cluster。社区中有人开发了redis-py-cluster(非同一个作者)。两种方法获取。

第一：使用pip进行安装(推荐)

1
pip install redis-py-cluster

第三：使用源码安装(以1.3.4版本为例子，可以在release列表选取需要)

1
2
3
4
wget https://github.com/Grokzen/redis-py-cluster/releases/download/1.3.4/redis-py-cluster-1.3.4.tar.gz
unzip redis-py-cluster-1.3.4.tar.gz
cd redis-py-cluster-1.3.4
python setup.py install
阅读全文 »

发表于   |   分类于   |     |   阅读次数

RedisCluster使用P2P实现分布式，对于每个Redis节点来说，它既存储数据又要管理分布式状态，Redis节点之间使用Gossip协议进行通信，那么对于每个节点来说，它又是客户端又是服务端。所以了解节点通信的代码很重要。

1
所有代码均在cluster.h和cluster.c中。

1.Redis Cluster节点在初始化时，会创建AcceptTcpHandler(clusterAcceptHandler)。

可以看出RedisCluster节点内部使用port + 10000进行通信。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
void clusterInit(void) {
    /* Port sanity check II
     * The other handshake port check is triggered too late to stop
     * us from trying to use a too-high cluster port number. */
    if (server.port > (65535-REDIS_CLUSTER_PORT_INCR)) {
        redisLog(REDIS_WARNING, "Redis port number too high. "
                   "Cluster communication port is 10,000 port "
                   "numbers higher than your Redis port. "
                   "Your Redis port number must be "
                   "lower than 55535.");
        exit(1);
    }
    if (listenToPort(server.port+REDIS_CLUSTER_PORT_INCR,
        server.cfd,&server.cfd_count) == REDIS_ERR)
    {
        exit(1);
    } else {
        int j;
        for (j = 0; j < server.cfd_count; j++) {
            if (aeCreateFileEvent(server.el, server.cfd[j], AE_READABLE,
                clusterAcceptHandler, NULL) == AE_ERR)
                    redisPanic("Unrecoverable error creating Redis Cluster "
                                "file event.");
        }
    }
    ...忽略...
}

clusterInit会在redis.c中初始化redis实例时候使用

1
2
3
4
5
6
void initServer(void) {
    ...忽略...
    if (server.cluster_enabled) clusterInit();
    ...忽略...
}

2. clusterAcceptHandler定义，进入clusterReadHandler逻辑。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
void clusterAcceptHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
    int cport, cfd;
    int max = MAX_CLUSTER_ACCEPTS_PER_CALL;
    char cip[REDIS_IP_STR_LEN];
    clusterLink *link;
    REDIS_NOTUSED(el);
    REDIS_NOTUSED(mask);
    REDIS_NOTUSED(privdata);
    /* If the server is starting up, don't accept cluster connections:
     * UPDATE messages may interact with the database content. */
    if (server.masterhost == NULL && server.loading) return;
    while(max--) {
        cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport);
        if (cfd == ANET_ERR) {
            if (errno != EWOULDBLOCK)
                redisLog(REDIS_VERBOSE,
                    "Error accepting cluster node: %s", server.neterr);
            return;
        }
        anetNonBlock(NULL,cfd);
        anetEnableTcpNoDelay(NULL,cfd);
        /* Use non-blocking I/O for cluster messages. */
        redisLog(REDIS_VERBOSE,"Accepted cluster node %s:%d", cip, cport);
        /* Create a link object we use to handle the connection.
         * It gets passed to the readable handler when data is available.
         * Initiallly the link->node pointer is set to NULL as we don't know
         * which node is, but the right node is references once we know the
         * node identity. */
        link = createClusterLink(NULL);
        link->fd = cfd;
        aeCreateFileEvent(server.el,cfd,AE_READABLE,clusterReadHandler,link);
    }
}

3. clusterReadHandler：解析请求，其中clusterProcessPacket是处理各种消息的逻辑(例如ping,pong,meet等)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
/* Read data. Try to read the first field of the header first to check the
 * full length of the packet. When a whole packet is in memory this function
 * will call the function to process the packet. And so forth. */
void clusterReadHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
    char buf[sizeof(clusterMsg)];
    ssize_t nread;
    clusterMsg *hdr;
    clusterLink *link = (clusterLink*) privdata;
    unsigned int readlen, rcvbuflen;
    REDIS_NOTUSED(el);
    REDIS_NOTUSED(mask);
    while(1) { /* Read as long as there is data to read. */
        rcvbuflen = sdslen(link->rcvbuf);
        if (rcvbuflen < 8) {
            /* First, obtain the first 8 bytes to get the full message
             * length. */
            readlen = 8 - rcvbuflen;
        } else {
            /* Finally read the full message. */
            hdr = (clusterMsg*) link->rcvbuf;
            if (rcvbuflen == 8) {
                /* Perform some sanity check on the message signature
                 * and length. */
                if (memcmp(hdr->sig,"RCmb",4) != 0 ||
                    ntohl(hdr->totlen) < CLUSTERMSG_MIN_LEN)
                {
                    redisLog(REDIS_WARNING,
                        "Bad message length or signature received "
                        "from Cluster bus.");
                    handleLinkIOError(link);
                    return;
                }
            }
            readlen = ntohl(hdr->totlen) - rcvbuflen;
            if (readlen > sizeof(buf)) readlen = sizeof(buf);
        }
        nread = read(fd,buf,readlen);
        if (nread == -1 && errno == EAGAIN) return; /* No more data ready. */
        if (nread <= 0) {
            /* I/O error... */
            redisLog(REDIS_DEBUG,"I/O error reading from node link: %s",
                (nread == 0) ? "connection closed" : strerror(errno));
            handleLinkIOError(link);
            return;
        } else {
            /* Read data and recast the pointer to the new buffer. */
            link->rcvbuf = sdscatlen(link->rcvbuf,buf,nread);
            hdr = (clusterMsg*) link->rcvbuf;
            rcvbuflen += nread;
        }
        /* Total length obtained? Process this packet. */
        if (rcvbuflen >= 8 && rcvbuflen == ntohl(hdr->totlen)) {
            if (clusterProcessPacket(link)) {
                sdsfree(link->rcvbuf);
                link->rcvbuf = sdsempty();
            } else {
                return; /* Link no longer valid. */
            }
        }
    }
}

发表于   |   分类于   |     |   阅读次数

1.获取指定rowkey对应region和regionserver

(1).获取一个表的RegionLocator

1
2
3
TableName tableName = TableName.valueOf(Bytes.toBytes("myTableName"))
RegionLocator locator = connection.getRegionLocator(tableName);

(2).使用RegionLocator定位rowkey对应的Region和Region Server

1
2
byte[] rowKey = xx
HRegionLocation location = locator.getRegionLocation(rowKey);

其中HRegionLocation包含了Region、RegionServer等信息。

1
2
public HRegionInfo getRegionInfo();
public ServerName getServerName();

(3).RegionLocator的其他功能

1
2
3
4
5
public List<HRegionLocation> getAllRegionLocations();
public byte [][] getStartKeys() throws IOException;
public byte[][] getEndKeys() throws IOException;
public Pair<byte[][],byte[][]> getStartEndKeys() throws IOException;
TableName getName();

2.获取regionServer下所有的Region

1
2
ServerName serverName = ServerName.valueOf(“xxxx”);
List<HRegionInfo> hRegionInfoList = connection.getAdmin().getOnlineRegions(serverName);
for (HRegionInfo hRegionInfo : hRegionInfoList){
    System.out.println(Bytes.toString(hRegionInfo.getRegionName()));
}

发表于   |   分类于   |     |   阅读次数

1.Redis实例启动的时候，创建TcpHandler来处理客户端的请求

(redis.c)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
void initServer(void) {
...忽略...
    /* Create an event handler for accepting new connections in TCP and Unix
     * domain sockets. */
    for (j = 0; j < server.ipfd_count; j++) {
        if (aeCreateFileEvent(server.el, server.ipfd[j], AE_READABLE,
            acceptTcpHandler,NULL) == AE_ERR)
            {
                redisPanic(
                    "Unrecoverable error creating server.ipfd file event.");
            }
    }
...忽略...
}
阅读全文 »

发表于   |   分类于   |     |   阅读次数

一、Makefile

例如项目有很多源文件，彼此关联。例如下面main.c用到了myutil.c。

1. myuti.h

1
2
int maxNum(int a, int b);
int minNum(int a, int b);

2. myutil.c

1
2
3
4
5
6
7
8
9
#include "myutil.h"
int max(int a, int b) {
    return a > b ? a : b;
}
int min(int a, int b) {
    return a < b ? a : b;
}
阅读全文 »