目錄
01 lars 系統架構回顧
02 lars-lbAgentV0.4-route_lb處理report業務流程
03 lars-lbAgentV0.4-負責均衡判斷參數配置
04 lars-lbAgentV0.4-負載均衡idle節點的失敗率判斷
05 lars-lbAgentV0.4-負載均衡overload節點的成功率判斷
06 lars-lbAgentV0.4-負載均衡上報提交模塊
07 lars-lbAgentV0.4-數據問題修改-上報業務測試
08 lars-lbAgentV0.4-模擬器測試工具
09 lars-lbAgentV0.5-idle連續成功-overload連續失敗的定期窗口重置機制
10 lars-lbAgentV0.6-loadbalance主動更新主機信息功能
11 昨日回顧
12 Lars-LbAgentV0.7-GetRouteAPI實現
13 Lars-LbAgentV0.7-Udp-server-獲取路由服務業務注冊
14 Lars-LbAgentV0.7-Udp-server-API和udpserver的測試
01 lars 系統架構回顧
**啟動 Lars Reporter**
```bash
$ ./run_lars reporter
?
????????▄▄???????????????????????????????????????????????
????????██???????????????????????????????????????????????
????????██?????????▄█████▄???██▄████??▄▄█████▄???????????
????????██?????????? ▄▄▄██???██???????██▄▄▄▄ ????????????
????????██????????▄██???██???██????????????██▄???????????
????????██▄▄▄▄▄▄??██▄▄▄███???██???????█▄▄▄▄▄██???????????
??????????????????????? ?????????????????????????????????
????Load balance And Remote service schedule System?
?????????????????????????????????????????????????????????
????????_____???????????????????????_???????????????????????
???????|??__ \?????????????????????| |
???????| |__) |___ _ __???___??_ __| |_ ___ _ __
???????|??_??// _ \ '_ \ / _ \| '__| __/ _ \ '__|
???????| | \ \??__/ |_) | (_) | |??| ||??__/ |
???????|_|??\_\___| .__/ \___/|_|???\__\___|_|
??????????????????| |
??????????????????|_|
????????????ITCAST(https://www.itcast.cn)
?????????------------------------------------?
create 0 thread
create 1 thread
create 2 thread
create 3 thread
create 4 thread
add msg cb msgid = 3
```
**啟動 Lars dns**
```bash
$ ./run_lars dns
?
????????▄▄???????????????????????????????????????????????
????????██???????????????????????????????????????????????
????????██?????????▄█████▄???██▄████??▄▄█████▄???????????
????????██?????????? ▄▄▄██???██???????██▄▄▄▄ ????????????
????????██????????▄██???██???██????????????██▄???????????
????????██▄▄▄▄▄▄??██▄▄▄███???██???????█▄▄▄▄▄██???????????
??????????????????????? ?????????????????????????????????
????Load balance And Remote service schedule System?
?????????????????????????????????????????????????????????
????????????????_____?
???????????????|??__ \?
???????????????| |??| |_ __??___?
???????????????| |??| | '_ \/ __|
???????????????| |__| | | | \__ \?
???????????????|_____/|_| |_|___/?
????????????ITCAST(https://www.itcast.cn)
?????????------------------------------------?
create 0 thread
create 1 thread
create 2 thread
create 3 thread
create 4 thread
add msg cb msgid = 1
lars dns service ....
now route version is 1574674421
```
**啟動 Lars Lb Agent**
```c
$ ./run_lars lbagent
????????▄▄???????????????????????????????????????????????
????????██???????????????????????????????????????????????
????????██?????????▄█████▄???██▄████??▄▄█████▄???????????
????????██?????????? ▄▄▄██???██???????██▄▄▄▄ ????????????
????????██????????▄██???██???██????????????██▄???????????
????????██▄▄▄▄▄▄??██▄▄▄███???██???????█▄▄▄▄▄██???????????
??????????????????????? ?????????????????????????????????
????Load balance And Remote service schedule System?
?????????????????????????????????????????????????????????
??????_??????_????????????????????????????????_??
?????| |????| |?????????/\???????????????????| |?
?????| |????| |__??????/??\???__ _??___ _ __ | |_?
?????| |????| '_ \????/ /\ \ / _` |/ _ \ '_ \| __|?
?????| |____| |_) |??/ ____ \ (_| |??__/ | | | |_?
?????|______|_.__/??/_/????\_\__, |\___|_| |_|\__|?
??????????????????????????????__/ |?
?????????????????????????????|___/??
????????????ITCAST(https://www.itcast.cn)
?????????------------------------------------?
```
02 lars-lbAgentV0.4-route_lb處理report業務流程

LB Agent擁有5個線程,一個LB算法:
-?UDP Server服務,并運行LB算法,對業務提供節點獲取和節點調用結果上報服務;為了增大系統吞吐量,使用3個UDP Server服務互相獨立運行LB算法:`modid+cmdid % 3 = i`的那些模塊的服務與調度,由第`i+1`個UDP Server線程負責
-?Dns Service Client:是dnsserver的客戶端線程,負責根據需要,向dnsserver獲取一個模塊的節點集合(或稱為獲取路由);UDP Server會按需向此線程的MQ寫入獲取路由請求,DSS Client將MQ到來的請求轉發到dnsserver,之后將dnsserver返回的路由信息更新到對應的UDP Server線程維護的路由信息中
-?Report Service Client:是reporter的客戶端線程,負責將每個模塊下所有節點在一段時間內的調用結果、過載情況上報到reporter Service端,便于觀察情況、做報警;本身消費MQ數據,UDP Server會按需向MQ寫入上報狀態請求
03 lars-lbAgentV0.4-負責均衡判斷參數配置
## 2) 構建Lars-Agent項目????
### 2.1 構建目錄結構
依次創建如下目錄:
> Lars/lars_loadbalance_agent
```bash
lars_loadbalance_agent/
├── bin/
├── conf/
│?? └── lars_lb_agent.conf
├── include/
├── Makefile
└── src/
```
其中Makefile如下:
```makefile
TARGET= bin/lars_lb_agent
CXX=g++
CFLAGS=-g -O2 -Wall -Wno-deprecated
BASE=../base
BASE_H=$(BASE)/include
PROTO = $(BASE)/proto
PROTO_H = $(BASE)/proto
LARS_REACTOR=../lars_reactor
LARS_REACTOR_H =$(LARS_REACTOR)/include
LARS_REACTOR_LIB=$(LARS_REACTOR)/lib??-llreactor
MYSQL=$(BASE)/mysql-connector-c
MYSQL_H=$(MYSQL)/include
MYSQL_LIB=$(MYSQL)/lib/libmysqlclient.a
OTHER_LIB = -lpthread -ldl -lprotobuf
SRC= ./src
INC= -I./include -I$(BASE_H) -I$(LARS_REACTOR_H) -I$(MYSQL_H) -I$(PROTO_H)
LIB= $(MYSQL_LIB) -L$(LARS_REACTOR_LIB) $(OTHER_LIB)?
OBJS = $(addsuffix .o, $(basename $(wildcard $(SRC)/*.cpp)))
OBJS += $(PROTO)/lars.pb.o
$(TARGET): $(OBJS)
????????mkdir -p bin
????????$(CXX) $(CFLAGS) -o $(TARGET) $(OBJS) $(INC) $(LIB)
%.o: %.cpp
????????$(CXX) $(CFLAGS) -c -o $@ $< $(INC)?
.PHONY: clean
clean:
????????-rm -f src/*.o??$(PROTO)/lars.pb.o $(TARGET)
```
?????????實際上reporter、dns相似。
> conf/lars_lb_agent.conf
```ini
[reporter]
ip = 127.0.0.1
port = 7779
[dnsserver]
ip = 127.0.0.1
port = 7778
```
?????????目前的基本配置文件,因為loadbalance_agent是充當reporter和dnsserver的客戶端,所以需要知道對應的reporter和dnsserver的ip和port信息。
04 lars-lbAgentV0.4-負載均衡idle節點的失敗率判斷
### 2.2 主模塊業務搭建

?????????首先我們要在主線程中,啟動3個UDP Server線程,這個是提供業務層/API層的服務。然后分別啟動report_client線程,用來和reporter Service進行通信,將請求上報信息發送給Reporter Service。 然后再啟動dns_client線程,用來和dns service通信。
> lars_loadbalance_agent/include/main_server.h?
```c
#pragma once
#include "lars_reactor.h"
#include "lars.pb.h"
//與report_client通信的thread_queue消息隊列
extern thread_queue<lars::ReportStatusRequest>* report_queue;
//與dns_client通信的thread_queue消息隊列
extern thread_queue<lars::GetRouteRequest>* dns_queue;
// 啟動udp server服務,用來接收業務層(調用者/使用者)的消息
void start_UDP_servers(void);
// 啟動lars_reporter client 線程
void start_report_client(void);
// 啟動lars_dns client 線程
void start_dns_client(void);
```
05 lars-lbAgentV0.4-負載均衡overload節點的成功率判斷
> lars_loadbalance_agent/src/main_server.cpp?
```c
#include "main_server.h"
#include "lars.pb.h"
//與report_client通信的thread_queue消息隊列
thread_queue<lars::ReportStatusRequest>* report_queue = NULL;
//與dns_client通信的thread_queue消息隊列
thread_queue<lars::GetRouteRequest>* dns_queue = NULL;
int main(int argc, char **argv)
{
????//1 加載配置文件
????
????//2 啟動udp server服務,用來接收業務層(調用者/使用者)的消息
????start_UDP_servers();
????
????//3 啟動lars_reporter client 線程
????report_queue = new thread_queue<lars::ReportStatusRequest>();
????if (report_queue == NULL) {
????????fprintf(stderr, "create report queue error!\n");
????????exit(1);
????}
????start_report_client();
????
????//4 啟動lars_dns client 線程
????dns_queue = new thread_queue<lars::GetRouteRequest>();
????if (dns_queue == NULL) {
????????fprintf(stderr, "create dns queue error!\n");
????????exit(1);
????}
????start_dns_client();
????
????std::cout <<"done!" <<std::endl;
????while (1) {
????????sleep(10);
????}
????return 0;
}
```
06 lars-lbAgentV0.4-負載均衡上報提交模塊
這里我們分別在main()中 ,開啟以上線程。
?????????其中`report_client`線程需要攜帶`thread_queue<lars::ReportStatusRequest>`消息隊列通道。`agent`負責將上報請求消息`lars::ReportStatusRequest`通過thread_queue發送給reporter service。
?????????其中`dns_client`線程需要攜帶`thread_queue<lars::GetRouteRequest>`。`agent`負責將請求modid/cmdid的route消息`lars::GetRouteRequest`通過thread_queue發送給dns service。
3個udp server的線程開辟實現如下:
> lars_loadbalance_agent/src/agent_udp_server.cpp
```c
#include "lars_reactor.h"
#include "main_server.h"
void * agent_server_main(void * args)
{
????int *index = (int*)args;
????short port = *index + 8888;
????event_loop loop;
????udp_server server(&loop, "0.0.0.0", port);
????//TODO 給server注冊消息分發路由業務
????printf("agent UDP server :port %d is started...\n", port);
????loop.event_process();
????return NULL;
}
07 lars-lbAgentV0.4-數據問題修改-上報業務測試
void start_UDP_servers(void)
{
????for (int i = 0; i < 3; i ++) {
????????pthread_t tid;
????????
????????int ret = pthread_create(&tid, NULL, agent_server_main, &i);
????????if (ret == -1) {
????????????perror("pthread_create");
????????????exit(1);
????????}
????????pthread_detach(tid);
????}
}
```
reporter thread創建實現如下:
> lars_loadbalance_agent/src/reporter_client.cpp
```c
#include "lars_reactor.h"
#include "main_server.h"
#include <pthread.h>
void *report_client_thread(void* args)
{
????printf("report client thread start\n");
#if 0
????event_loop loop;
????//1 加載配置文件得到repoter ip + port
????std::string ip = config_file::instance()->GetString("reporter", "ip", "");
????short port = config_file::instance()->GetNumber("reporter", "port", 0);
????//2 創建客戶端
????tcp_client client(&loop, ip.c_str(), port, "reporter client");
????//3 將 thread_queue消息回調事件,綁定到loop中
????report_queue->set_loop(&loop);
????report_queue->set_callback()
????//4 啟動事件監聽
????loop.event_process();
#endif
????return NULL;
}
08 lars-lbAgentV0.4-模擬器測試工具
void start_report_client()
{
????//開辟一個線程
????pthread_t tid;
????//啟動線程業務函數
????int ret = pthread_create(&tid, NULL, report_client_thread, NULL);
????if (ret == -1) {
????????perror("pthread_create");
????????exit(1);
????}
????//設置分離模式
????pthread_detach(tid);
}
```
dns thread創建實現如下:
> lars_loadbalance_agent/src/dns_client.cpp
```c
#include "lars_reactor.h"
#include "main_server.h"
#include <pthread.h>
void *dns_client_thread(void* args)
{
????printf("dns client thread start\n");
????return NULL;
}
void start_dns_client()
{
????//開辟一個線程
????pthread_t tid;
????//啟動線程業務函數
????int ret = pthread_create(&tid, NULL, dns_client_thread, NULL);
????if (ret == -1) {
????????perror("pthread_create");
????????exit(1);
????}
????//設置分離模式
????pthread_detach(tid);
}
```
09 lars-lbAgentV0.5-idle連續成功-overload連續失敗的定期窗
口重置機制
### 2.3 測試lb_agentV0.1開發
編譯,然后我們簡單啟動一下`./bin/lars_lb_agent`
```bash
$ ./bin/lars_lb_agent?
dns client thread start
report client thread start
done!
msg_router init...
server on 0.0.0.0:8888 is running...
agent UDP server :port 8888 is started...
msg_router init...
server on 0.0.0.0:8888 is running...
agent UDP server :port 8888 is started...
msg_router init...
server on 0.0.0.0:8888 is running...
agent UDP server :port 8888 is started...
...
```
?
10 lars-lbAgentV0.6-loadbalance主動更新主機信息功能
## 3) Report Client設計與實現
?????????report client主要是實現thread_queue的回調業務,udp server會定期的上傳上報數據到reporter,那么請求對于report client就是透傳給reporter serivce即可。
> lars_loadbalance_agent/src/reporter_client.cpp
```c
#include "lars_reactor.h"
#include "main_server.h"
#include <string>
#include <pthread.h>
//typedef void io_callback(event_loop *loop, int fd, void *args);
//只要thread_queue有數據,loop就會觸發此回調函數來處理業務
void new_report_request(event_loop *loop, int fd, void *args)
{
????tcp_client *client = (tcp_client*)args;
????//1. 將請求數據從thread_queue中取出,
????std::queue<lars::ReportStatusRequest>??msgs;
????//2. 將數據放在queue隊列中
????report_queue->recv(msgs);
????
????//3. 遍歷隊列,通過client依次將每個msg發送給reporter service
????while (!msgs.empty()) {
????????lars::ReportStatusRequest req = msgs.front();
????????msgs.pop();
????????std::string requestString;
????????req.SerializeToString(&requestString);
????????//client 發送數據
????????client->send_message(requestString.c_str(), requestString.size(), lars::ID_ReportStatusRequest);
????}
}
11 昨日回顧
void *report_client_thread(void* args)
{
????printf("report client thread start\n");
????
????event_loop loop;
????//1 加載配置文件得到repoter ip + port
????std::string ip = config_file::instance()->GetString("reporter", "ip", "");
????short port = config_file::instance()->GetNumber("reporter", "port", 0);
????//2 創建客戶端
????tcp_client client(&loop, ip.c_str(), port, "reporter client");
????//3 將 thread_queue消息回調事件,綁定到loop中
????report_queue->set_loop(&loop);
????report_queue->set_callback(new_report_request, &client);
????//4 啟動事件監聽
????loop.event_process();
????return NULL;
}
12 Lars-LbAgentV0.7-GetRouteAPI實現
void start_report_client()
{
????//開辟一個線程
????pthread_t tid;
????//啟動線程業務函數
????int ret = pthread_create(&tid, NULL, report_client_thread, NULL);
????if (ret == -1) {
????????perror("pthread_create");
????????exit(1);
????}
????//設置分離模式
????pthread_detach(tid);
}
```
## 4) Dns Client設計與實現
?????????dns client 和report client的業務十分相似,只是針對的協議不同了。dns client的thread_queue 回調業務主要是透傳`lars::GetRouteRequest`數據包。
> lars_loadbalance_agent/src/dns_client.cpp
```c
#include "lars_reactor.h"
#include "main_server.h"
#include <pthread.h>
//typedef void io_callback(event_loop *loop, int fd, void *args);
//只要thread_queue有數據,loop就會觸發此回調函數來處理業務
void new_dns_request(event_loop *loop, int fd, void *args)
{
????tcp_client *client = (tcp_client*)args;
????//1. 將請求數據從thread_queue中取出,
????std::queue<lars::GetRouteRequest>??msgs;
????//2. 將數據放在queue隊列中
????dns_queue->recv(msgs);
????
????//3. 遍歷隊列,通過client依次將每個msg發送給reporter service
????while (!msgs.empty()) {
????????lars::GetRouteRequest req = msgs.front();
????????msgs.pop();
????????std::string requestString;
????????req.SerializeToString(&requestString);
????????//client 發送數據
????????client->send_message(requestString.c_str(), requestString.size(), lars::ID_GetRouteRequest);
????}
}
13 Lars-LbAgentV0.7-Udp-server-獲取路由服務業務注冊
void *dns_client_thread(void* args)
{
????printf("dns client thread start\n");
????event_loop loop;
????//1 加載配置文件得到dns service ip + port
????std::string ip = config_file::instance()->GetString("dnsserver", "ip", "");
????short port = config_file::instance()->GetNumber("dnsserver", "port", 0);
????//2 創建客戶端
????tcp_client client(&loop, ip.c_str(), port, "dns client");
????//3 將thread_queue消息回調事件,綁定到loop中
????dns_queue->set_loop(&loop);
????dns_queue->set_callback(new_dns_request, &client);
????//4 啟動事件監聽
????loop.event_process();?
????return NULL;
}
void start_dns_client()
{
????//開辟一個線程
????pthread_t tid;
????//啟動線程業務函數
????int ret = pthread_create(&tid, NULL, dns_client_thread, NULL);
????if (ret == -1) {
????????perror("pthread_create");
????????exit(1);
????}
????//設置分離模式
????pthread_detach(tid);
}
```
14 Lars-LbAgentV0.7-Udp-server-API和udpserver的測試
## 5) 負載均衡模塊基礎設計(V0.2)
### 5.1 基礎
?????????每個模塊`modid/cmdid`下有若干節點,節點的集合稱為此模塊的路由; 對于每個節點,有兩種狀態:
-?`idle`:此節點可用,可作為API**(相當于Agent的客戶端)**請求的節點使用;
-?`overload`:此節點過載,暫時不可作為API請求的節點使用
???????在請求節點時,有幾個關鍵屬性:
-?虛擬成功次數`vsucc`,API匯報節點調用結果是成功時,該值+1
-?虛擬失敗次數`verr`,API匯報節點調用結果是失敗時,該值+1
-?連續成功次數`contin_succ`,連續請求成功的次數
-?連續失敗次數`contin_err`,連續請求失敗的次數
這4個字段,在節點狀態改變時(idle<—>overload),會被重置。
?