gRPC之gRPC負載均衡(客戶端負載均衡)(etcd)

1、gRPC負載均衡(客戶端負載均衡)(etcd)

本篇將基于etcd的服務發現前提下，介紹如何實現gRPC客戶端負載均衡。

1.1 gRPC負載均衡

gRPC官方文檔提供了關于gRPC負載均衡方案Load Balancing in gRPC

https://github.com/grpc/grpc/blob/master/doc/load-balancing.md

此方案是為gRPC設計的，下面我們對此進行分析。

1.1.1 對每次調用進行負載均衡

gRPC中的負載平衡是以每次調用為基礎，而不是以每個連接為基礎。換句話說，即使所有的請求都來自一個客戶

端，我們仍希望它們在所有的服務器上實現負載平衡。

1.1.2 負載均衡的方法

集中式（Proxy Model）

在這里插入圖片描述

在服務消費者和服務提供者之間有一個獨立的負載均衡（LB），通常是專門的硬件設備如 F5，或者基于軟件如

LVS，HAproxy等實現。LB上有所有服務的地址映射表，通常由運維配置注冊，當服務消費方調用某個目標服務

時，它向LB發起請求，由LB以某種策略，比如輪詢（Round-Robin）做負載均衡后將請求轉發到目標服務。LB一

般具備健康檢查能力，能自動摘除不健康的服務實例。

該方案主要問題：服務消費方、提供方之間增加了一級，有一定性能開銷，請求量大時，效率較低。

可能有讀者會認為集中式負載均衡存在這樣的問題，一旦負載均衡服務掛掉，那整個系統將不能使用。

解決方案：可以對負載均衡服務進行DNS負載均衡，通過對一個域名設置多個IP地址，每次DNS解析時輪詢

返回負載均衡服務地址，從而實現簡單的DNS負載均衡。

客戶端負載（Balancing-aware Client）

在這里插入圖片描述

針對第一個方案的不足，此方案將LB的功能集成到服務消費方進程里，也被稱為軟負載或者客戶端負載方案。服

務提供方啟動時，首先將服務地址注冊到服務注冊表，同時定期報心跳到服務注冊表以表明服務的存活狀態，相當

于健康檢查，服務消費方要訪問某個服務時，它通過內置的LB組件向服務注冊表查詢，同時緩存并定期刷新目標

服務地址列表，然后以某種負載均衡策略選擇一個目標服務地址，最后向目標服務發起請求。LB和服務發現能力

被分散到每一個服務消費者的進程內部，同時服務消費方和服務提供方之間是直接調用，沒有額外開銷，性能比較

好。

該方案主要問題：要用多種語言、多個版本的客戶端編寫和維護負載均衡策略，使客戶端的代碼大大復雜化。

獨立LB服務（External Load Balancing Service）

在這里插入圖片描述

該方案是針對第二種方案的不足而提出的一種折中方案，原理和第二種方案基本類似。

不同之處是將LB和服務發現功能從進程內移出來，變成主機上的一個獨立進程。主機上的一個或者多個服務要訪

問目標服務時，他們都通過同一主機上的獨立LB進程做服務發現和負載均衡。該方案也是一種分布式方案沒有單

點問題，服務調用方和LB之間是進程內調用性能好，同時該方案還簡化了服務調用方，不需要為不同語言開發客

戶庫。

本篇將介紹第二種負載均衡方法，客戶端負載均衡。

1.2 實現gRPC客戶端負載均衡

gRPC已提供了簡單的負載均衡策略（如：Round Robin），我們只需實現它提供的Builder和Resolver接口，

就能完成gRPC客戶端負載均衡。

type Builder interface {Build(target Target, cc ClientConn, opts BuildOption) (Resolver, error)Scheme() string
}

Builder接口：創建一個resolver（本文稱之服務發現），用于監視名稱解析更新。

Build方法：為給定目標創建一個新的resolver，當調用grpc.Dial()時執行。

Scheme方法：返回此resolver支持的方案。

Scheme定義可參考：https://github.com/grpc/grpc/blob/master/doc/naming.md

type Resolver interface {ResolveNow(ResolveNowOption)Close()
}

Resolver接口：監視指定目標的更新，包括地址更新和服務配置更新。

ResolveNow方法：被 gRPC 調用，以嘗試再次解析目標名稱。只用于提示，可忽略該方法。

Close方法：關閉resolver。

根據以上兩個接口，我們把服務發現的功能寫在Build方法中，把獲取到的負載均衡服務地址返回到客戶端，并

監視服務更新情況，以修改客戶端連接。

1.3 服務發現代碼修改

服務發現代碼./etcdv3/discovery.go的內容：

package etcdv3import ("context""github.com/coreos/etcd/mvcc/mvccpb""go.etcd.io/etcd/clientv3""google.golang.org/grpc/resolver""log""sync""time"
)const schema = "grpclb"// ServiceDiscovery 服務發現
type ServiceDiscovery struct {// etcd clientcli *clientv3.Clientcc  resolver.ClientConn// 服務列表serverList sync.Map
}// NewServiceDiscovery新建發現服務
func NewServiceDiscovery(endpoints []string) resolver.Builder {cli, err := clientv3.New(clientv3.Config{Endpoints:   endpoints,DialTimeout: 5 * time.Second,})if err != nil {log.Fatal(err)}return &ServiceDiscovery{cli: cli,}
}// Build為給定目標創建一個新的resolver,當調用grpc.Dial()時執行
func (s *ServiceDiscovery) Build(target resolver.Target, cc resolver.ClientConn, opts resolver.BuildOption) (resolver.Resolver, error) {log.Println("Build")s.cc = cc// /grpclb/simple_grpc/prefix := "/" + target.Scheme + "/" + target.Endpoint + "/"// 根據前綴獲取現有的keyresp, err := s.cli.Get(context.Background(), prefix, clientv3.WithPrefix())if err != nil {return nil, err}for _, ev := range resp.Kvs {s.SetServiceList(string(ev.Key), string(ev.Value))}s.cc.UpdateState(resolver.State{Addresses: s.getServices()})// 監視前綴，修改變更的servergo s.watcher(prefix)return s, nil
}// ResolveNow監視目標更新
func (s *ServiceDiscovery) ResolveNow(rn resolver.ResolveNowOption) {log.Println("ResolveNow")
}// Scheme return schema
func (s *ServiceDiscovery) Scheme() string {return schema
}// Close關閉
func (s *ServiceDiscovery) Close() {log.Println("Close")s.cli.Close()
}// watcher監聽前綴
func (s *ServiceDiscovery) watcher(prefix string) {rch := s.cli.Watch(context.Background(), prefix, clientv3.WithPrefix())log.Printf("watching prefix:%s now...", prefix)for wresp := range rch {for _, ev := range wresp.Events {switch ev.Type {// 新增或修改case mvccpb.PUT:s.SetServiceList(string(ev.Kv.Key), string(ev.Kv.Value))// 刪除case mvccpb.DELETE:s.DelServiceList(string(ev.Kv.Key))}}}
}// SetServiceList新增服務地址
func (s *ServiceDiscovery) SetServiceList(key, val string) {s.serverList.Store(key, resolver.Address{Addr: val})s.cc.UpdateState(resolver.State{Addresses: s.getServices()})log.Println("put key :", key, "val:", val)
}// DelServiceList刪除服務地址
func (s *ServiceDiscovery) DelServiceList(key string) {s.serverList.Delete(key)s.cc.UpdateState(resolver.State{Addresses: s.getServices()})log.Println("del key:", key)
}// GetServices獲取服務地址
func (s *ServiceDiscovery) getServices() []resolver.Address {addrs := make([]resolver.Address, 0, 10)s.serverList.Range(func(k, v interface{}) bool {addrs = append(addrs, v.(resolver.Address))return true})// [{localhost:8000  <nil> 0 <nil>}]return addrs
}

代碼主要修改以下地方：

1、把獲取的服務地址轉成resolver.Address，供gRPC客戶端連接。

2、根據schema的定義規則，修改key格式。

1.4 服務注冊代碼修改

服務發現代碼./etcdv3/register.go的內容：

package etcdv3import ("context""go.etcd.io/etcd/clientv3""log""time"
)// ServiceRegister創建租約注冊服務
type ServiceRegister struct {// etcd clientcli *clientv3.Client// 租約IDleaseID clientv3.LeaseID// 租約keepalieve相應chankeepAliveChan <-chan *clientv3.LeaseKeepAliveResponse// keykey string// valueval string
}// NewServiceRegister 新建注冊服務
func NewServiceRegister(endpoints []string, serName, addr string, lease int64) (*ServiceRegister, error) {cli, err := clientv3.New(clientv3.Config{Endpoints:   endpoints,DialTimeout: 5 * time.Second,})if err != nil {log.Fatal(err)}ser := &ServiceRegister{cli: cli,// /grpclb/simple_grpc/localhost:8000key: "/" + schema + "/" + serName + "/" + addr,val: addr,}// 申請租約設置時間keepaliveif err := ser.putKeyWithLease(lease); err != nil {return nil, err}return ser, nil
}// 設置租約
func (s *ServiceRegister) putKeyWithLease(lease int64) error {// 設置租約時間resp, err := s.cli.Grant(context.Background(), lease)if err != nil {return err}// 注冊服務并綁定租約_, err = s.cli.Put(context.Background(), s.key, s.val, clientv3.WithLease(resp.ID))if err != nil {return err}// 設置續租 定期發送需求請求leaseRespChan, err := s.cli.KeepAlive(context.Background(), resp.ID)if err != nil {return err}s.leaseID = resp.IDs.keepAliveChan = leaseRespChanlog.Printf("Put key:%s  val:%s  success!", s.key, s.val)return nil
}// ListenLeaseRespChan監聽續租情況
func (s *ServiceRegister) ListenLeaseRespChan() {for leaseKeepResp := range s.keepAliveChan {log.Println("續約成功", leaseKeepResp)}log.Println("關閉續租")
}// Close注銷服務
func (s *ServiceRegister) Close() error {// 撤銷租約if _, err := s.cli.Revoke(context.Background(), s.leaseID); err != nil {return err}log.Println("撤銷租約")return s.cli.Close()
}

服務注冊主要修改key存儲格式。

1.5 proto編寫和編譯

simple.proto文件的內容：

// 協議為proto3
syntax = "proto3";
package proto;
option go_package = "./proto;proto";// 定義發送請求信息
message SimpleRequest{// 定義發送的參數，采用駝峰命名方式，小寫加下劃線，如：student_name// 參數類型 參數名 標識號(不可重復)string data = 1;
}// 定義響應信息
message SimpleResponse{// 定義接收的參數// 參數類型 參數名 標識號(不可重復)int32 code = 1;string value = 2;
}// 定義我們的服務（可定義多個服務,每個服務可定義多個接口）
service Simple{rpc Route (SimpleRequest) returns (SimpleResponse){};
}

編譯simple.proto文件：

$ protoc --go_out=plugins=grpc:. simple.proto

1.6 gRPC客戶端

gRPC內置了簡單的負載均衡策略round_robin，根據負載均衡地址，以輪詢的方式進行調用服務。

客戶端修改gRPC連接服務的部分代碼，client.go內容如下：

package mainimport ("context""etcd2/etcdv3"pb "etcd2/proto""fmt""google.golang.org/grpc""google.golang.org/grpc/resolver""log""strconv""time"
)var (// EtcdEndpoints etcd集群地址EtcdEndpoints = []string{"localhost:2379"}// SerName服務名稱SerName    = "simple_grpc"grpcClient pb.SimpleClient
)func main() {r := etcdv3.NewServiceDiscovery(EtcdEndpoints)resolver.Register(r)// 連接服務器conn, err := grpc.Dial(// grpclb:///simple_grpcfmt.Sprintf("%s:///%s", r.Scheme(), SerName),grpc.WithBalancerName("round_robin"),grpc.WithInsecure(),)if err != nil {log.Fatalf("net.Connect err: %v", err)}defer conn.Close()// 建立gRPC連接grpcClient = pb.NewSimpleClient(conn)for i := 0; i < 100; i++ {route(i)time.Sleep(1 * time.Second)}
}// route調用服務端Route方法
func route(i int) {// 創建發送結構體req := pb.SimpleRequest{Data: "grpc " + strconv.Itoa(i),}// 調用我們的服務(Route方法)// 同時傳入了一個 context.Context ，在有需要時可以讓我們改變RPC的行為，比如超時/取消一個正在運行的RPCres, err := grpcClient.Route(context.Background(), &req)if err != nil {log.Fatalf("Call Route err: %v", err)}// 打印返回值log.Println(res)
}

1.7 gRPC服務端

服務端啟動時，把服務地址注冊到etcd中即可，server.go內容如下：

package mainimport ("context""etcd2/etcdv3"pb "etcd2/proto""google.golang.org/grpc""log""net"
)// SimpleService定義我們的服務
type SimpleService struct{}const (// Address監聽地址,服務端地址Address string = "localhost:8000"// Network網絡通信協議Network string = "tcp"// SerName服務名稱SerName string = "simple_grpc"
)// EtcdEndpoints etcd集群地址
var EtcdEndpoints = []string{"localhost:2379"}func main() {// 監聽本地端口listener, err := net.Listen(Network, Address)if err != nil {log.Fatalf("net.Listen err: %v", err)}log.Println(Address + " net.Listing...")// 新建gRPC服務器實例grpcServer := grpc.NewServer()// 在gRPC服務器注冊我們的服務pb.RegisterSimpleServer(grpcServer, &SimpleService{})// 把服務注冊到etcdser, err := etcdv3.NewServiceRegister(EtcdEndpoints, SerName, Address, 5)if err != nil {log.Fatalf("register service err: %v", err)}defer ser.Close()// 用服務器Serve()方法以及我們的端口信息區實現阻塞等待,直到進程被殺死或者Stop()被調用err = grpcServer.Serve(listener)if err != nil {log.Fatalf("grpcServer.Serve err: %v", err)}
}// Route實現Route方法
func (s *SimpleService) Route(ctx context.Context, req *pb.SimpleRequest) (*pb.SimpleResponse, error) {log.Println("receive: " + req.Data)res := pb.SimpleResponse{Code:  200,Value: "hello " + req.Data,}return &res, nil
}

1.8 運行效果

我們先啟動并注冊三個服務：

[root@zsx etcd2]# go run server.go
2023-02-13 20:51:54.173418 I | localhost:8000 net.Listing...
2023-02-13 20:51:54.207555 I | Put key:/grpclb/simple_grpc/localhost:8000  val:localhost:8000  success!

[root@zsx etcd2]# go run server1.go
2023-02-13 20:52:02.041409 I | localhost:8001 net.Listing...
2023-02-13 20:52:02.046411 I | Put key:/grpclb/simple_grpc/localhost:8001  val:localhost:8001  success!

[root@zsx etcd2]# go run server2.go
2023-02-13 20:52:06.644427 I | localhost:8002 net.Listing...
2023-02-13 20:52:06.647214 I | Put key:/grpclb/simple_grpc/localhost:8002  val:localhost:8002  success!

然后客戶端進行調用：

[root@zsx etcd2]# go run client.go
2023-02-13 20:53:43.841502 I | Build
2023-02-13 20:53:43.844638 I | put key : /grpclb/simple_grpc/localhost:8000 val: localhost:8000
2023-02-13 20:53:43.844659 I | put key : /grpclb/simple_grpc/localhost:8001 val: localhost:8001
2023-02-13 20:53:43.844668 I | put key : /grpclb/simple_grpc/localhost:8002 val: localhost:8002
2023-02-13 20:53:43.849315 I | watching prefix:/grpclb/simple_grpc/ now...
2023-02-13 20:53:43.849949 I | code:200 value:"hello grpc 0"
2023-02-13 20:53:44.851034 I | code:200 value:"hello grpc 1"
2023-02-13 20:53:45.852734 I | code:200 value:"hello grpc 2"
2023-02-13 20:53:46.854096 I | code:200 value:"hello grpc 3"
2023-02-13 20:53:47.855628 I | code:200 value:"hello grpc 4"
2023-02-13 20:53:48.857095 I | code:200 value:"hello grpc 5"
2023-02-13 20:53:49.858414 I | code:200 value:"hello grpc 6"
2023-02-13 20:53:50.859152 I | code:200 value:"hello grpc 7"
2023-02-13 20:53:51.860268 I | code:200 value:"hello grpc 8"
2023-02-13 20:53:52.860825 I | code:200 value:"hello grpc 9"
2023-02-13 20:53:53.861698 I | code:200 value:"hello grpc 10"
2023-02-13 20:53:54.864353 I | code:200 value:"hello grpc 11"
2023-02-13 20:53:55.865019 I | code:200 value:"hello grpc 12"
2023-02-13 20:53:56.865771 I | code:200 value:"hello grpc 13"
2023-02-13 20:53:57.866400 I | code:200 value:"hello grpc 14"
2023-02-13 20:53:58.867066 I | code:200 value:"hello grpc 15"
2023-02-13 20:53:59.868412 I | code:200 value:"hello grpc 16"
2023-02-13 20:54:00.869399 I | code:200 value:"hello grpc 17"
2023-02-13 20:54:01.870803 I | code:200 value:"hello grpc 18"
2023-02-13 20:54:02.872201 I | code:200 value:"hello grpc 19"
2023-02-13 20:54:03.872696 I | code:200 value:"hello grpc 20"
2023-02-13 20:54:04.874159 I | code:200 value:"hello grpc 21"
2023-02-13 20:54:05.875990 I | code:200 value:"hello grpc 22"
2023-02-13 20:54:06.877799 I | code:200 value:"hello grpc 23"
2023-02-13 20:54:07.878814 I | code:200 value:"hello grpc 24"
2023-02-13 20:54:08.881218 I | code:200 value:"hello grpc 25"
......

看服務端接收到的請求：

[root@zsx etcd2]# go run server.go
2023-02-13 20:51:54.173418 I | localhost:8000 net.Listing...
2023-02-13 20:51:54.207555 I | Put key:/grpclb/simple_grpc/localhost:8000  val:localhost:8000  success!
2023-02-13 20:53:43.849692 I | receive: grpc 0
2023-02-13 20:53:45.852147 I | receive: grpc 2
2023-02-13 20:53:48.856715 I | receive: grpc 5
2023-02-13 20:53:51.860086 I | receive: grpc 8
2023-02-13 20:53:54.863486 I | receive: grpc 11
2023-02-13 20:53:57.866286 I | receive: grpc 14
2023-02-13 20:54:00.869018 I | receive: grpc 17
2023-02-13 20:54:03.872545 I | receive: grpc 20
2023-02-13 20:54:06.877412 I | receive: grpc 23
......

[root@zsx etcd2]# go run server1.go
2023-02-13 20:52:02.041409 I | localhost:8001 net.Listing...
2023-02-13 20:52:02.046411 I | Put key:/grpclb/simple_grpc/localhost:8001  val:localhost:8001  success!
2023-02-13 20:53:46.853933 I | receive: grpc 3
2023-02-13 20:53:49.858075 I | receive: grpc 6
2023-02-13 20:53:52.860683 I | receive: grpc 9
2023-02-13 20:53:55.864896 I | receive: grpc 12
2023-02-13 20:53:58.866946 I | receive: grpc 15
2023-02-13 20:54:01.870692 I | receive: grpc 18
2023-02-13 20:54:04.874044 I | receive: grpc 21
......

[root@zsx etcd2]# go run server2.go
2023-02-13 20:52:06.644427 I | localhost:8002 net.Listing...
2023-02-13 20:52:06.647214 I | Put key:/grpclb/simple_grpc/localhost:8002  val:localhost:8002  success!
2023-02-13 20:53:44.850843 I | receive: grpc 1
2023-02-13 20:53:47.855201 I | receive: grpc 4
2023-02-13 20:53:50.859003 I | receive: grpc 7
2023-02-13 20:53:53.861487 I | receive: grpc 10
2023-02-13 20:53:56.865643 I | receive: grpc 13
2023-02-13 20:53:59.868263 I | receive: grpc 16
2023-02-13 20:54:02.872082 I | receive: grpc 19
2023-02-13 20:54:05.875695 I | receive: grpc 22
2023-02-13 20:54:08.880472 I | receive: grpc 25
......

關閉localhost:8000服務，剩余localhost:8001和localhost:8002服務接收請求：

[root@zsx etcd2]# go run server1.go
2023-02-13 20:56:48.915356 I | localhost:8001 net.Listing...
2023-02-13 20:56:48.917716 I | Put key:/grpclb/simple_grpc/localhost:8001  val:localhost:8001  success!
2023-02-13 20:56:57.599656 I | receive: grpc 1
2023-02-13 20:57:00.603955 I | receive: grpc 4
2023-02-13 20:57:03.607802 I | receive: grpc 7
2023-02-13 20:57:06.612719 I | receive: grpc 10
2023-02-13 20:57:09.615636 I | receive: grpc 13
2023-02-13 20:57:12.621450 I | receive: grpc 16
2023-02-13 20:57:15.624067 I | receive: grpc 19
2023-02-13 20:57:18.627407 I | receive: grpc 22
2023-02-13 20:57:21.630674 I | receive: grpc 25
2023-02-13 20:57:24.635254 I | receive: grpc 28
2023-02-13 20:57:27.638432 I | receive: grpc 31
2023-02-13 20:57:30.642211 I | receive: grpc 34
2023-02-13 20:57:32.644497 I | receive: grpc 36
2023-02-13 20:57:34.646649 I | receive: grpc 38
2023-02-13 20:57:36.650191 I | receive: grpc 40
2023-02-13 20:57:38.653029 I | receive: grpc 42
2023-02-13 20:57:40.654724 I | receive: grpc 44
2023-02-13 20:57:42.656665 I | receive: grpc 46
2023-02-13 20:57:44.658922 I | receive: grpc 48
2023-02-13 20:57:46.661156 I | receive: grpc 50
2023-02-13 20:57:48.663613 I | receive: grpc 52
2023-02-13 20:57:50.666520 I | receive: grpc 54
2023-02-13 20:57:52.668219 I | receive: grpc 56
2023-02-13 20:57:54.670777 I | receive: grpc 58
2023-02-13 20:57:56.673138 I | receive: grpc 60
2023-02-13 20:57:58.674908 I | receive: grpc 62
2023-02-13 20:58:00.677698 I | receive: grpc 64
2023-02-13 20:58:02.680816 I | receive: grpc 66
2023-02-13 20:58:04.686075 I | receive: grpc 68
......

[root@zsx etcd2]# go run server2.go
2023-02-13 20:56:51.462814 I | localhost:8002 net.Listing...
2023-02-13 20:56:51.467130 I | Put key:/grpclb/simple_grpc/localhost:8002  val:localhost:8002  success!
2023-02-13 20:56:58.600367 I | receive: grpc 2
2023-02-13 20:57:01.605054 I | receive: grpc 5
2023-02-13 20:57:04.608898 I | receive: grpc 8
2023-02-13 20:57:07.613784 I | receive: grpc 11
2023-02-13 20:57:10.617774 I | receive: grpc 14
2023-02-13 20:57:13.622261 I | receive: grpc 17
2023-02-13 20:57:16.624596 I | receive: grpc 20
2023-02-13 20:57:19.628076 I | receive: grpc 23
2023-02-13 20:57:22.632461 I | receive: grpc 26
2023-02-13 20:57:25.636681 I | receive: grpc 29
2023-02-13 20:57:28.639889 I | receive: grpc 32
2023-02-13 20:57:31.643062 I | receive: grpc 35
2023-02-13 20:57:33.645208 I | receive: grpc 37
2023-02-13 20:57:35.648217 I | receive: grpc 39
2023-02-13 20:57:37.652005 I | receive: grpc 41
2023-02-13 20:57:39.653990 I | receive: grpc 43
2023-02-13 20:57:41.656116 I | receive: grpc 45
2023-02-13 20:57:43.657680 I | receive: grpc 47
2023-02-13 20:57:45.659936 I | receive: grpc 49
2023-02-13 20:57:47.662667 I | receive: grpc 51
2023-02-13 20:57:49.665158 I | receive: grpc 53
2023-02-13 20:57:51.667787 I | receive: grpc 55
2023-02-13 20:57:53.669130 I | receive: grpc 57
2023-02-13 20:57:55.672291 I | receive: grpc 59
2023-02-13 20:57:57.673711 I | receive: grpc 61
2023-02-13 20:57:59.676281 I | receive: grpc 63
2023-02-13 20:58:01.679348 I | receive: grpc 65
2023-02-13 20:58:03.682384 I | receive: grpc 67
2023-02-13 20:58:05.687477 I | receive: grpc 69
......

重新打開localhost:8000服務

[root@zsx etcd2]# go run server.go
2023-02-13 20:58:05.315863 I | localhost:8000 net.Listing...
2023-02-13 20:58:05.322321 I | Put key:/grpclb/simple_grpc/localhost:8000  val:localhost:8000  success!
2023-02-13 20:58:06.688247 I | receive: grpc 70
2023-02-13 20:58:09.691796 I | receive: grpc 73
2023-02-13 20:58:12.697446 I | receive: grpc 76
2023-02-13 20:58:15.700649 I | receive: grpc 79
2023-02-13 20:58:18.704804 I | receive: grpc 82
2023-02-13 20:58:21.711890 I | receive: grpc 85
2023-02-13 20:58:24.721398 I | receive: grpc 88
2023-02-13 20:58:27.726881 I | receive: grpc 91
2023-02-13 20:58:30.733417 I | receive: grpc 94
2023-02-13 20:58:33.739445 I | receive: grpc 97

[root@zsx etcd2]# go run server1.go
2023-02-13 20:56:48.915356 I | localhost:8001 net.Listing...
2023-02-13 20:56:48.917716 I | Put key:/grpclb/simple_grpc/localhost:8001  val:localhost:8001  success!
2023-02-13 20:56:57.599656 I | receive: grpc 1
2023-02-13 20:57:00.603955 I | receive: grpc 4
2023-02-13 20:57:03.607802 I | receive: grpc 7
2023-02-13 20:57:06.612719 I | receive: grpc 10
2023-02-13 20:57:09.615636 I | receive: grpc 13
2023-02-13 20:57:12.621450 I | receive: grpc 16
2023-02-13 20:57:15.624067 I | receive: grpc 19
2023-02-13 20:57:18.627407 I | receive: grpc 22
2023-02-13 20:57:21.630674 I | receive: grpc 25
2023-02-13 20:57:24.635254 I | receive: grpc 28
2023-02-13 20:57:27.638432 I | receive: grpc 31
2023-02-13 20:57:30.642211 I | receive: grpc 34
2023-02-13 20:57:32.644497 I | receive: grpc 36
2023-02-13 20:57:34.646649 I | receive: grpc 38
2023-02-13 20:57:36.650191 I | receive: grpc 40
2023-02-13 20:57:38.653029 I | receive: grpc 42
2023-02-13 20:57:40.654724 I | receive: grpc 44
2023-02-13 20:57:42.656665 I | receive: grpc 46
2023-02-13 20:57:44.658922 I | receive: grpc 48
2023-02-13 20:57:46.661156 I | receive: grpc 50
2023-02-13 20:57:48.663613 I | receive: grpc 52
2023-02-13 20:57:50.666520 I | receive: grpc 54
2023-02-13 20:57:52.668219 I | receive: grpc 56
2023-02-13 20:57:54.670777 I | receive: grpc 58
2023-02-13 20:57:56.673138 I | receive: grpc 60
2023-02-13 20:57:58.674908 I | receive: grpc 62
2023-02-13 20:58:00.677698 I | receive: grpc 64
2023-02-13 20:58:02.680816 I | receive: grpc 66
2023-02-13 20:58:04.686075 I | receive: grpc 68
2023-02-13 20:58:07.689105 I | receive: grpc 71
2023-02-13 20:58:10.693713 I | receive: grpc 74
2023-02-13 20:58:13.698055 I | receive: grpc 77
2023-02-13 20:58:16.701562 I | receive: grpc 80
2023-02-13 20:58:19.706860 I | receive: grpc 83
2023-02-13 20:58:22.715951 I | receive: grpc 86
2023-02-13 20:58:25.723692 I | receive: grpc 89
2023-02-13 20:58:28.728865 I | receive: grpc 92
2023-02-13 20:58:31.735971 I | receive: grpc 95
2023-02-13 20:58:34.740998 I | receive: grpc 98

[root@zsx etcd2]# go run server2.go
2023-02-13 20:56:51.462814 I | localhost:8002 net.Listing...
2023-02-13 20:56:51.467130 I | Put key:/grpclb/simple_grpc/localhost:8002  val:localhost:8002  success!
2023-02-13 20:56:58.600367 I | receive: grpc 2
2023-02-13 20:57:01.605054 I | receive: grpc 5
2023-02-13 20:57:04.608898 I | receive: grpc 8
2023-02-13 20:57:07.613784 I | receive: grpc 11
2023-02-13 20:57:10.617774 I | receive: grpc 14
2023-02-13 20:57:13.622261 I | receive: grpc 17
2023-02-13 20:57:16.624596 I | receive: grpc 20
2023-02-13 20:57:19.628076 I | receive: grpc 23
2023-02-13 20:57:22.632461 I | receive: grpc 26
2023-02-13 20:57:25.636681 I | receive: grpc 29
2023-02-13 20:57:28.639889 I | receive: grpc 32
2023-02-13 20:57:31.643062 I | receive: grpc 35
2023-02-13 20:57:33.645208 I | receive: grpc 37
2023-02-13 20:57:35.648217 I | receive: grpc 39
2023-02-13 20:57:37.652005 I | receive: grpc 41
2023-02-13 20:57:39.653990 I | receive: grpc 43
2023-02-13 20:57:41.656116 I | receive: grpc 45
2023-02-13 20:57:43.657680 I | receive: grpc 47
2023-02-13 20:57:45.659936 I | receive: grpc 49
2023-02-13 20:57:47.662667 I | receive: grpc 51
2023-02-13 20:57:49.665158 I | receive: grpc 53
2023-02-13 20:57:51.667787 I | receive: grpc 55
2023-02-13 20:57:53.669130 I | receive: grpc 57
2023-02-13 20:57:55.672291 I | receive: grpc 59
2023-02-13 20:57:57.673711 I | receive: grpc 61
2023-02-13 20:57:59.676281 I | receive: grpc 63
2023-02-13 20:58:01.679348 I | receive: grpc 65
2023-02-13 20:58:03.682384 I | receive: grpc 67
2023-02-13 20:58:05.687477 I | receive: grpc 69
2023-02-13 20:58:08.690259 I | receive: grpc 72
2023-02-13 20:58:11.696068 I | receive: grpc 75
2023-02-13 20:58:14.699459 I | receive: grpc 78
2023-02-13 20:58:17.702863 I | receive: grpc 81
2023-02-13 20:58:20.709975 I | receive: grpc 84
2023-02-13 20:58:23.718648 I | receive: grpc 87
2023-02-13 20:58:26.725422 I | receive: grpc 90
2023-02-13 20:58:29.731039 I | receive: grpc 93
2023-02-13 20:58:32.737702 I | receive: grpc 96
2023-02-13 20:58:35.742928 I | receive: grpc 99

可以看到，gRPC客戶端負載均衡運行良好。

# 項目結構
[root@zsx protoc]# tree etcd2
etcd2
├── client.go
├── etcdv3
│   ├── discovery.go
│   └── register.go
├── go.mod
├── go.sum
├── proto
│   └── simple.pb.go
├── server1.go
├── server2.go
├── server.go
└── simple.proto2 directories, 10 files