98 greenplum 集群搭建過程中碰到的幾個問題

前言

最近有搭建 greenplum 集群的需求?

然后在搭建的過程中碰到了一些問題, 還是有一些時間開銷

并且問題也稍微有些復雜, 因此記錄一下

1. Do not have enough valid segments to start the array.

報錯日志信息如下?

20220408:14:15:29:021638 gpstart:gp1:gpadmin-[INFO]:-Master Started...
20220408:14:15:29:021638 gpstart:gp1:gpadmin-[INFO]:-Shutting down master
20220408:14:15:30:021638 gpstart:gp1:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-Process results...
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[ERROR]:-No segment started for content: 2.
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-dumping success segments: ['localhost.localdomain:/data/gpdb/pdata1/gpseg0:content=0:dbid=2:role=p:preferred_role=p:mode=n:status=u', 'localhost.localdomain:/data/gpdb/pdata2/gpseg1:content=1:dbid=3:role=p:preferred_role=p:mode=n:status=u']
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-----------------------------------------------------
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-DBID:4  FAILED  host:'localhost.localdomain' datadir:'/data/gpdb/pdata1/gpseg2' with reason:'Segment data directory does not exist for: '/data/gpdb/pdata1/gpseg2''
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-DBID:5  FAILED  host:'localhost.localdomain' datadir:'/data/gpdb/pdata2/gpseg3' with reason:'Segment data directory does not exist for: '/data/gpdb/pdata2/gpseg3''
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-----------------------------------------------------
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-----------------------------------------------------
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-   Successful segment starts                                            = 2
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[WARNING]:-Failed segment starts                                                = 2   <<<<<<<<
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-----------------------------------------------------
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-Successfully started 2 of 4 segment instances <<<<<<<<
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-----------------------------------------------------
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[WARNING]:-Segment instance startup failures reported
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[WARNING]:-Failed start 2 of 4 segment instances <<<<<<<<
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[WARNING]:-Review /home/gpadmin/gpAdminLogs/gpstart_20220408.log
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-----------------------------------------------------
20220408:14:15:31:021638 gpstart:gp1:gpadmin-[INFO]:-Commencing parallel segment instance shutdown, please wait...
20220408:14:15:32:021638 gpstart:gp1:gpadmin-[ERROR]:-gpstart error: Do not have enough valid segments to start the array.

問題分析?

這里觀察一個細節, 啟動成功的兩個 segment 是 localhost.localdomain:/data/gpdb/pdata1/gpseg0 和 localhost.localdomain:/data/gpdb/pdata2/gpseg1
然后 啟動失敗的兩個 segment 是 localhost.localdomain:/data/gpdb/pdata1/gpseg2 和 localhost.localdomain:/data/gpdb/pdata2/gpseg3
但是 實際我們期望的應該是 gp2:/data/gpdb/pdata1/gpseg0 和 gp2:/data/gpdb/pdata2/gpseg1 以及 gp3:/data/gpdb/pdata1/gpseg2 和 gp3:/data/gpdb/pdata2/gpseg3
因為可能涉及到需要 ssh 到對應的主機上面執行 啟動的相關命令
但是由于 hostname 本身存在問題, 導致 ssh 的時候出現問題
此處有一些不太明晰的地方, 為什么 gp2 所在的機器 兩個 segment 啟動成功了?
從日志中可以看到的是 在一個地方 ssh 到了 gp2 所在的機器

gpstart 啟動 segments?

相關代碼
bin/gpstartself.gparray = GpArray.initFromCatalog(self.dburl, utility=True)// 省略部分代碼segmentStartResult = segmentStartOp.startSegments(self.gparray, segmentsToStart, startMode, self.era)
gparray.initFromCatalogSELECT dbid, content, role, preferred_role, mode, status, hostname, address, port, datadir FROM pg_catalog.gp_segment_configuration ORDER BY content, preferred_role DESC然后通過這個查詢結果封裝 gparray
# 然后通過 hostname, 以及 datadir 的相關配置拿去 遠程執行 pg_ctl 命令20220409:15:12:18:007540 gpsegstart.py_gp3:gpadmin:gp3:gpadmin-[DEBUG]:-[worker1] finished cmd: Starting seg at dir /data/gpdb/pdata2/gpseg3 cmdStr='env GPSESSID=0000000000 GPERA=ea7fa135b0ce8dae_220409151208 $GPHOME/bin/pg_ctl -D /data/gpdb/pdata2/gpseg3 -l /data/gpdb/pdata2/gpseg3/pg_log/startup.log -w -t 600 -o " -p 6001 " start 2>&1'  had result: cmd had rc=0 completed=True halted=False20220409:15:12:18:007540 gpsegstart.py_gp3:gpadmin:gp3:gpadmin-[DEBUG]:-[worker0] finished cmd: Starting seg at dir /data/gpdb/pdata1/gpseg2 cmdStr='env GPSESSID=0000000000 GPERA=ea7fa135b0ce8dae_220409151208 $GPHOME/bin/pg_ctl -D /data/gpdb/pdata1/gpseg2 -l /data/gpdb/pdata1/gpseg2/pg_log/startup.log -w -t 600 -o " -p 6000 " start 2>&1'  had result: cmd had rc=0 completed=True halted=False20220409:15:12:17:008990 gpsegstart.py_gp2:gpadmin:gp2:gpadmin-[DEBUG]:-[worker0] finished cmd: Starting seg at dir /data/gpdb/pdata1/gpseg0 cmdStr='env GPSESSID=0000000000 GPERA=ea7fa135b0ce8dae_220409151208 $GPHOME/bin/pg_ctl -D /data/gpdb/pdata1/gpseg0 -l /data/gpdb/pdata1/gpseg0/pg_log/startup.log -w -t 600 -o " -p 6000 " start 2>&1'  had result: cmd had rc=0 completed=True halted=False20220409:15:12:17:008990 gpsegstart.py_gp2:gpadmin:gp2:gpadmin-[DEBUG]:-[worker1] finished cmd: Starting seg at dir /data/gpdb/pdata2/gpseg1 cmdStr='env GPSESSID=0000000000 GPERA=ea7fa135b0ce8dae_220409151208 $GPHOME/bin/pg_ctl -D /data/gpdb/pdata2/gpseg1 -l /data/gpdb/pdata2/gpseg1/pg_log/startup.log -w -t 600 -o " -p 6001 " start 2>&1'  had result: cmd had rc=0 completed=True halted=False
涉及到的相關腳本, gpstart, startSegments, gpsegstart, gp

gp_segment_configuraion 的來源

pg_catalog.gp_segment_configuration 的數據來自于 gpinitsystem.LOAD_QE_SYSTEM_DATA拿到 seg_hosts 之后, HOST_LOOKUP 根據 ip 獲取 hostname具體的方式是優先從 ~/.gphostcache 中獲取 ip -> hostname, 其次 遠程到 ip 對應的機器 通過 hostname 獲取到主機名, 然后 緩存到 ~/.gphostcache注意到這個緩存, 是存在時間有效性的, 因此 我之前更新了三臺機器的 hostname 之后, 數據庫表中的 gp_segment_configuration 的 hostname 依然為 localhost.localdomain涉及到的相關腳本 gpinitsystem, gphostcachelookup, gphostcache, unix

查看?gp_segment_configuraion

進入 postgres 的方式, PGOPTIONS="-c gp_session_role=utility" psql -d postgres
查詢 segment 的相關信息, select * from gp_segment_configuration order by dbid;
postgres=# select * from gp_segment_configuration order by dbid;dbid | content | role | preferred_role | mode | status | port |     hostname      | address |          datadir
------+---------+------+----------------+------+--------+------+-------------------+---------+---------------------------1 |      -1 | p    | p              | n    | u      | 5432 | host-172-19-165-8 | gp1     | /data/gpdb/master/gpseg-12 |       0 | p    | p              | n    | u      | 6000 | localhost.localdomain       | gp2     | /data/gpdb/pdata1/gpseg03 |       1 | p    | p              | n    | u      | 6001 | localhost.localdomain       | gp2     | /data/gpdb/pdata2/gpseg14 |       2 | p    | p              | n    | u      | 6000 | localhost.localdomain       | gp3     | /data/gpdb/pdata1/gpseg25 |       3 | p    | p              | n    | u      | 6001 | localhost.localdomain       | gp3     | /data/gpdb/pdata2/gpseg3

處理方式

    將 hostname 修改為對應的 gp2, gp3ERROR:  permission denied: "gp_segment_configuration" is a system catalog 的處理方式 set allow_system_table_mods='TRUE';

2. 業務表查詢阻塞

現象

創建一張測試表, 然后查詢這張測試表, 客戶端阻塞住

查詢當前查詢相關的進程信息

[root@gp1 gpdb]# ps -ef | grep SELEC
gpadmin   7784  7761  0 16:02 ?        00:00:00 postgres:  5432, gpadmin test 192.168.33.2(50275) con10 cmd28 SELECT
root      7900 25953  0 16:09 pts/0    00:00:00 grep --color=auto SELEC

查看查詢進程的堆棧信息如下, 可以看到 udp 的相關函數

    Thread 2 (Thread 0x7fd751f9c700 (LWP 29767)):#0  0x00007fd74da9420d in poll () from /lib64/libc.so.6#1  0x0000000000a7e031 in rxThreadFunc (arg=<optimized out>) at ic_udpifc.c:6234#2  0x00007fd74e60ddd5 in start_thread () from /lib64/libpthread.so.0#3  0x00007fd74da9eead in clone () from /lib64/libc.so.6Thread 1 (Thread 0x7fd752042880 (LWP 29766)):#0  0x00007fd74da9420d in poll () from /lib64/libc.so.6#1  0x00000000008012f3 in WaitLatchOrSocket (latch=latch@entry=0x1072530 <ic_control_info+144>, wakeEvents=wakeEvents@entry=27, sock=sock@entry=17, timeout=timeout@entry=250) at pg_latch.c:334#2  0x0000000000a7c38e in receiveChunksUDPIFC (pTransportStates=pTransportStates@entry=0x1566a78, pEntry=pEntry@entry=0x14637b8, motNodeID=motNodeID@entry=1, srcRoute=srcRoute@entry=0x7ffc897b6d6e, conn=conn@entry=0x0) at ic_udpifc.c:3799#3  0x0000000000a7c892 in RecvTupleChunkFromAnyUDPIFC_Internal (srcRoute=0x7ffc897b6d6e, motNodeID=1, transportStates=0x1566a78) at ic_udpifc.c:3913#4  RecvTupleChunkFromAnyUDPIFC (transportStates=0x1566a78, motNodeID=1, srcRoute=0x7ffc897b6d6e) at ic_udpifc.c:3933#5  0x0000000000a70eee in processIncomingChunks (srcRoute=<optimized out>, motNodeID=1, pMNEntry=0x13e1b88, transportStates=0x1566a78, mlStates=0x13e1ae8) at cdbmotion.c:659#6  RecvTupleFrom (mlStates=<optimized out>, transportStates=0x1566a78, motNodeID=1, srcRoute=srcRoute@entry=-100) at cdbmotion.c:615#7  0x0000000000706488 in execMotionUnsortedReceiver (node=0x14547a0) at nodeMotion.c:392#8  ExecMotion (node=node@entry=0x14547a0) at nodeMotion.c:213#9  0x00000000006c7d48 in ExecProcNode (node=node@entry=0x14547a0) at execProcnode.c:1121#10 0x00000000006eb528 in ExecLimit_guts (node=node@entry=0x14540a0) at nodeLimit.c:94#11 ExecLimit (node=node@entry=0x14540a0) at nodeLimit.c:234#12 0x00000000006c7d58 in ExecProcNode (node=node@entry=0x14540a0) at execProcnode.c:1117#13 0x00000000006bf269 in ExecutePlan (estate=estate@entry=0x1453ac8, planstate=0x14540a0, operation=operation@entry=CMD_SELECT, sendTuples=sendTuples@entry=1 '\001', numberTuples=numberTuples@entry=0, direction=direction@entry=ForwardScanDirection, dest=0x12cc9c0) at execMain.c:2980#14 0x00000000006bffdc in standard_ExecutorRun (queryDesc=0x156ea48, direction=ForwardScanDirection, count=0) at execMain.c:969#15 0x000000000088dbe7 in PortalRunSelect (portal=0x12d6a08, forward=<optimized out>, count=0, dest=<optimized out>) at pquery.c:1149#16 0x000000000088fb91 in PortalRun (portal=portal@entry=0x12d6a08, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=1 '\001', dest=dest@entry=0x12cc9c0, altdest=altdest@entry=0x12cc9c0, completionTag=completionTag@entry=0x7ffc897b7380 "") at pquery.c:990#17 0x000000000088a260 in exec_simple_query (query_string=0x12caaa8 "SELECT * FROM \"public\".\"test\" LIMIT 1000 OFFSET 0") at postgres.c:1798#18 0x000000000088ca05 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x12aa4c0, dbname=<optimized out>, username=<optimized out>) at postgres.c:5214#19 0x00000000004cb8cb in BackendRun (port=0x12daf10) at postmaster.c:4803#20 BackendStartup (port=0x12daf10) at postmaster.c:4460#21 ServerLoop () at postmaster.c:1945#22 0x0000000000814a86 in PostmasterMain (argc=argc@entry=6, argv=argv@entry=0x12a8540) at postmaster.c:1515#23 0x00000000004ce0e1 in main (argc=6, argv=0x12a8540) at main.c:245

查看進程 7784 占用 upd 端口的相關情況, 可以看到的是占用了兩個高位端口

多半可能是防火墻的問題, 增加 30000-65535 的端口開放, 限定 udp 協議

[root@gp1 gpdb]# netstat -nlp | grep 7784
udp6       0      0 :::54096                :::*                                7784/postgres:  543
udp6       0      0 :::47865                :::*                                7784/postgres:  543

處理方式

防火墻增加 30000-65535 端口的 udp 服務開放, 客戶端重新查詢, 查詢成功

完?