OpenTenBase簡介
OpenTenBase是一個關系型數據庫集群平臺,提供寫入可靠性和多節點數據同步功能。可以在一臺或多臺主機上配置OpenTenBase,并將數據存儲在多個物理主機上。
OpenTenBase架構組件:
- Coordinator Node (CN):應用程序訪問入口,負責數據分布和查詢計劃。多個節點位于同一位置,每個節點提供相同的數據庫視圖
- Datanode Node (DN):每個DN存儲用戶數據的分區。在功能上,DN節點負責完成CN分發的執行請求
- GTM Node (Global Transaction Manager):負責集群事務信息的管理,以及集群的全局對象(如序列)
系統要求
硬件要求:
- 內存:最低4GB RAM
- 操作系統:OpenCloudOS 9
- 服務器:騰訊云CVM實例
軟件依賴:
gcc make readline-devel zlib-devel openssl-devel uuid-devel bison flex git
環境準備
1. 更新系統并安裝依賴包
由于OpenCloudOS支持dnf和yum兩種包管理軟件,強烈推薦用戶更多地使用dnf,我們使用dnf來安裝依賴:
# 更新系統
sudo dnf update -y
# 安裝OpenTenBase編譯依賴
sudo dnf install -y \gcc \gcc-c++ \make \cmake \readline-devel \zlib-devel \openssl-devel \uuid-devel \bison \flex \git \libcurl-devel \libxml2-devel \libxslt-devel \perl-IPC-Run \perl-Test-Simple \tcl-devel \python3-devel \rpm-build \pkgconfig \krb5-devel \openldap-devel
# 下載zstd源碼
cd /tmp
wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz
tar -xzf zstd-1.5.2.tar.gz
cd zstd-1.5.2# 編譯安裝
make
sudo make install PREFIX=/usr/local# 更新庫路徑
sudo ldconfig# 設置環境變量
export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
# 下載lz4源碼
cd /tmp
wget https://github.com/lz4/lz4/archive/v1.9.4.tar.gz
tar -xzf v1.9.4.tar.gz
cd lz4-1.9.4# 編譯安裝
make
sudo make install PREFIX=/usr/local# 更新庫路徑
sudo ldconfig# 設置環境變量
export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
# 首先檢查當前的包管理器狀態
sudo dnf clean all# 安裝libxml2-devel及其依賴包
sudo dnf install -y \libxml2-devel \libxml2 \cmake-filesystem \xz-devel \zlib-devel \pkgconfig# 驗證安裝
rpm -qa | grep libxml2
# 檢查xml2-config命令是否可用
which xml2-config# 檢查pkg-config是否能找到libxml-2.0
pkg-config --exists libxml-2.0 && echo "libxml2 found" || echo "libxml2 NOT found"# 查看libxml2的配置信息
xml2-config --version
xml2-config --cflags
xml2-config --libs
sudo dnf search cli11
sudo dnf install -y cli11-devel
2. 創建專用用戶
所有需要安裝OpenTenBase集群的機器都需要創建專用用戶:
# 創建數據目錄
sudo mkdir /data# 創建opentenbase用戶
sudo useradd -d /data/opentenbase -s /bin/bash -m opentenbase# 設置密碼
sudo passwd opentenbase
3. 切換到opentenbase用戶
su - opentenbase
源碼編譯安裝
1. 獲取源碼
cd /data/opentenbase
git clone https://gitee.com/mirrors/OpenTenBase.git
2. 編譯源碼
# 設置環境變量
export SOURCECODE_PATH=/data/opentenbase/OpenTenBase
export INSTALL_PATH=/data/opentenbase/install
# 進入源碼目錄
cd ${SOURCECODE_PATH}# 配置編譯選項
chmod +x configure*
# 編譯和安裝
make distclean 2>/dev/null || true
rm -rf /data/opentenbase/install/opentenbase_bin_v2.0
rm -f config.status config.log
# 重新配置,添加SSE4.2支持
CFLAGS="-g -O2 -w -msse4.2 -mcrc32" \
CXXFLAGS="-g -O2 -w -msse4.2 -mcrc32" \
./configure --prefix=/data/opentenbase/install/opentenbase_bin_v2.0 \--enable-user-switch \--with-openssl \--with-ossp-uuid \--with-libxml# 編譯
make
make install# 編譯contrib模塊
chmod +x contrib/pgxc_ctl/make_signature
cd contrib
make
make install
集中式單節點集群配置
1. 配置環境變量
# 編輯bashrc文件
vim ~/.bashrc# 添加以下內容:
export OPENTENBASE_HOME=/data/opentenbase/install/opentenbase_bin_v2.0
export PATH=$OPENTENBASE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$OPENTENBASE_HOME/lib:${LD_LIBRARY_PATH}
export LC_ALL=C# 生效環境變量
source ~/.bashrc
2. 創建集群配置目錄
mkdir /data/opentenbase/pgxc_ctl
cd /data/opentenbase/pgxc_ctl
3. 創建集中式配置文件
cat > /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf << 'EOF'
#!/usr/bin/env bash#---- OVERALL -----------------------------------------------------------------------------
pgxcOwner=opentenbase
pgxcUser=$pgxcOwner
tmpDir=/tmp
localTmpDir=$tmpDir
configBackup=y
configBackupHost=localhost
configBackupDir=$HOME/pgxc
configBackupFile=pgxc_ctl.bak#---- GTM --------------------------------------------------------------------------------
gtmName=gtm
gtmMasterServer=localhost
gtmMasterPort=6666
gtmMasterDir=/data/opentenbase/data/gtm
gtmExtraConfig=none
gtmMasterSpecificExtraConfig=none# GTM Slave - disabled
gtmSlave=n
gtmSlaveName=gtmSlave
gtmSlaveServer=none
gtmSlavePort=20001
gtmSlaveDir=none
gtmSlaveSpecificExtraConfig=none# GTM Proxy - disabled
gtmProxy=n
gtmProxyNames=()
gtmProxyServers=()
gtmProxyPorts=()
gtmProxyDirs=()
gtmPxyExtraConfig=none
gtmPxySpecificExtraConfig=()#---- Coordinators --------------------------------------------------------------------
coordMasterDir=/data/opentenbase/data/coord_master
coordSlaveDir=/data/opentenbase/data/coord_slave
coordArchLogDir=/data/opentenbase/data/coord_archlog# 協調器配置數組 - 所有數組必須有相同數量的元素
coordNames=(cn001)
coordPorts=(30004)
poolerPorts=(30014)
coordForwardPorts=(30024)
coordPgHbaEntries=(0.0.0.0/0)# Master Coordinators
coordMasterServers=(localhost)
coordMasterDirs=(/data/opentenbase/data/coord_master/cn001)
coordMaxWALsender=5
coordMaxWALSenders=(5)# Coordinator Slave - disabled
coordSlave=n
coordSlaveSync=n
coordSlaveServers=(none)
coordSlavePorts=(30005)
coordSlavePoolerPorts=(30015)
coordSlaveForwardPorts=(30025)
coordSlaveDirs=(none)
coordArchLogDirs=(none)# Configuration files
coordExtraConfig=none
coordSpecificExtraConfig=(none)
coordSpecificExtraPgHba=(none)#---- Datanodes -----------------------------------------------------------------------
datanodeMasterDir=/data/opentenbase/data/dn_master
datanodeSlaveDir=/data/opentenbase/data/dn_slave
datanodeArchLogDir=/data/opentenbase/data/datanode_archlog# 數據節點配置數組 - 所有數組必須有相同數量的元素
primaryDatanode=dn001
datanodeNames=(dn001)
datanodePorts=(20008)
datanodePoolerPorts=(20018)
datanodeForwardPorts=(20028)
datanodePgHbaEntries=(0.0.0.0/0)# Master Datanodes
datanodeMasterServers=(localhost)
datanodeMasterDirs=(/data/opentenbase/data/dn_master/dn001)
datanodeMaxWalSender=5
datanodeMaxWALSenders=(5)# Datanode Slave - disabled
datanodeSlave=n
datanodeSlaveServers=(none)
datanodeSlavePorts=(20009)
datanodeSlavePoolerPorts=(20019)
datanodeSlaveForwardPorts=(20029)
datanodeSlaveDirs=(none)
datanodeArchLogDirs=(none)# Configuration files
datanodeExtraConfig=none
datanodeSpecificExtraConfig=(none)
datanodeSpecificExtraPgHba=(none)# WAL Archive - disabled
walArchive=n
EOF
# 檢查配置文件語法
bash -n /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf
echo "配置文件語法檢查結果: $?"# 查看文件內容確認
head -20 /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf
檢查環境變量
# 檢查當前環境變量
echo $PATH
echo $OPENTENBASE_HOME
which initdb
which gtm_ctl
重新設置環境變量
# 退出pgxc_ctl
quit# 重新設置環境變量
export OPENTENBASE_HOME=/data/opentenbase/install/opentenbase_bin_v2.0
export PATH=$OPENTENBASE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$OPENTENBASE_HOME/lib:${LD_LIBRARY_PATH}
export LC_ALL=C# 驗證命令是否可用
which initdb
which gtm_ctl
which pg_ctl
永久保存環境變量
# 編輯 .bashrc 文件
vim ~/.bashrc# 添加以下內容到文件末尾:
export OPENTENBASE_HOME=/data/opentenbase/install/opentenbase_bin_v2.0
export PATH=$OPENTENBASE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$OPENTENBASE_HOME/lib:${LD_LIBRARY_PATH}
export LC_ALL=C# 重新加載環境變量
source ~/.bashrc
重新創建完整的配置文件
# 刪除不完整的配置文件
rm /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf# 重新創建完整配置文件
cat > /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf << 'EOF'
#!/usr/bin/env bash#---- OVERALL -----------------------------------------------------------------------------
pgxcOwner=opentenbase
pgxcUser=$pgxcOwner
tmpDir=/tmp
localTmpDir=$tmpDir
configBackup=y
configBackupHost=localhost
configBackupDir=$HOME/pgxc
configBackupFile=pgxc_ctl.bak#---- GTM --------------------------------------------------------------------------------
gtmName=gtm
gtmMasterServer=localhost
gtmMasterPort=6666
gtmMasterDir=/data/opentenbase/data/gtm
gtmExtraConfig=none
gtmMasterSpecificExtraConfig=nonegtmSlave=n
gtmSlaveName=gtmSlave
gtmSlaveServer=none
gtmSlavePort=20001
gtmSlaveDir=none
gtmSlaveSpecificExtraConfig=nonegtmProxy=n
gtmProxyNames=()
gtmProxyServers=()
gtmProxyPorts=()
gtmProxyDirs=()
gtmPxyExtraConfig=none
gtmPxySpecificExtraConfig=()#---- Coordinators --------------------------------------------------------------------
coordMasterDir=/data/opentenbase/data/coord_master
coordSlaveDir=/data/opentenbase/data/coord_slave
coordArchLogDir=/data/opentenbase/data/coord_archlogcoordNames=(cn001)
coordPorts=(30004)
poolerPorts=(30014)
coordForwardPorts=(30024)
coordPgHbaEntries=(0.0.0.0/0)coordMasterServers=(localhost)
coordMasterDirs=(/data/opentenbase/data/coord_master/cn001)
coordMaxWALsender=5
coordMaxWALSenders=(5)coordSlave=n
coordSlaveSync=n
coordSlaveServers=(none)
coordSlavePorts=(30005)
coordSlavePoolerPorts=(30015)
coordSlaveForwardPorts=(30025)
coordSlaveDirs=(none)
coordArchLogDirs=(none)coordExtraConfig=none
coordSpecificExtraConfig=(none)
coordSpecificExtraPgHba=(none)#---- Datanodes -----------------------------------------------------------------------
datanodeMasterDir=/data/opentenbase/data/dn_master
datanodeSlaveDir=/data/opentenbase/data/dn_slave
datanodeArchLogDir=/data/opentenbase/data/datanode_archlogprimaryDatanode=dn001
datanodeNames=(dn001)
datanodePorts=(20008)
datanodePoolerPorts=(20018)
datanodeForwardPorts=(20028)
datanodePgHbaEntries=(0.0.0.0/0)datanodeMasterServers=(localhost)
datanodeMasterDirs=(/data/opentenbase/data/dn_master/dn001)
datanodeMaxWalSender=5
datanodeMaxWALSenders=(5)datanodeSlave=n
datanodeSlaveServers=(none)
datanodeSlavePorts=(20009)
datanodeSlavePoolerPorts=(20019)
datanodeSlaveForwardPorts=(20029)
datanodeSlaveDirs=(none)
datanodeArchLogDirs=(none)datanodeExtraConfig=none
datanodeSpecificExtraConfig=(none)
datanodeSpecificExtraPgHba=(none)walArchive=n
EOF
驗證配置文件
# 檢查配置文件語法
bash -n /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf
echo "語法檢查結果: $?"# 查看文件完整性
wc -l /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf
tail -10 /data/opentenbase/install/opentenbase_bin_v2.0/pgxc_ctl.conf
配置SSH免密登錄
# 生成SSH密鑰(如果還沒有)
if [ ! -f ~/.ssh/id_rsa ]; thenssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
fi# 配置本地免密登錄
ssh-copy-id opentenbase@localhost
# 或者手動添加
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
測試SSH連接
# 測試SSH連接是否正常
ssh opentenbase@localhost "echo 'SSH connection test successful'"
重新運行pgxc_ctl
# 現在重新運行pgxc_ctl
pgxc_ctl# 在pgxc_ctl中執行:
deploy all
init all
start all
monitor all
4. 部署和初始化集群
# 啟動pgxc_ctl工具
pgxc_ctl# 在pgxc_ctl命令行中執行:
deploy all
init all
# 退出pgxc_ctl
exit# 設置opentenbase用戶的SSH密鑰認證
su - opentenbase# 生成SSH密鑰對
ssh-keygen -t rsa -b 2048 -f ~/.ssh/id_rsa -N ""# 將公鑰添加到authorized_keys
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
chmod 700 ~/.ssh# 測試SSH連接(應該不需要密碼)
ssh opentenbase@127.0.0.1 "echo 'SSH連接測試成功'"MQgnDKIwotjP9+vkGc9jehXXIzfXSJ2+ZAnFP5IDvIc.
5. 驗證集群狀態
# 在pgxc_ctl中檢查集群狀態
monitor all
正常情況下應該顯示:
Running: gtm master
Running: coordinator master cn001
Running: datanode master dn001
配置防火墻(可選)
如果啟用了防火墻,需要開放相應端口:
# 開放GTM端口
sudo firewall-cmd --permanent --add-port=50001/tcp# 開放Coordinator端口
sudo firewall-cmd --permanent --add-port=30004/tcp
sudo firewall-cmd --permanent --add-port=31110/tcp# 開放Datanode端口
sudo firewall-cmd --permanent --add-port=40004/tcp
sudo firewall-cmd --permanent --add-port=41110/tcp# 重新加載防火墻規則
sudo firewall-cmd --reload
數據庫初始化和使用
1. 連接數據庫
psql -h localhost -p 30004 -d postgres -U opentenbase
2. 創建必要的節點組和分片組
OpenTenBase使用數據節點組來增加節點管理的靈活性。需要創建一個默認組來使用,因此需要提前創建。通常,所有數據節點都會被添加到默認組中:
-- 創建默認節點組
CREATE DEFAULT NODE GROUP default_group WITH (dn001);-- 創建分片組
CREATE SHARDING GROUP TO GROUP default_group;
3. 創建數據庫和表
-- 創建測試數據庫
CREATE DATABASE testdb;-- 創建用戶
CREATE USER testuser WITH PASSWORD 'testpass';-- 授權
ALTER DATABASE testdb OWNER TO testuser;-- 切換到測試數據庫
\c testdb testuser-- 創建分片表
CREATE TABLE test_table(id BIGINT,name TEXT,created_time TIMESTAMP DEFAULT NOW()
) DISTRIBUTE BY SHARD(id);-- 插入測試數據
INSERT INTO test_table(id, name) VALUES
(1, 'OpenTenBase'),
(2, 'TencentCloud'),
(3, 'OpenCloudOS');-- 查詢測試
SELECT * FROM test_table;
集群管理
1. 啟動集群
pgxc_ctl
start all
2. 停止集群
pgxc_ctl
stop all
3. 清理集群(重新初始化時使用)
pgxc_ctl
clean all
故障排查
1. 查看日志
如果初始化失敗,可以查看日志:
# 查看pgxc_ctl日志
ls ~/pgxc_ctl/pgxc_log/
cat ~/pgxc_ctl/pgxc_log/最新的日志文件# 查看各組件日志
ls /data/opentenbase/data/gtm/pg_log/
ls /data/opentenbase/data/coord/pg_log/
ls /data/opentenbase/data/dn001/pg_log/
2. 常見問題解決
- 權限問題:確保opentenbase用戶對所有數據目錄有讀寫權限
- 端口沖突:檢查配置的端口是否被其他服務占用
- 內存不足:調整shared_buffers等內存參數
- 網絡問題:檢查防火墻和網絡連接
性能優化建議
1. 內存優化
根據服務器配置調整postgresql.conf中的內存參數:
shared_buffers = 25% of RAM # 例如8GB內存設置為2GB
effective_cache_size = 75% of RAM
work_mem = 4MB
maintenance_work_mem = 64MB
2. 連接優化
max_connections = 200 # 根據應用需求調整
max_pool_size = 1000 # 連接池大小
3. 日志優化
log_min_duration_statement = 1000 # 記錄執行時間超過1秒的查詢
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '
log_checkpoints = on
log_connections = on
log_disconnections = on