【HBase】HBaseJMX 接口監控信息實現釘釘告警

目錄

一、JMX 簡介

二、JMX監控信息釘釘告警實現

一、JMX 簡介

官網：Apache HBase ? Reference Guide

JMX （Java管理擴展）提供了內置的工具，使您能夠監視和管理Java VM。要啟用遠程系統的監視和管理，需要在啟動Java VM時設置系統屬性com.sun.management.jmxremote.port（希望通過該端口號啟用JMX RMI連接）。

訪問：

curl http://hdp-node2:16030/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server

輸出的 指標如下：

{"beans" : [ {"name" : "Hadoop:service=HBase,name=RegionServer,sub=Server","modelerType" : "RegionServer,sub=Server","tag.zookeeperQuorum" : "hdp-node2:2181,hdp-node3:2181,hdp-node1:2181","tag.serverName" : "hdp-node2,16020,1738720067137","tag.clusterId" : "85aa06c7-b28c-41fd-aa17-a49376641751","tag.Context" : "regionserver","tag.Hostname" : "hdp-node2","regionCount" : 34,"storeCount" : 51,"hlogFileCount" : 1,"hlogFileSize" : 0,"storeFileCount" : 31,"memStoreSize" : 0,"storeFileSize" : 212110208,"maxStoreFileAge" : 23910274739,"minStoreFileAge" : 595075791,"avgStoreFileAge" : 12083745007,"numReferenceFiles" : 0,"regionServerStartTime" : 1738720067137,"averageRegionSize" : 6238535,"storeFileIndexSize" : 529736,"staticIndexSize" : 1348988,"staticBloomSize" : 2438222,"mutationsWithoutWALCount" : 0,"mutationsWithoutWALSize" : 0,"percentFilesLocal" : 100.0,"percentFilesLocalSecondaryRegions" : 0.0,"splitQueueLength" : 0,"compactionQueueLength" : 0,"smallCompactionQueueLength" : 0,"largeCompactionQueueLength" : 0,"flushQueueLength" : 0,"blockCacheFreeSize" : 1716727624,"blockCacheCount" : 0,"blockCacheSize" : 1259320,"blockCacheCountHitPercent" : 0.0,"blockCacheExpressHitPercent" : 0.0,"l1CacheHitCount" : 0,"l1CacheMissCount" : 0,"l1CacheHitRatio" : 0.0,"l1CacheMissRatio" : 0.0,"l2CacheHitCount" : 0,"l2CacheMissCount" : 0,"l2CacheHitRatio" : 0.0,"l2CacheMissRatio" : 0.0,"mobFileCacheCount" : 0,"mobFileCacheHitPercent" : 0.0,"totalRequestCount" : 2,"totalRowActionRequestCount" : 0,"readRequestCount" : 0,"filteredReadRequestCount" : 0,"writeRequestCount" : 0,"rpcGetRequestCount" : 0,"rpcScanRequestCount" : 0,"rpcMultiRequestCount" : 0,"rpcMutateRequestCount" : 0,"checkMutateFailedCount" : 0,"checkMutatePassedCount" : 0,"blockCacheHitCount" : 0,"blockCacheHitCountPrimary" : 0,"blockCacheMissCount" : 0,"blockCacheMissCountPrimary" : 0,"blockCacheEvictionCount" : 0,"blockCacheEvictionCountPrimary" : 0,"blockCacheFailedInsertionCount" : 0,"blockCacheDataMissCount" : 0,"blockCacheLeafIndexMissCount" : 0,"blockCacheBloomChunkMissCount" : 0,"blockCacheMetaMissCount" : 0,"blockCacheRootIndexMissCount" : 0,"blockCacheIntermediateIndexMissCount" : 0,"blockCacheFileInfoMissCount" : 0,"blockCacheGeneralBloomMetaMissCount" : 0,"blockCacheDeleteFamilyBloomMissCount" : 0,"blockCacheTrailerMissCount" : 0,"blockCacheDataHitCount" : 0,"blockCacheLeafIndexHitCount" : 0,"blockCacheBloomChunkHitCount" : 0,"blockCacheMetaHitCount" : 0,"blockCacheRootIndexHitCount" : 0,"blockCacheIntermediateIndexHitCount" : 0,"blockCacheFileInfoHitCount" : 0,"blockCacheGeneralBloomMetaHitCount" : 0,"blockCacheDeleteFamilyBloomHitCount" : 0,"blockCacheTrailerHitCount" : 0,"updatesBlockedTime" : 0,"flushedCellsCount" : 0,"compactedCellsCount" : 0,"majorCompactedCellsCount" : 0,"flushedCellsSize" : 0,"compactedCellsSize" : 0,"majorCompactedCellsSize" : 0,"cellsCountCompactedFromMob" : 0,"cellsCountCompactedToMob" : 0,"cellsSizeCompactedFromMob" : 0,"cellsSizeCompactedToMob" : 0,"mobFlushCount" : 0,"mobFlushedCellsCount" : 0,"mobFlushedCellsSize" : 0,"mobScanCellsCount" : 0,"mobScanCellsSize" : 0,"mobFileCacheAccessCount" : 0,"mobFileCacheMissCount" : 0,"mobFileCacheEvictedCount" : 0,"hedgedReads" : 0,"hedgedReadWins" : 0,"blockedRequestCount" : 0,"MajorCompactionTime_num_ops" : 2,"MajorCompactionTime_min" : 0,"MajorCompactionTime_max" : 0,"MajorCompactionTime_mean" : 0,"MajorCompactionTime_25th_percentile" : 0,"MajorCompactionTime_median" : 0,"MajorCompactionTime_75th_percentile" : 0,"MajorCompactionTime_90th_percentile" : 0,"MajorCompactionTime_95th_percentile" : 0,"MajorCompactionTime_98th_percentile" : 0,"MajorCompactionTime_99th_percentile" : 0,"MajorCompactionTime_99.9th_percentile" : 0,"MajorCompactionTime_TimeRangeCount_600000-inf" : 2,"PauseTimeWithGc_num_ops" : 0,"PauseTimeWithGc_min" : 0,"PauseTimeWithGc_max" : 0,"PauseTimeWithGc_mean" : 0,"PauseTimeWithGc_25th_percentile" : 0,"PauseTimeWithGc_median" : 0,"PauseTimeWithGc_75th_percentile" : 0,"PauseTimeWithGc_90th_percentile" : 0,"PauseTimeWithGc_95th_percentile" : 0,"PauseTimeWithGc_98th_percentile" : 0,"PauseTimeWithGc_99th_percentile" : 0,"PauseTimeWithGc_99.9th_percentile" : 0,"compactedOutputBytes" : 8924,"pauseWarnThresholdExceeded" : 0,"ScanTime_num_ops" : 0,"ScanTime_min" : 0,"ScanTime_max" : 0,"ScanTime_mean" : 0,"ScanTime_25th_percentile" : 0,"ScanTime_median" : 0,"ScanTime_75th_percentile" : 0,"ScanTime_90th_percentile" : 0,"ScanTime_95th_percentile" : 0,"ScanTime_98th_percentile" : 0,"ScanTime_99th_percentile" : 0,"ScanTime_99.9th_percentile" : 0,"Increment_num_ops" : 0,"Increment_min" : 0,"Increment_max" : 0,"Increment_mean" : 0,"Increment_25th_percentile" : 0,"Increment_median" : 0,"Increment_75th_percentile" : 0,"Increment_90th_percentile" : 0,"Increment_95th_percentile" : 0,"Increment_98th_percentile" : 0,"Increment_99th_percentile" : 0,"Increment_99.9th_percentile" : 0,"Delete_num_ops" : 0,"Delete_min" : 0,"Delete_max" : 0,"Delete_mean" : 0,"Delete_25th_percentile" : 0,"Delete_median" : 0,"Delete_75th_percentile" : 0,"Delete_90th_percentile" : 0,"Delete_95th_percentile" : 0,"Delete_98th_percentile" : 0,"Delete_99th_percentile" : 0,"Delete_99.9th_percentile" : 0,"Put_num_ops" : 0,"Put_min" : 0,"Put_max" : 0,"Put_mean" : 0,"Put_25th_percentile" : 0,"Put_median" : 0,"Put_75th_percentile" : 0,"Put_90th_percentile" : 0,"Put_95th_percentile" : 0,"Put_98th_percentile" : 0,"Put_99th_percentile" : 0,"Put_99.9th_percentile" : 0,"DeleteBatch_num_ops" : 0,"DeleteBatch_min" : 0,"DeleteBatch_max" : 0,"DeleteBatch_mean" : 0,"DeleteBatch_25th_percentile" : 0,"DeleteBatch_median" : 0,"DeleteBatch_75th_percentile" : 0,"DeleteBatch_90th_percentile" : 0,"DeleteBatch_95th_percentile" : 0,"DeleteBatch_98th_percentile" : 0,"DeleteBatch_99th_percentile" : 0,"DeleteBatch_99.9th_percentile" : 0,"splitRequestCount" : 0,"FlushMemstoreSize_num_ops" : 0,"FlushMemstoreSize_min" : 0,"FlushMemstoreSize_max" : 0,"FlushMemstoreSize_mean" : 0,"FlushMemstoreSize_25th_percentile" : 0,"FlushMemstoreSize_median" : 0,"FlushMemstoreSize_75th_percentile" : 0,"FlushMemstoreSize_90th_percentile" : 0,"FlushMemstoreSize_95th_percentile" : 0,"FlushMemstoreSize_98th_percentile" : 0,"FlushMemstoreSize_99th_percentile" : 0,"FlushMemstoreSize_99.9th_percentile" : 0,"CompactionInputFileCount_num_ops" : 2,"CompactionInputFileCount_min" : 0,"CompactionInputFileCount_max" : 0,"CompactionInputFileCount_mean" : 0,"CompactionInputFileCount_25th_percentile" : 0,"CompactionInputFileCount_median" : 0,"CompactionInputFileCount_75th_percentile" : 0,"CompactionInputFileCount_90th_percentile" : 0,"CompactionInputFileCount_95th_percentile" : 0,"CompactionInputFileCount_98th_percentile" : 0,"CompactionInputFileCount_99th_percentile" : 0,"CompactionInputFileCount_99.9th_percentile" : 0,"PutBatch_num_ops" : 0,"PutBatch_min" : 0,"PutBatch_max" : 0,"PutBatch_mean" : 0,"PutBatch_25th_percentile" : 0,"PutBatch_median" : 0,"PutBatch_75th_percentile" : 0,"PutBatch_90th_percentile" : 0,"PutBatch_95th_percentile" : 0,"PutBatch_98th_percentile" : 0,"PutBatch_99th_percentile" : 0,"PutBatch_99.9th_percentile" : 0,"CompactionTime_num_ops" : 2,"CompactionTime_min" : 0,"CompactionTime_max" : 0,"CompactionTime_mean" : 0,"CompactionTime_25th_percentile" : 0,"CompactionTime_median" : 0,"CompactionTime_75th_percentile" : 0,"CompactionTime_90th_percentile" : 0,"CompactionTime_95th_percentile" : 0,"CompactionTime_98th_percentile" : 0,"CompactionTime_99th_percentile" : 0,"CompactionTime_99.9th_percentile" : 0,"CompactionTime_TimeRangeCount_600000-inf" : 2,"Get_num_ops" : 0,"Get_min" : 0,"Get_max" : 0,"Get_mean" : 0,"Get_25th_percentile" : 0,"Get_median" : 0,"Get_75th_percentile" : 0,"Get_90th_percentile" : 0,"Get_95th_percentile" : 0,"Get_98th_percentile" : 0,"Get_99th_percentile" : 0,"Get_99.9th_percentile" : 0,"MajorCompactionInputFileCount_num_ops" : 2,"MajorCompactionInputFileCount_min" : 0,"MajorCompactionInputFileCount_max" : 0,"MajorCompactionInputFileCount_mean" : 0,"MajorCompactionInputFileCount_25th_percentile" : 0,"MajorCompactionInputFileCount_median" : 0,"MajorCompactionInputFileCount_75th_percentile" : 0,"MajorCompactionInputFileCount_90th_percentile" : 0,"MajorCompactionInputFileCount_95th_percentile" : 0,"MajorCompactionInputFileCount_98th_percentile" : 0,"MajorCompactionInputFileCount_99th_percentile" : 0,"MajorCompactionInputFileCount_99.9th_percentile" : 0,"CheckAndPut_num_ops" : 0,"CheckAndPut_min" : 0,"CheckAndPut_max" : 0,"CheckAndPut_mean" : 0,"CheckAndPut_25th_percentile" : 0,"CheckAndPut_median" : 0,"CheckAndPut_75th_percentile" : 0,"CheckAndPut_90th_percentile" : 0,"CheckAndPut_95th_percentile" : 0,"CheckAndPut_98th_percentile" : 0,"CheckAndPut_99th_percentile" : 0,"CheckAndPut_99.9th_percentile" : 0,"SplitTime_num_ops" : 0,"SplitTime_min" : 0,"SplitTime_max" : 0,"SplitTime_mean" : 0,"SplitTime_25th_percentile" : 0,"SplitTime_median" : 0,"SplitTime_75th_percentile" : 0,"SplitTime_90th_percentile" : 0,"SplitTime_95th_percentile" : 0,"SplitTime_98th_percentile" : 0,"SplitTime_99th_percentile" : 0,"SplitTime_99.9th_percentile" : 0,"MajorCompactionOutputSize_num_ops" : 2,"MajorCompactionOutputSize_min" : 0,"MajorCompactionOutputSize_max" : 0,"MajorCompactionOutputSize_mean" : 0,"MajorCompactionOutputSize_25th_percentile" : 0,"MajorCompactionOutputSize_median" : 0,"MajorCompactionOutputSize_75th_percentile" : 0,"MajorCompactionOutputSize_90th_percentile" : 0,"MajorCompactionOutputSize_95th_percentile" : 0,"MajorCompactionOutputSize_98th_percentile" : 0,"MajorCompactionOutputSize_99th_percentile" : 0,"MajorCompactionOutputSize_99.9th_percentile" : 0,"MajorCompactionOutputSize_SizeRangeCount_100000000-inf" : 2,"majorCompactedInputBytes" : 8924,"slowAppendCount" : 0,"flushedOutputBytes" : 0,"CompactionOutputFileCount_num_ops" : 2,"CompactionOutputFileCount_min" : 0,"CompactionOutputFileCount_max" : 0,"CompactionOutputFileCount_mean" : 0,"CompactionOutputFileCount_25th_percentile" : 0,"CompactionOutputFileCount_median" : 0,"CompactionOutputFileCount_75th_percentile" : 0,"CompactionOutputFileCount_90th_percentile" : 0,"CompactionOutputFileCount_95th_percentile" : 0,"CompactionOutputFileCount_98th_percentile" : 0,"CompactionOutputFileCount_99th_percentile" : 0,"CompactionOutputFileCount_99.9th_percentile" : 0,"slowDeleteCount" : 0,"Replay_num_ops" : 0,"Replay_min" : 0,"Replay_max" : 0,"Replay_mean" : 0,"Replay_25th_percentile" : 0,"Replay_median" : 0,"Replay_75th_percentile" : 0,"Replay_90th_percentile" : 0,"Replay_95th_percentile" : 0,"Replay_98th_percentile" : 0,"Replay_99th_percentile" : 0,"Replay_99.9th_percentile" : 0,"FlushTime_num_ops" : 0,"FlushTime_min" : 0,"FlushTime_max" : 0,"FlushTime_mean" : 0,"FlushTime_25th_percentile" : 0,"FlushTime_median" : 0,"FlushTime_75th_percentile" : 0,"FlushTime_90th_percentile" : 0,"FlushTime_95th_percentile" : 0,"FlushTime_98th_percentile" : 0,"FlushTime_99th_percentile" : 0,"FlushTime_99.9th_percentile" : 0,"MajorCompactionInputSize_num_ops" : 2,"MajorCompactionInputSize_min" : 0,"MajorCompactionInputSize_max" : 0,"MajorCompactionInputSize_mean" : 0,"MajorCompactionInputSize_25th_percentile" : 0,"MajorCompactionInputSize_median" : 0,"MajorCompactionInputSize_75th_percentile" : 0,"MajorCompactionInputSize_90th_percentile" : 0,"MajorCompactionInputSize_95th_percentile" : 0,"MajorCompactionInputSize_98th_percentile" : 0,"MajorCompactionInputSize_99th_percentile" : 0,"MajorCompactionInputSize_99.9th_percentile" : 0,"MajorCompactionInputSize_SizeRangeCount_100000000-inf" : 2,"pauseInfoThresholdExceeded" : 0,"splitSuccessCount" : 0,"CheckAndDelete_num_ops" : 0,"CheckAndDelete_min" : 0,"CheckAndDelete_max" : 0,"CheckAndDelete_mean" : 0,"CheckAndDelete_25th_percentile" : 0,"CheckAndDelete_median" : 0,"CheckAndDelete_75th_percentile" : 0,"CheckAndDelete_90th_percentile" : 0,"CheckAndDelete_95th_percentile" : 0,"CheckAndDelete_98th_percentile" : 0,"CheckAndDelete_99th_percentile" : 0,"CheckAndDelete_99.9th_percentile" : 0,"CompactionInputSize_num_ops" : 2,"CompactionInputSize_min" : 0,"CompactionInputSize_max" : 0,"CompactionInputSize_mean" : 0,"CompactionInputSize_25th_percentile" : 0,"CompactionInputSize_median" : 0,"CompactionInputSize_75th_percentile" : 0,"CompactionInputSize_90th_percentile" : 0,"CompactionInputSize_95th_percentile" : 0,"CompactionInputSize_98th_percentile" : 0,"CompactionInputSize_99th_percentile" : 0,"CompactionInputSize_99.9th_percentile" : 0,"CompactionInputSize_SizeRangeCount_100000000-inf" : 2,"MajorCompactionOutputFileCount_num_ops" : 2,"MajorCompactionOutputFileCount_min" : 0,"MajorCompactionOutputFileCount_max" : 0,"MajorCompactionOutputFileCount_mean" : 0,"MajorCompactionOutputFileCount_25th_percentile" : 0,"MajorCompactionOutputFileCount_median" : 0,"MajorCompactionOutputFileCount_75th_percentile" : 0,"MajorCompactionOutputFileCount_90th_percentile" : 0,"MajorCompactionOutputFileCount_95th_percentile" : 0,"MajorCompactionOutputFileCount_98th_percentile" : 0,"MajorCompactionOutputFileCount_99th_percentile" : 0,"MajorCompactionOutputFileCount_99.9th_percentile" : 0,"ScanSize_num_ops" : 0,"ScanSize_min" : 0,"ScanSize_max" : 0,"ScanSize_mean" : 0,"ScanSize_25th_percentile" : 0,"ScanSize_median" : 0,"ScanSize_75th_percentile" : 0,"ScanSize_90th_percentile" : 0,"ScanSize_95th_percentile" : 0,"ScanSize_98th_percentile" : 0,"ScanSize_99th_percentile" : 0,"ScanSize_99.9th_percentile" : 0,"slowGetCount" : 0,"flushedMemstoreBytes" : 0,"CompactionOutputSize_num_ops" : 2,"CompactionOutputSize_min" : 0,"CompactionOutputSize_max" : 0,"CompactionOutputSize_mean" : 0,"CompactionOutputSize_25th_percentile" : 0,"CompactionOutputSize_median" : 0,"CompactionOutputSize_75th_percentile" : 0,"CompactionOutputSize_90th_percentile" : 0,"CompactionOutputSize_95th_percentile" : 0,"CompactionOutputSize_98th_percentile" : 0,"CompactionOutputSize_99th_percentile" : 0,"CompactionOutputSize_99.9th_percentile" : 0,"CompactionOutputSize_SizeRangeCount_100000000-inf" : 2,"majorCompactedOutputBytes" : 8924,"PauseTimeWithoutGc_num_ops" : 0,"PauseTimeWithoutGc_min" : 0,"PauseTimeWithoutGc_max" : 0,"PauseTimeWithoutGc_mean" : 0,"PauseTimeWithoutGc_25th_percentile" : 0,"PauseTimeWithoutGc_median" : 0,"PauseTimeWithoutGc_75th_percentile" : 0,"PauseTimeWithoutGc_90th_percentile" : 0,"PauseTimeWithoutGc_95th_percentile" : 0,"PauseTimeWithoutGc_98th_percentile" : 0,"PauseTimeWithoutGc_99th_percentile" : 0,"PauseTimeWithoutGc_99.9th_percentile" : 0,"slowPutCount" : 0,"slowIncrementCount" : 0,"compactedInputBytes" : 8924,"Append_num_ops" : 0,"Append_min" : 0,"Append_max" : 0,"Append_mean" : 0,"Append_25th_percentile" : 0,"Append_median" : 0,"Append_75th_percentile" : 0,"Append_90th_percentile" : 0,"Append_95th_percentile" : 0,"Append_98th_percentile" : 0,"Append_99th_percentile" : 0,"Append_99.9th_percentile" : 0,"FlushOutputSize_num_ops" : 0,"FlushOutputSize_min" : 0,"FlushOutputSize_max" : 0,"FlushOutputSize_mean" : 0,"FlushOutputSize_25th_percentile" : 0,"FlushOutputSize_median" : 0,"FlushOutputSize_75th_percentile" : 0,"FlushOutputSize_90th_percentile" : 0,"FlushOutputSize_95th_percentile" : 0,"FlushOutputSize_98th_percentile" : 0,"FlushOutputSize_99th_percentile" : 0,"FlushOutputSize_99.9th_percentile" : 0,"Bulkload_count" : 0,"Bulkload_mean_rate" : 0.0,"Bulkload_1min_rate" : 0.0,"Bulkload_5min_rate" : 0.0,"Bulkload_15min_rate" : 0.0,"Bulkload_num_ops" : 0,"Bulkload_min" : 0,"Bulkload_max" : 0,"Bulkload_mean" : 0,"Bulkload_25th_percentile" : 0,"Bulkload_median" : 0,"Bulkload_75th_percentile" : 0,"Bulkload_90th_percentile" : 0,"Bulkload_95th_percentile" : 0,"Bulkload_98th_percentile" : 0,"Bulkload_99th_percentile" : 0,"Bulkload_99.9th_percentile" : 0} ]
}

如上監控主要是HBase 內某個RegionServer 詳細信息。具體有gc， scan，flush ，block,compaction 等細粒度的監控。

二、JMX監控信息釘釘告警實現

下面我們實現了一個RegionServer運行時長的釘釘通知消息

# -*- coding: utf-8 -*-import time
import requests
import jsonimport schedule as schedule"""
~~~~~~~~~~~~
author: kangll
date: 2025/02/11 11:50
desc: reid cluster  HBase JMX  獲取指標信息
-- curl 請求: curl http://hdp-node2:16030/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server2小時發出一次正常的RS運行狀態信息，RS運行時間小于10分鐘且當前小時立即發出一次告警
"""__author__ = 'kanglilong  <kangll@winnerinf.com>'headers = {'Content-Type': 'application/json;charset=utf-8'}
hostArr = {"hdp-node1", "hdp-node2", "hdp-node3"}
dingding_url = "https://oapi.dingtalk.com/robot/send?access_token=ba7693ae5a1a5a4cda1358f35b19785a6d8a7659da92ba3685d6532994a6d82c"# 記錄上一次發送運行時間小于 10 分鐘告警的小時
last_less_than_10mins_alert_hour = Nonedef jmxGetHBaseStatus(regionserver_host):"""從 HBase JMX 接口獲取 RegionServer 運行時長信息:return: 告警信息"""jmx_port = 16030# 構建JMX查詢URL,用于獲取運行時間指標jmx_url = f'http://{regionserver_host}:{jmx_port}/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server'try:# 發送HTTP請求獲取JMX數據response = requests.get(jmx_url)# 檢查響應狀態碼response.raise_for_status()# 解析JSON響應jmx_data = response.json()# 從JMX數據中提取運行時間（單位：毫秒）region_server_start_time = jmx_data['beans'][0]['regionServerStartTime']# 獲取當前時間戳（毫秒）current_time = int(time.time() * 1000)# 計算RegionServer運行時長（毫秒）uptime = current_time - region_server_start_time# 將運行時長轉換為時分秒格式uptime_hms = convert_milliseconds_to_hms(uptime)text = f"hostname: {regionserver_host}, RegionServer uptime: {uptime_hms}"return textexcept requests.exceptions.RequestException as e:print(f'請求出錯: {e}')except (KeyError, IndexError, json.JSONDecodeError) as e:print(f'解析 JMX 數據出錯: {e}')def jmxGetHBaseAlarmStatus(regionserver_host):"""從 HBase JMX 接口獲取 RegionServer 重啟的運行時長，也就是運行時間小于10min:return: 告警信息"""jmx_port = 16030# 構建JMX查詢URL,用于獲取運行時間指標jmx_url = f'http://{regionserver_host}:{jmx_port}/jmx?qry=Hadoop:service=HBase,name=RegionServer,sub=Server'try:text = ""now_time = time.localtime(time.time())formatted_time = time.strftime('%Y-%m-%d %H:%M:%S', now_time)# 發送HTTP請求獲取JMX數據response = requests.get(jmx_url)if response.status_code == 200:# 檢查響應狀態碼# response.raise_for_status()# 解析JSON響應jmx_data = response.json()if len(jmx_data['beans'][0]) > 400:# print("---", less_than_10mins_alert_sent)# if jmx_data is not None and len(jmx_data) > 0:# 從JMX數據中提取運行時間（單位：毫秒）region_server_start_time = jmx_data['beans'][0]['regionServerStartTime']# 獲取當前時間戳（毫秒）current_time = int(time.time() * 1000)# 計算 RegionServer 運行時長（毫秒）uptime = current_time - region_server_start_time# 將運行時長轉換為時分秒格式uptime_hms = convert_milliseconds_to_hms(uptime)#current_hour = time.localtime().tm_hourglobal last_less_than_10mins_alert_hourif uptime is not None:if uptime < 10 * 60 * 1000:  # 運行時間小于 10 分鐘if last_less_than_10mins_alert_hour is None or last_less_than_10mins_alert_hour != current_hour:print("++++", last_less_than_10mins_alert_hour)text = "告警類型: reid 集群HBase 重啟告警通知 \n" + "告警信息: \n" + f"hostname: {regionserver_host} ,RegionServer uptime: {uptime_hms} " + "\n告警時間：" + formatted_time# 發出告警msg(text, dingding_url)last_less_than_10mins_alert_hour = current_hour# print(f"hostname: {regionserver_host}, RegionServer uptime: {uptime_hms}")return textexcept requests.exceptions.RequestException as e:print(f'請求出錯: {e}')except (KeyError, IndexError, json.JSONDecodeError) as e:print(f'解析 JMX 數據出錯: {e}')return Nonedef convert_milliseconds_to_hms(milliseconds):"""將毫秒轉換為時分秒的格式:param milliseconds: 毫秒數:return: 時分秒格式的字符串"""seconds = milliseconds // 1000hours = seconds // 3600seconds %= 3600minutes = seconds // 60seconds %= 60return f"{hours}小時 {minutes}分鐘 {seconds}秒."def getAllHostsHBase(alert_message=""):"""從 HBase JMX 接口獲取 RegionServer 運行時長信息:return: 正常通知信息或 None"""count = 0now_time = time.localtime(time.time())formatted_time = time.strftime('%Y-%m-%d %H:%M:%S', now_time)# 將集合轉換為列表，并進行排序sorted_hosts = sorted(list(hostArr))alert_message += "告警類型: reid 集群HBase告警通知 \n" + "告警信息: \n"for host in sorted_hosts:line_alarm = str(jmxGetHBaseStatus(host))count += 1alert_message += "\t" + str(count) + "." + line_alarm + "\n"alert_message += "\n告警時間：" + formatted_timeprint(alert_message)notify_msg(alert_message, dingding_url)def check_and_alert():"""檢查運行時長，若小于 10 分鐘且滿足條件則立即發送"""now_time = time.localtime(time.time())formatted_time = time.strftime('%Y-%m-%d %H:%M:%S', now_time)# 將集合轉換為列表，并進行排序sorted_hosts = sorted(list(hostArr))for host in sorted_hosts:alarm_str = jmxGetHBaseAlarmStatus(host)print(alarm_str)if alarm_str is not None and alarm_str != "":print("時間: ", formatted_time, "主機：", host, 'RegionServer 重啟告警發出!')else:print("時間: ", formatted_time, "主機：", host, 'RegionServer 狀 態 正 常!')def msg(text, api_url):"""釘釘告警發出 通知具體負責人:param text: 告警文本:param api_url: 釘釘URL:return: 無返回值"""json_text = {"msgtype": "text","text": {"content": text}, "at": {"atMobiles": [""]}}requests.post(api_url, json.dumps(json_text), headers=headers).contentdef notify_msg(text, api_url):"""釘釘告警發出:param text: 告警文本:param api_url: 釘釘URL:return: 無返回值"""json_text = {"msgtype": "text","text": {"content": text}, "at": {"atMobiles": [""]}}requests.post(api_url, json.dumps(json_text), headers=headers).contentdef correct_msg(text, api_url):"""釘釘告警發出, 組件正常的告警信息，不艾特告警人:param text: 告警文本:param api_url: 釘釘URL:return: 無返回值"""json_text = {"msgtype": "text","text": {"content": text}, "at": {"atMobiles": [""]}}requests.post(api_url, json.dumps(json_text), headers=headers).contentif __name__ == '__main__':# 設定整點執行常規告警任務schedule.every().hour.at(":00").do(getAllHostsHBase)while True:check_and_alert()schedule.run_pending()time.sleep(10)

釘釘告警通知：

本文來自互聯網用戶投稿，該文觀點僅代表作者本人，不代表本站立場。本站僅提供信息存儲空間服務，不擁有所有權，不承擔相關法律責任。
如若轉載，請注明出處：http://www.pswp.cn/web/93418.shtml
繁體地址，請注明出處：http://hk.pswp.cn/web/93418.shtml
英文地址，請注明出處：http://en.pswp.cn/web/93418.shtml

如若內容造成侵權/違法違規/事實不符，請聯系多彩編程網進行投訴反饋email:809451989@qq.com，一經查實，立即刪除！

相關文章

SQL 語言規范與基礎操作指南

SQL 語言規范與基礎操作指南

SQL 語言規范與基礎操作指南 SQL 作為數據庫操作的核心語言&#xff0c;遵循規范的語法和書寫習慣不僅能提高代碼可讀性&#xff0c;還能減少錯誤。本文整理了 SQL 的基礎規則、書寫規范及常用操作&#xff0c;適合初學者快速上手。 一、SQL 基本規則 1. 書寫格式 SQL 語句可寫…
閱讀更多...
產業園IBMS智能化集成系統功能有哪些？

產業園IBMS智能化集成系統功能有哪些？

產業園 IBMS&#xff08;建筑集成管理系統&#xff09;智能化集成系統是針對產業園 “多業態、多系統、多租戶” 特點設計的全局管理平臺&#xff0c;通過整合樓宇自控、安防、消防、能源、停車、租戶服務等子系統&#xff0c;實現 “集中監控、協同聯動、數據驅動、靈活服務”…
閱讀更多...
線性代數之兩個宇宙文明關于距離的對話

線性代數之兩個宇宙文明關于距離的對話

矢量的客觀性和主觀性宇宙中飄過來一個自由矢量&#xff0c;全世界的人都可以看到&#xff0c;大家都在想&#xff0c;怎么描述它呢&#xff0c;總不能指著它說“那個矢量”吧。數學家很聰明&#xff0c;于是建立了一個坐標系&#xff0c;這個矢量投影到坐標系下&#xff0c;就…
閱讀更多...
Camx-Tuning參數加載流程分析

Camx-Tuning參數加載流程分析

調用時序圖 一、效果參數在開機時加載 CreateTuningDataManager邏輯分析 1.從xxx_module.xml獲取sensor名稱和效果參數名稱&#xff0c; 比如效果參數名稱為&#xff1a;xtc_tsp_sc520cs那么效果庫的完整名稱就是&#xff1a;com.qti.tuned.xtc_tsp_sc520cs.bin 2.優先從/data/…
閱讀更多...
《P4180 [BJWC2010] 嚴格次小生成樹》

《P4180 [BJWC2010] 嚴格次小生成樹》

題目描述小 C 最近學了很多最小生成樹的算法&#xff0c;Prim 算法、Kruskal 算法、消圈算法等等。正當小 C 洋洋得意之時&#xff0c;小 P 又來潑小 C 冷水了。小 P 說&#xff0c;讓小 C 求出一個無向圖的次小生成樹&#xff0c;而且這個次小生成樹還得是嚴格次小的&#xff…
閱讀更多...
Transformer淺說

Transformer淺說

rag系列文章目錄 文章目錄rag系列文章目錄前言一、簡介二、注意力機制三、架構優勢四、模型加速總結前言 近兩年大模型爆火&#xff0c;大模型的背后是transformer架構&#xff0c;transformer成為家喻戶曉的詞&#xff0c;人人都知道它&#xff0c;但是想要詳細講清楚&#x…
閱讀更多...
后臺管理系統-3-vue3之左側菜單欄和頭部導航欄的靜態搭建

后臺管理系統-3-vue3之左側菜單欄和頭部導航欄的靜態搭建

文章目錄1 CommonAside組件(靜態搭建)1.1 Menu菜單1.2 準備菜單數據1.3 循環渲染菜單1.3.1 el-menu結構1.3.2 動態渲染圖標1.4 樣式設計1.5 整體代碼(CommonAside.vue)2 CommonHeader組件(靜態搭建)2.1 準備圖片URL數據2.2 頁面布局2.3 樣式設計2.4 整體代碼(CommonHeader.vue)…
閱讀更多...
VS Code配置MinGW64編譯非線性優化庫NLopt

VS Code配置MinGW64編譯非線性優化庫NLopt

VS Code用MinGW64編譯C代碼安裝MSYS2軟件并配置非線性優化庫NLopt和測試引用庫代碼的完整具體步驟。 1. 安裝MSYS2 下載安裝程序&#xff1a; 訪問 MSYS2官網下載 msys2-x86_64-xxxx.exe 并運行 完成安裝&#xff1a; 默認安裝路徑&#xff1a;C:\msys64安裝完成后&#xff0c…
閱讀更多...
C#通過TCP_IP與PLC通信

C#通過TCP_IP與PLC通信

C#通過TCP/IP與PLC通信 本文將全面介紹如何使用C#通過TCP/IP協議與各種PLC進行通信&#xff0c;包括西門子、羅克韋爾、三菱等主流品牌PLC的連接方法。 一、PLC通信基礎 PLC通信協議概覽協議類型適用品牌特點Modbus TCP通用協議簡單易用&#xff0c;廣泛支持Siemens S7西門子PL…
閱讀更多...
Java 學習筆記（基礎篇3）

Java 學習筆記（基礎篇3）

1. 數組&#xff1a;① 靜態初始化&#xff1a;(1) 格式&#xff1a;int[] arr {1, 2, 3};② 遍歷/* 格式&#xff1a; 數組名.length */ for(int i 0; i < arr.length; i){//在循環的過程中&#xff0c;i依次表示數組中的每一個索引sout(arr[i]);//就可以把數組里面的每一…
閱讀更多...
知識點匯總linuxC高級-3 shell腳本編程

知識點匯總linuxC高級-3 shell腳本編程

shell腳本編程shell ---> 解析器&#xff1a;sh csh ksh bashshell命令 ---> shell解析的命令shell腳本 --> shell命令的有序集合shell腳本編程&#xff1a;將shell命令結合按照一定邏輯集合到一起&#xff0c;寫到一個 .sh 文件&#xff0c;去實現一個或多個功能&…
閱讀更多...
【C++學習篇】：基礎

【C++學習篇】：基礎

文章目錄前言1. main() 函數2. 變量賦值3. cin和cout的一些細節4. 基本類型運算5. 內存占用6. 引用7. 常量前言 C 語法的學習整理&#xff0c;作為個人總結使用。 1. main() 函數 #include <iostream> //使用輸入輸出流庫&#xff08;cin&#xff0c;cout&#xff09;…
閱讀更多...
使用nginx反向代理kkfile

使用nginx反向代理kkfile

這篇說一下我解決的思路和方式哈&#xff0c;不一定適用于大家&#xff0c;可以做個參考比如我們的系統服務是http://10.63.25.35:80&#xff0c;而我們的文件服務是在10.63.25.37:8012上&#xff0c;正常不使用代理的話&#xff0c;我們前端調用后端接口&#xff0c;后端調用k…
閱讀更多...
【低成本擴容】動態擴容實戰指南

【低成本擴容】動態擴容實戰指南

面對擴容操作時&#xff0c;下面這種操作是否也會迷惑你&#xff1f;下面來為大家解惑~size_t newcapacity 2*_capacity > (_size len)?2*_capacity:(_sizelen); //len為即將插入的字符串有效字符個數//_size為當前字符串有效字符個數//_capacity為當前容量大小//newcapa…
閱讀更多...
Product Hunt 每日熱榜 | 2025-08-14

Product Hunt 每日熱榜 | 2025-08-14

1. Autumn 標語&#xff1a;為AI初創公司簡化的Stripe服務 介紹&#xff1a;Autumn幫助AI初創公司通過只需三個API調用來定價、計量和控制使用情況。基于Stripe搭建&#xff0c;它可以在一個地方管理訂閱、使用情況和訪問權限。無需復雜的webhooks或后端邏輯&#xff0c;非常…
閱讀更多...
Scrapy + Django爬蟲可視化項目實戰（二） 詳細版

Scrapy + Django爬蟲可視化項目實戰（二） 詳細版

系列文章 Scrapy + Django爬蟲可視化項目實戰（一）_django scrapy-CSDN博客 實現技術 Scrapy Django Echarts 引言 可視化部分需要讀者具備一定的Django基礎！！！ 上一個文章我們已經實現了爬取景點的數據，那么接下來就是根據爬取到的數據進行可視化 一、環境搭建 (一) 創…
閱讀更多...
選擇式與生成式超啟發算法總結

選擇式與生成式超啟發算法總結

這里寫目錄標題Selection HHGeneration HHGPHH示例存在大量針對特定問題設計的啟發式算法&#xff0c;近年來學術界提出了一個關鍵問題&#xff1a;如何選擇最合適的啟發式方法。這一問題推動了超啟發式&#xff08;hyper-heuristic&#xff09;方法的研究發展。超啟發式是一種…
閱讀更多...
NetBIOS 設置

NetBIOS 設置

在 Windows 系統中，WINS (Windows Internet Name Service) 和 NetBIOS 緊密相關，主要用于 NetBIOS 名稱解析（將計算機名轉換為 IP 地址）。WINS 是一個動態數據庫，類似于 DNS，但專門用于 NetBIOS 名稱解析，適用于早期 Windows 網絡（如 Windows NT/2000/XP）。 1. 查看 N…
閱讀更多...
vue2 + SimpleMindMap 制作思維導圖

vue2 + SimpleMindMap 制作思維導圖

vue2 SimpleMindMap 制作思維導圖 該代碼包含SimpleMindMap已知的所有功能&#xff0c;有需要的小伙伴可自行copy&#xff0c;框架使用el-ementui。其中有些圖標是阿里巴巴矢量圖的圖片&#xff0c;可自行進行替換。保姆級教程 以下是vue文件&#xff1a; <template><…
閱讀更多...
Discord x Pulsar: 使用 Pulsar、Flink 和 Iceberg 搭建流式機器學習平臺

Discord x Pulsar: 使用 Pulsar、Flink 和 Iceberg 搭建流式機器學習平臺

本文整理自 Discord 機器學習工程師 David Christle 在 Pulsar Summit NA 上的演講內容&#xff0c;一起來看 Discord 是如何基于 Pulsar 實現兼顧安全和個性化功能的實時流式機器學習平臺的&#xff5e;1. 背景Discord 是一個實時?視頻通信平臺&#xff0c;?持?本/語?/視頻…
閱讀更多...
最新文章

Copyright @ 2022~2023