上篇文章:
Spring Cloud系列—SkyWalking鏈路追蹤https://blog.csdn.net/sniper_fandc/article/details/149948321?fromshare=blogdetail&sharetype=blogdetail&sharerId=149948321&sharerefer=PC&sharesource=sniper_fandc&sharefrom=from_link
目錄
1 告警規則
1.1 告警規則配置項
1.2 Webhook郵箱告警
1.2.1 引入依賴
1.2.2 添加配置項
1.2.3 接口開發
1.2.4 配置Webhook
1.2.5 重啟服務和SkyWalking
2 Webhook接入飛書
1 告警規則
1.1 告警規則配置項
????????當發生異常信息時,比如接口訪問非常慢或超時,請求成功率很低,就需要SkyWalking來通知開發人員和運維人員來及時排查問題:
????????在SkyWalking安裝目錄apache-skywalking-apm-bin\config的alarm-settings.yml文件中,存在默認的告警規則:
# Sample alarm rules.rules:# Rule unique name, must be ended with `_rule`.service_resp_time_rule:# A MQE expression, the result type must be `SINGLE_VALUE` and the root operation of the expression must be a Compare Operation# which provides `1`(true) or `0`(false) result. When the result is `1`(true), the alarm will be triggered.expression: sum(service_resp_time > 1000) >= 3period: 10silence-period: 5message: Response time of service {name} is more than 1000ms in 3 minutes of last 10 minutes.# ?service_resp_time_rule:# ???expression: avg(service_resp_time) > 1000# ???period: 10# ???silence-period: 5# ???message: Avg response time of service {name} is more than 1000ms in last 10 minutes.service_sla_rule:expression: sum(service_sla < 8000) >= 2# The length of time to evaluate the metricsperiod: 10# How many times of checks, the alarm keeps silence after alarm triggered, default as same as period.silence-period: 3message: Successful rate of service {name} is lower than 80% in 2 minutes of last 10 minutesservice_resp_time_percentile_rule:expression: sum(service_percentile{p='50,75,90,95,99'} > 1000) >= 3period: 10silence-period: 5message: Percentile response time of service {name} alarm in 3 minutes of last 10 minutes, due to more than one condition of p50 > 1000, p75 > 1000, p90 > 1000, p95 > 1000, p99 > 1000service_instance_resp_time_rule:expression: sum(service_instance_resp_time > 1000) >= 2period: 10silence-period: 5message: Response time of service instance {name} is more than 1000ms in 2 minutes of last 10 minutesdatabase_access_resp_time_rule:expression: sum(database_access_resp_time > 1000) >= 2period: 10message: Response time of database access {name} is more than 1000ms in 2 minutes of last 10 minutesendpoint_relation_resp_time_rule:expression: sum(endpoint_relation_resp_time > 1000) >= 2period: 10message: Response time of endpoint relation {name} is more than 1000ms in 2 minutes of last 10 minutes# ?Active endpoint related metrics alarm will cost more memory than service and service instance metrics alarm.# ?Because the number of endpoint is much more than service and instance.## ?endpoint_resp_time_rule:# ???expression: sum(endpoint_resp_time > 1000) >= 2# ???period: 10# ???silence-period: 5# ???message: Response time of endpoint {name} is more than 1000ms in 2 minutes of last 10 minutes#hooks:# ?webhook:# ???default:# ?????is-default: true# ?????urls:# ???????- http://127.0.0.1/notify/# ???????- http://127.0.0.1/go-wechat/
????????告警規則的定義必須以`_rule`結尾,其中:
????????expression是告警表達式,結果為1時觸發告警;
????????period是告警周期(minute),在該時間范圍內滿足expression的觸發高級;
????????silence-period是靜默時間(minute),觸發告警后,靜默時間內不再觸發;
????????message是告警信息。
????????比如service_resp_time_rule規則,表示某個服務的響應時間在最近10分鐘的3分鐘內持續超過1000毫秒時觸發告警。
1.2 Webhook郵箱告警
????????Webhook是一種允許應用程序向外部系統實時推送事件或數據的機制,通常通過HTTP回調實現,從而實現跨系統自動化的信息傳遞。核心特性:
????????事件驅動:當預設條件觸發時(如告警觸發、數據更新),主動向目標URL發送HTTP請求(通常為POST)。
????????輕量級集成:接收方只需提供一個可訪問的HTTP端點即可接收數據,無需輪詢查詢。
????????靈活擴展:適用于告警通知、流程觸發、數據同步等場景。
1.2.1 引入依賴
????<dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-web</artifactId></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-mail</artifactId></dependency></dependencies>
????????spring-boot-starter-mail是主要進行郵件的依賴。
1.2.2 添加配置項
server:port: 8084logging:pattern:dateformat: HH:mm:ss:SSSspring:mail:# 指定郵件服務器地址host: smtp.qq.com# 登錄賬戶username: "發件人郵箱賬號"# 授權碼password: "授權碼"# 端口port: 465# 默認編碼default-encoding: UTF-8# 使用的協議protocol: smtps# 其他的屬性properties:# 默認屬性"mail.smtp.connectiontimeout": 5000"mail.smtp.timeout": 3000"mail.smtp.writetimeout": 5000"mail.smtp.auth": true"mail.smtp.starttls.enable": true"mail.smtp.starttls.required": true# 自定義屬性"personal": "告警系統""subject": "訂單系統告警"
????????需要在QQ郵箱設置中開啟SMTP服務,并且記得把配置文件中的spring.mail.username和password替換為自己的郵箱賬號和授權碼。
????????授權碼開啟方式可以查看QQ郵箱官網的幫助文檔:https://service.mail.qq.com/detail/0/75
1.2.3 接口開發
????????SkyWalking告警消息接收實體類:
@Datapublic class AlarmMessage {private int scopeId;private String scope;private String name;private String id0;private String id1;private String ruleName;private String alarmMessage;private List<Tag> tags;private long startTime;private transient int period;private Set<String> hooks = new HashSet<>();private String expression;@Datapublic static class Tag {private String key;private String value;}}
????????郵件發送配置類:
@Slf4j@Configurationpublic class Mail {@Autowired//讀取spring.mail配置并注冊成MailProperties對象private MailProperties mailProperties;@Autowiredprivate JavaMailSender javaMailSender;public void send(String to,String content) {try {// 創建一個郵件消息MimeMessage message = javaMailSender.createMimeMessage();// 創建MimeMessageHelperMimeMessageHelper helper = new MimeMessageHelper(message, false);// 發件人郵箱和名稱String personal = Optional.ofNullable(mailProperties.getProperties().get("personal")).orElse(mailProperties.getUsername());helper.setFrom(mailProperties.getUsername(), personal);// 收件人郵箱helper.setTo(to);// 郵件標題helper.setSubject(mailProperties.getProperties().getOrDefault("subject","告警通知"));// 郵件正文,第二個參數表示是否是HTML正文helper.setText(content, true);// 發送javaMailSender.send(message);} catch (Exception e) {log.error("郵件發送失敗, e:" + e);}}}
????????控制層接口:
@Slf4j@RequestMapping("/alarm")@RestControllerpublic class AlarmController {@Autowiredprivate Mail mail;@RequestMapping("/handler")public String handler(@RequestBody List<AlarmMessage> alarmMessages) {log.info("收到報警, alarmMessages:{}", alarmMessages);mail.send("收件人郵箱",buildMessage(alarmMessages));return "接收報警成功";}private String buildMessage(List<AlarmMessage> alarmMessages) {StringBuilder builder = new StringBuilder();builder.append("系統告警: <br/>");for (AlarmMessage alarmMessage : alarmMessages) {builder.append("scopeId: ").append(alarmMessage.getScopeId()).append("<br/> scope: ").append(alarmMessage.getScope()).append("<br/> 目標 Scope 的實體名稱: ").append(alarmMessage.getName()).append("<br/> Scope 實體的 ID: ").append(alarmMessage.getId0()).append("<br/> 告警規則名稱: ").append(alarmMessage.getRuleName()).append("<br/> 告警消息內容: ").append(alarmMessage.getAlarmMessage()).append("<br/>告警時間: ").append(alarmMessage.getStartTime()).append("<br/><br/>---------------");}return builder.toString();}}
1.2.4 配置Webhook
????????主要是配置apache-skywalking-apm-bin\config的alarm-settings.yml文件,配置告警向URL進行通知,這里就是配置向alarm-service服務進行通知,然后由alarm-service服務將告警信息處理并發送郵箱:
hooks:webhook:default:is-default: trueurls:- http://127.0.0.1:8084/alarm/handler
1.2.5 重啟服務和SkyWalking
????????由于開啟分布式事務,因此創建訂單操作比較慢,在郵件中就會出現告警信息。
????????注意:該信息出現可能比較慢,因為告警規則統計周期默認是10分鐘,加上信息處理等就更慢了。
2 Webhook接入飛書
????????Webhook還可以接入企業微信、飛書、釘釘等應用,從而讓開發和運維人員更及時接收告警信息。在飛書任意一個群組,點擊右上角,添加機器人:
????????點擊自定義機器人,然后配置機器人信息,點擊添加:
????????在打開的界面復制Webhook地址和簽名校驗碼(如果開啟設置),修改SkyWalking的配置文件:
hooks:feishu:default:is-default: truetext-template: |{"msg_type":"text","content": {"text": "Apache SkyWalking Alarm: \n %s."}}webhooks:- url: 飛書獲取secret: 飛書獲取
????????然后重啟SkyWalking,測試觀察飛書機器人推送告警消息:
????????可以發現,告警信息被成功推送到飛書。其它應用接入也同理,具體可看應用的開發者文檔。