問題背景:
為了分析udp數據通信中端到端的延遲,我們需要對整個通信鏈路的每個階段進行監控,找出延遲最長的階段.

udp接收端有2個主要路徑
1.數據包到達本機后，由軟中斷處理程序將數據包接收并放入udp socket的接收緩沖區

數據接收流程
2. 應用程序調用recvmsg等api將數據從socket緩沖區讀出

應用程序讀取數據流程
2和1之間可能由于調度等造成延遲，我們寫一個bcc程序對指定接收端口和延遲大于某個值的情況進行監控

bcc程序原理
我們在流程1放入udp緩沖區時(udp_unicast_rcv_skb),記錄此skb的時間

然后在流程2讀取udp緩沖區時(__skb_recv_udp)時取出1中記錄的skb時間，并與當前時間做差值得到延遲.

#!/usr/bin/python3
# @lint-avoid-python-3-compatibility-imports
#
# udplatency    Trace long udp recvmsg delays.
#               For Linux, uses BCC, eBPF.
#
# This script traces high delays between skb being
# ready to in recv queue and them recvmsg on CPU after that.
#
# USAGE: udplatency [-d dport] [-l lat]import argparse
import ctypes as ct
from time import strftime
from bcc import BPFbpf_text = '''
# include <linux/ip.h>
# include <linux/netfilter.h>
# include <net/ip.h>
# include <uapi/linux/bpf.h>struct data_t {u64 ts;u64 lat;
};BPF_PERF_OUTPUT(events);
BPF_HASH(recv_lat, struct sk_buff*);int kprobe__udp_unicast_rcv_skb(struct pt_regs *ctx, struct sock *sk, struct sk_buff* skb)
{struct udphdr *udp_hdr = (struct udphdr *)(skb->head + skb->transport_header);u16 dst_port = bpf_ntohs(udp_hdr->dest);if (dst_port == DST_PORT) {u64 ts = bpf_ktime_get_ns();recv_lat.update(&skb, &ts);}return 0;
};int kretprobe____skb_recv_udp(struct pt_regs *ctx)
{struct sk_buff* skb = (struct sk_buff*)PT_REGS_RC(ctx);struct udphdr *udp_hdr = (struct udphdr *)(skb->head + skb->transport_header);u16 dst_port = bpf_ntohs(udp_hdr->dest);if (dst_port == DST_PORT) {struct data_t data = {};u64 *tsp = recv_lat.lookup(&skb);if (tsp != 0) {Home = bpf_ktime_get_ns() - *tsp;}recv_lat.delete(&skb);if (data.lat >= LAT_NS) {bpf_probe_read_kernel(&(data.ts), sizeof(*tsp), tsp);events.perf_submit(ctx, &data, sizeof(data));}}return 0;
}
'''class EventData(ct.Structure):_fields_ = [("ts", ct.c_ulonglong),("lat", ct.c_ulonglong)]def print_event(cpu, data, size):event = ct.cast(data, ct.POINTER(EventData)).contentsprint("%-8s ts:%d lat: %dms\n" % (strftime("%H:%M:%S"), event.ts, event.lat / 3000000))parser = argparse.ArgumentParser(description="Track udp recv latency")
parser.add_argument("-d", "--dport", type=int, required=True,help="udp dst port")
parser.add_argument("-l", "--lat", type=int,help="report latency > ns, default is 3000000")
args = parser.parse_args()lat_ns = 3000000if args.lat:lat_ns = args.latbpf_text = bpf_text.replace('DST_PORT', str(args.dport))
bpf_text = bpf_text.replace('LAT_NS', str(lat_ns))# initialize BPF
b = BPF(text=bpf_text)b["events"].open_perf_buffer(print_event)
while True:try:b.perf_buffer_poll()except KeyboardInterrupt:exit()