Skip to content

8.md

Latest commit

 

History

History
347 lines (281 loc) · 12.5 KB

8.md

File metadata and controls

347 lines (281 loc) · 12.5 KB
Raw

eBPF 跟踪网络包的踪迹

引出问题

丢弃数据包在网络中是无法完全避免的一个情况. 数据包无法到达最终目的地的原因可能有很多种. 这可能导致数据传输失败、网络性能下降、延迟增加，甚至应用程序出错.

网络丢包可能由多种因素引起，主要包括：

  • 网络拥塞：当网络流量超过链路或设备的处理能力时.

  • 硬件故障：如网卡、交换机或路由器出现问题.

  • 软件问题：操作系统、驱动程序或应用程序的 bug.

  • 信号干扰：无线网络中的电磁干扰.

  • 网络配置错误：如 MTU 不匹配、路由配置错误等.

  • 安全设置：防火墙规则过于严格.

先把关注点聚焦在 Linux 服务器上，分析 Linux 协议栈的丢包情况.

Linux 协议栈的丢包情况主要包括以下几种：

  • 网络接口缓冲区溢出

  • CPU 负载过高，无法及时处理网络数据包

  • 内核参数配置不当

  • iptables 规则不合理

  • 网卡驱动问题

  • 网络拥塞控制算法不适合当前网络环境

当然现在有一些工具比如 cilium/pwru、dropwatch 来跟踪和分析丢包的原因，这里就主要通过 eBPF 来分析丢包的原因，后面有就会说说上述提到的工具.那么最终问题来了:

网络包到底去哪了呢?

kfree_skb_reason

free_skb 函数在早期的内核版本中广泛使用，这是最基本的 sk_buff 释放函数. 它简单地释放与 sk_buff 相关的所有内存，不提供任何关于为什么释放的信息.

void kfree_skb(struct sk_buff *skb)
{
    // 函数实现...
}

Linux 内核协议栈中有很多方法可以让数据包消亡，以至于管理员都很难知道为什么某个数据包会被丢弃. 因此，在用户抱怨丢包率高的时候，这么多种可能也导致很难确定如何解决用户的问题. 从 5.17 版本开始，内核得到了一些改进过的工具，应该可以告诉人们为什么内核会决定要将数据包抛弃掉. 腾讯的 Menglong Dong 为 5.17 版贡献了一些可以有助于提供更好方法的基础结构. 在 5.17 中，这个函数变成了：

void kfree_skb_reason(struct sk_buff *skb, enum skb_drop_reason reason)
{
    // 函数实现...
}

第二个参数提供了一个枚举值，可以告诉用户为什么数据包被丢弃. 这个枚举值是一个新的 skb_drop_reason 类型，定义在 include/net/dropreason-core.h 中：

enum skb_drop_reason {
 /**
  * @SKB_NOT_DROPPED_YET: skb is not dropped yet (used for no-drop case)
  */
 SKB_NOT_DROPPED_YET = 0,
 /** @SKB_CONSUMED: packet has been consumed */
 SKB_CONSUMED,
 /** @SKB_DROP_REASON_NOT_SPECIFIED: drop reason is not specified */
 SKB_DROP_REASON_NOT_SPECIFIED,
 /**
  * @SKB_DROP_REASON_NO_SOCKET: no valid socket that can be used.
  * Reason could be one of three cases:
  * 1) no established/listening socket found during lookup process
  * 2) no valid request socket during 3WHS process
  * 3) no valid child socket during 3WHS process
  */
 SKB_DROP_REASON_NO_SOCKET,
 /** @SKB_DROP_REASON_PKT_TOO_SMALL: packet size is too small */
 SKB_DROP_REASON_PKT_TOO_SMALL,
 /** @SKB_DROP_REASON_TCP_CSUM: TCP checksum error */
 SKB_DROP_REASON_TCP_CSUM,
 /** @SKB_DROP_REASON_SOCKET_FILTER: dropped by socket filter */
 SKB_DROP_REASON_SOCKET_FILTER,
 /** @SKB_DROP_REASON_UDP_CSUM: UDP checksum error */
 SKB_DROP_REASON_UDP_CSUM,
 /** @SKB_DROP_REASON_NETFILTER_DROP: dropped by netfilter */
 SKB_DROP_REASON_NETFILTER_DROP,

    ...

随着 Linux 版本的不断更新，内核中越来越多的 kfree_skb 被替换成了kfree_skb_reason.

所以 kfree_skb_reason 是分析丢包的一个重要的入口，我们可以通过 eBPF 来跟踪这个函数，从而分析丢包的原因.

使用bpftrace跟踪 kfree_skb_reason

使用 bpftrace 可以快速验证我们的想法，我们要关注的 UDP 和 TCP 的丢包情况，丢包的时候把五元组信息和丢包原因打印出来.

首先我们先包含必要的一些头文件，并且把表格行首信息打印出来，包括线程 ID、进程 ID、进程名、协议、丢包原因、源地址端口和目的地址端口.

#!/usr/bin/env bpftrace

#include <linux/skbuff.h>
#include <linux/ip.h>
#include <net/sock.h>

BEGIN
{
  printf("Tracing Packet drops. Hit Ctrl-C to end.\n");
  printf("%-8s %-8s %-16s %-10s %-16s %-21s %-21s\n", "TIME", "PID", "COMM", "IP_PROTO", "REASON", "SADDR:SPORT", "DADDR:DPORT");
}

然后我们定义一个kfree_skb_reason的 probe，这个 probe 会在kfree_skb_reason函数被调用的时候触发，我们在这个 probe 中获取五元组信息和丢包原因，然后打印出来.

首先要看一下自己的内核版本

uname -r
6.8.0-49-generic

然后开始针对这个进行映射:

#!/usr/bin/env bpftrace

#include <linux/skbuff.h>
#include <linux/ip.h>
#include <net/sock.h>

BEGIN
{
  @reasons[0] = "SKB_DROP_REASON_NOT_SPECIFIED";
  @reasons[1] = "SKB_DROP_REASON_NO_SOCKET";
  @reasons[2] = "SKB_DROP_REASON_PKT_TOO_SMALL";
  @reasons[3] = "SKB_DROP_REASON_TCP_CSUM";
  @reasons[4] = "SKB_DROP_REASON_SOCKET_FILTER";
  @reasons[5] = "SKB_DROP_REASON_UDP_CSUM";
  @reasons[6] = "SKB_DROP_REASON_NETFILTER_DROP";
  @reasons[7] = "SKB_DROP_REASON_OTHERHOST";
  @reasons[8] = "SKB_DROP_REASON_IP_CSUM";
  @reasons[9] = "SKB_DROP_REASON_IP_INHDR";
  @reasons[10] = "SKB_DROP_REASON_IP_RPFILTER";
  @reasons[11] = "SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST";
  @reasons[12] = "SKB_DROP_REASON_XFRM_POLICY";
  @reasons[13] = "SKB_DROP_REASON_IP_NOPROTO";
  @reasons[14] = "SKB_DROP_REASON_SOCKET_RCVBUFF";
  @reasons[15] = "SKB_DROP_REASON_PROTO_MEM";
  @reasons[16] = "SKB_DROP_REASON_TCP_MD5NOTFOUND";
  @reasons[17] = "SKB_DROP_REASON_TCP_MD5UNEXPECTED";
  @reasons[18] = "SKB_DROP_REASON_TCP_MD5FAILURE";
  @reasons[19] = "SKB_DROP_REASON_SOCKET_BACKLOG";
  @reasons[20] = "SKB_DROP_REASON_TCP_FLAGS";
  @reasons[21] = "SKB_DROP_REASON_TCP_ZEROWINDOW";
  @reasons[22] = "SKB_DROP_REASON_TCP_OLD_DATA";
  @reasons[23] = "SKB_DROP_REASON_TCP_OVERWINDOW";
  @reasons[24] = "SKB_DROP_REASON_TCP_OFOMERGE";
  @reasons[25] = "SKB_DROP_REASON_TCP_RFC7323_PAWS";
  @reasons[26] = "SKB_DROP_REASON_TCP_INVALID_SEQUENCE";
  @reasons[27] = "SKB_DROP_REASON_TCP_RESET";
  @reasons[28] = "SKB_DROP_REASON_TCP_INVALID_SYN";
  @reasons[29] = "SKB_DROP_REASON_TCP_CLOSE";
  @reasons[30] = "SKB_DROP_REASON_TCP_FASTOPEN_LISTEN_QUEUE_FULL";
  @reasons[31] = "SKB_DROP_REASON_TCP_FASTOPEN_COOKIE_INVALID";
  @reasons[32] = "SKB_DROP_REASON_TCP_FASTOPEN_SYN_NODATA";
  @reasons[33] = "SKB_DROP_REASON_TCP_TOO_SHORT";
  @reasons[34] = "SKB_DROP_REASON_IP_OUTNOROUTES";
  @reasons[35] = "SKB_DROP_REASON_BPF_CGROUP_EGRESS";
  @reasons[36] = "SKB_DROP_REASON_IPV6DISABLED";
  @reasons[37] = "SKB_DROP_REASON_NEIGH_CREATEFAIL";
  @reasons[38] = "SKB_DROP_REASON_NEIGH_FAILED";
  @reasons[39] = "SKB_DROP_REASON_NEIGH_QUEUEFULL";
  @reasons[40] = "SKB_DROP_REASON_NEIGH_DEAD";
  @reasons[41] = "SKB_DROP_REASON_TC_EGRESS";
  @reasons[42] = "SKB_DROP_REASON_QDISC_DROP";
  @reasons[43] = "SKB_DROP_REASON_CPU_BACKLOG";
  @reasons[44] = "SKB_DROP_REASON_XDP";
  @reasons[45] = "SKB_DROP_REASON_TC_INGRESS";
  @reasons[46] = "SKB_DROP_REASON_UNHANDLED_PROTO";
  @reasons[47] = "SKB_DROP_REASON_SKB_CSUM";
  @reasons[48] = "SKB_DROP_REASON_SKB_GSO_SEG";
  @reasons[49] = "SKB_DROP_REASON_SKB_UCOPY_FAULT";
  @reasons[50] = "SKB_DROP_REASON_DEV_HDR";
  @reasons[51] = "SKB_DROP_REASON_DEV_READY";
  @reasons[52] = "SKB_DROP_REASON_FULL_RING";
  @reasons[53] = "SKB_DROP_REASON_NOMEM";
  @reasons[54] = "SKB_DROP_REASON_HDR_TRUNC";
  @reasons[55] = "SKB_DROP_REASON_TAP_FILTER";
  @reasons[56] = "SKB_DROP_REASON_TAP_TXFILTER";
  @reasons[57] = "SKB_DROP_REASON_ICMP_CSUM";
  @reasons[58] = "SKB_DROP_REASON_INVALID_PROTO";
  @reasons[59] = "SKB_DROP_REASON_IP_INADDRERRORS";
  @reasons[60] = "SKB_DROP_REASON_IP_INNOROUTES";
  @reasons[61] = "SKB_DROP_REASON_PKT_TOO_BIG";
  @reasons[62] = "SKB_DROP_REASON_MAX";
  @protocols[1] = "ICMP";
  @protocols[2] = "IGMP";
  @protocols[4] = "IP";
  @protocols[6] = "TCP";
  @protocols[17] = "UDP";
  @protocols[41] = "IPv6";
  @protocols[89] = "OSPF";
  @protocols[132] = "SCTP";
  printf("Tracing Packet drops. Hit Ctrl-C to end.\n");
  printf("%-8s %-8s %-16s %-10s %-16s %-21s %-21s\n", "TIME", "PID", "COMM", "IP_PROTO", "REASON", "SADDR:SPORT", "DADDR:DPORT");
}

kprobe:kfree_skb_reason
{
    // 获取当前的进程ID和释放原因
    $pid = pid;
    // 获取第二个参数，也就是包丢弃的原因
    $reason = arg1;

    // 排除特定的 reason
    if ($reason <= 2 || $reason >= 42) {
        // 跳过这些 PID
        return;
    }

    // 将 reason 转换为字符串，这些省略了很堵枚举类型，你根据Linux内核的版本自行添加即可
    // 使用映射获取原因字符串
    $reason_str = @reasons[$reason];
    if ($reason_str == "") {
        $reason_str = "UNKNOWN";
    }

    // 获取skb指针
    $skb = (struct sk_buff *)arg0;
    $sk = ((struct sock *) $skb->sk);
    $inet_family = $sk->__sk_common.skc_family;

    // 获取IP层的源地址、目的地址、源端口和目的端口
    if ($inet_family == AF_INET) {
      $daddr = ntop($sk->__sk_common.skc_daddr);
      $saddr = ntop($sk->__sk_common.skc_rcv_saddr);
    } else {
      $daddr = ntop($sk->__sk_common.skc_v6_daddr.in6_u.u6_addr8);
      $saddr = ntop($sk->__sk_common.skc_v6_rcv_saddr.in6_u.u6_addr8);
    }
    $lport = $sk->__sk_common.skc_num;
    $dport = $sk->__sk_common.skc_dport;

    // 目的端口是大端序，需要转成小端序
    $dport = bswap($dport);


    // 提取协议信息
    $ipheader = ((struct iphdr *) ($skb->head + $skb->network_header));
    $version = ($ipheader->version) >>4;

    // 将协议号转换为字符串
    $protocol_str = @protocols[$ipheader->protocol];
    if ($protocol_str == "") {
        $protocol_str = "UNKNOWN";
    }


    // 输出时间和各字段的值
    time("%H:%M:%S ");
    printf("%-8d %-16s %-10s %-16s ", pid, comm, $protocol_str, $reason_str);
    printf("%15s:%-6d %15s:%-6d\n", $saddr, $lport, $daddr, $dport);

    // 打印出当前的栈信息，方便查看丢包的栈调用情况
    printf("%s\n", kstack);
}

总结一下，这个脚本做了以下几件事情：

  • 在kfree_skb_reason函数被调用的时候，获取五元组信息和丢包原因

  • 把五元组信息和丢包原因打印出来

  • 打印出当前的栈信息，方便查看丢包的栈调用情况

最后我们把这个脚本保存为 net_package_drop.bt，然后使用 bpftrace 来运行这个脚本：

sudo bpftrace -v net_package_drop.bt

在另外一个窗口，使用 nc 尝试连接一个不存在的端口，我们可以看到 bpftrace 输出了丢包的原因和五元组信息：

可以看到 net_package_drop.bt 会把这个丢弃的五元组以及堆栈信息打印出来，这样我们就可以通过这个信息来分析丢包的原因了：

        __tcp_transmit_skb+1460
        tcp_connect+1062
        tcp_v6_connect+1398
        __inet_stream_connect+246
        inet_stream_connect+59
        __sys_connect_file+107
        __sys_connect+181
        __x64_sys_connect+24
        x64_sys_call+9499
        do_syscall_64+127
        entry_SYSCALL_64_after_hwframe+120

20:40:18 4006     Chrome_DevTools  TCP        SKB_DROP_REASON_TCP_CSUM              :::0                   :::0     

        kfree_skb_reason+1
        ip_protocol_deliver_rcu+60
        ip_local_deliver_finish+119
        ip_local_deliver+110
        ip_rcv+394
        __netif_receive_skb_one_core+145
        __netif_receive_skb+21
        process_backlog+142
        __napi_poll+48
        net_rx_action+385
        handle_softirqs+216
        __do_softirq+16
        do_softirq.part.0+65
        __local_bh_enable_ip+114
        __dev_queue_xmit+648
        neigh_hh_output+147
        ip_finish_output2+500
        __ip_finish_output+182
        ip_finish_output+41
        ip_output+95
        ip_local_out+97
        __ip_queue_xmit+398
        ip_queue_xmit+21
        __tcp_transmit_skb+2614
        tcp_connect+1062
        tcp_v4_connect+1117
        __inet_stream_connect+246
        inet_stream_connect+59
        __sys_connect_file+107
        __sys_connect+181
        __x64_sys_connect+24
        x64_sys_call+9499
        do_syscall_64+127
        entry_SYSCALL_64_after_hwframe+120

20:40:19 4006     Chrome_DevTools  UNKNOWN    SKB_DROP_REASON_TCP_CSUM              :::0                   :::0