Features used in Calico

[fasaxc]

As discussed on the triage call, starting a thread to document the BPF features that we're using in Calico. Making these checkboxes so we can tick them off as they're supported...

BPF helper functions (general):

• bpf_trace_printk
• bpf_ktime_get_ns
• bpf_get_prandom_u32
• bpf_tail_call
• bpf_map_lookup_elem
• bpf_map_update_elem
• bpf_map_delete_elem

TC/XDP programs:

• bpf_csum_diff
• bpf_l3_csum_replace
• bpf_l4_csum_replace
• bpf_redirect
• bpf_fib_lookup (issue Add helper function : bpf_fib_lookup #525)
• bpf_skb_change_tail
• bpf_xdp_adjust_meta (issue Add XDP helper functions to adjust XDP context. #524)
• bpf_skb_adjust_room
• bpf_skb_pull_data

Connect-time load balancer (which runs in socket hooks):

• bpf_get_socket_cookie

Features and hooks that we make use of:

• XDP
• "tc" hooks at ingress and egress (issue Add TC hook functionality #686)
• Various socket hooks for the connect-time load balancer
• Direct packet access
• Tail calls
• XDP metadata to pass information from XDP to tc programs (issue Add XDP helper functions to adjust XDP context. #524)
• FIB lookups (issue Add helper function : bpf_fib_lookup #525)
• Packet redirection (bpf_redirect)
• Building BPF program at runtime and loading bytecode directly.
• Attaching to various kprobes to collect socket statistics around TCP connections.
• Ability to set mark bits on packets for other bpf programs to see

Map types:

• BPF_MAP_TYPE_LRU_HASH
• BPF_MAP_TYPE_HASH
• BPF_MAP_TYPE_LPM_TRIE
• BPF_MAP_TYPE_PERCPU_ARRAY
• BPF_MAP_TYPE_PROG_ARRAY
• BPF_MAP_TYPE_PERF_EVENT_ARRAY
• libbpf "global variables" pseudo map

libbpf functions:

• bpf_object__open
• libbpf_get_error
• bpf_object__load
• bpf_object__find_program_by_name
• bpf_program__fd
• bpf_tc_attach (issue Add TC libbpf APIs #685)
• bpf_tc_query (issue Add TC libbpf APIs #685)
• bpf_tc_hook_create (issue Add TC libbpf APIs #685)
• bpf_tc_hook_destroy (issue Add TC libbpf APIs #685)
• bpf_object__find_map_fd_by_name
• bpf_map_update_elem
• bpf_program__attach_kprobe
• bpf_link__destroy

We also have our own BPF syscall-based functions that we use for some operations; I think there are libbpf equivalents for most of these and we're generally transitioning over to libbpf over time:

• BPF_OBJ_GET (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)
• BPF_MAP_GET_FD_BY_ID (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)
• BPF_PROG_LOAD (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)
• BPF_PROG_TEST_RUN (issue Add shim for bpf syscall so other go libraries can be ported to windows #559, eBPF performance tests should perform end-to-end tests #594)
• BPF_OBJ_PIN (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)
• BPF_MAP_UPDATE_ELEM (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)
• BPF_MAP_LOOKUP_ELEM (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)
• BPF_MAP_DELETE_ELEM (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)
• BPF_OBJ_GET_INFO_BY_FD (issue Add shim for bpf syscall so other go libraries can be ported to windows #559)

Program types we use:

• BPF_PROG_TYPE_SCHED_CLS/ACT (issue Add TC hook functionality #686)
• BPF_PROG_TYPE_XDP
• BPF_PROG_TYPE_CGROUP_XXX (attached via libbpf cgroup/connect4 cgroup/sendmsg4 et al; I can't remember the exact program type that maps to)
• BPF_PROG_TYPE_KPROBE
• BPF_PROG_TYPE_TRACEPOINT

It's a little harder to pin down all the attach types because they're abstracted by libbpf but here's my guess:

• <whatever "tc" programs use> (issue Add TC hook functionality #686)
• BPF_XDP
• BPF_CGROUP_INET4_CONNECT
• BPF_CGROUP_INET6_CONNECT
• BPF_CGROUP_INET4_POST_BIND
• BPF_CGROUP_INET6_POST_BIND
• BPF_CGROUP_UDP4_SENDMSG
• BPF_CGROUP_UDP6_SENDMSG
• BPF_TRACE_FENTRY
• BPF_TRACE_FEXIT

[dthaler]

@fasaxc Thanks Shaun, great info! Can you also list the BPF_PROG_TYPE_* (and BPF_ATTACH_TYPE_* values if any) that Calico uses?

[dthaler]

@fasaxc Could you use BPF_MAP_TYPE_RINGBUF (on both Linux and Windows) instead of BPF_MAP_TYPE_PERF_EVENT_ARRAY? The former is being added in PR #659.

[fasaxc]

@dthaler Yes, we were planning to add RINGBUF support on Linux too (depending on kernel support).

[dthaler]

@fasaxc for the "Packet redirection" do you use XDP_TX or what API?

[fasaxc]

We don't currently do redirection from our XDP programs. We only implement policy there. We do do redirection from our TC programs, where we use bpf_redirect.

I can see us using XDP_TX in future.

[song-jiang]

@dthaler Got confirmation from @fasaxc that Calico uses bpf_xdp_adjust_meta to pass metadata from XDP to TC.