Flowtables
Flowtables allow you to accelerate packet forwarding in software (and in hardware if your NIC supports it) by using a conntrack-based network stack bypass.
Entries are represented through a tuple that is composed of the input interface, source and destination address, source and destination port; and layer 3/4 protocols. Each entry also caches the destination interface and the gateway address (to update the destination link-layer address) to forward packets.
The TTL and hoplimit fields are also decremented. Hence, flowtables provides an alternative path that allow packets to bypass the classic forwarding path.
userspace process
^ |
| |
_____|____ ____\/___
/ \ / \
| input | | output |
\__________/ \_________/
^ |
| |
_________ __________ --------- _____\/_____
/ \ / \ |Routing | / \
--> ingress ---> prerouting ---> |decision| | postrouting|--> neigh_xmit
\_________/ \__________/ ---------- \____________/ ^
| ^ | ^ |
flowtable | ____\/___ | |
| | / \ | |
__\/___ | | forward |------------ |
|-----| | \_________/ |
|-----| | 'flow offload' rule |
|-----| | adds entry to |
|_____| | flowtable |
| | |
/ \ | |
/hit\_no_| |
\ ? / |
\ / |
|__yes_________________fastpath bypass ____________________________|
Fig.1 Netfilter hooks and flowtable interactions
Flowtables reside in the ingress hook that is located before the prerouting hook. You can select which flows you want to offload through the flow expression from the forward chain. Flowtables are identified by their address family and their name. The address family must be one of ip, ip6, or inet. When no address family is specified, ip is used by default.
Flows are offloaded after the state is created. That means that usually the first reply packet will create the flowtable entry. A firewall rule to accept the initial traffic is required. The flow expression on the forward chain must match the return traffic of the initial connection. Be aware that the return route is deducted from the packet, that creates the flowtable entry. This also means if you are using special ip rules, you need to make sure that they match the reply packet traffic as well as the original traffic.
The *priority* can be a signed integer or *filter* which stands for 0. Addition and subtraction can be used to set relative priority, e.g. filter + 5 equals to 5.
The *devices* are specified as iifname(s) of the input interface(s) of the traffic that should be offloaded. Devices are required for both traffic directions.
Example:
table inet x {
flowtable f {
hook ingress priority 0; devices = { eth0, eth1 };
}
chain forward {
type filter hook forward priority 0; policy drop;
# offload established connections
meta l4proto { tcp, udp } flow offload @f counter packets 0 bytes 0
# established/related connections
ct state established,related counter accept
# allow initial connection
meta l4proto { tcp, udp } accept
}
}
Note that:
- The rule that uses the flow offload statement determines what flows are added to the flowtable. This ruleset above adds entries to the flowtable for TCP and UDP packets.
- The devices you specify in the flowtable declaration determine where the flowtable hooks in the pipeline for lookups, in the example above, it registers a hook for devices eth0 and eth0 in the ingress hook at priority 0.
See also
- Linux kernel documentation on Netfilter flowtable
- Netfilter Mini-Workshop, Netdev 0x13, 2019-03
- Mellanox flowtable hardware offload
- Some Mellanox flowtable hardware offload performance measurements by Wen Xu of UCloud
- Netfilter hardware offloads, Pablo Neira Ayuso, Linux Plumbers Conference, 2019-09