OVS_AFXDP is a new datapath implementation for Open vSwitch. The new datapath is based on eXpress Data Path (XDP) - the in-kernel hook for eBPF programs designed to provide high-performance packet processing subystem in Linux. AF_XDP is a new socket type (address family) that is built on top of XDP. AF_XDP redirects incoming packets to memory buffer in a user-space application.
AF_XDP in Open vSwitch is expected to provide similar performance to the DPDK datapath, but with a lower configuration overhead. If AF_XDP will become a successful implementation it would allow to port existing kernel-based OVS deployments to the user-space implementations providing better packet processing performance.
From my perspective, OVS_AFXDP is interesting as it can be the solution for P4rt-OVS (P4-capable Open vSwitch developed by my team at Orange Labs Poland), which requires to process packets in the user-space and is currently based on DPDK. Hence, I decided to test OVS_AFXDP and evaluate how it can be integrated to P4rt-OVS. This post describes the installation process that I went through. As there is only the official documentation on how to install OVS_AFXDP, this post can be a useful extension.
Installation of OVS_AFXDP
According to the official documentation, OVS_AFXDP requires at least kernel 5.0.0. I installed OVS_AFXDP on Ubuntu 18.10, which comes with kernel 4.18 already integrated. Kernel 4.18 has some initial support for XDP, but it is not suitable for OVS_AFXDP. Hence, we need to install the newer kernel first. I recommend you to install kernel 5.4.1 or higher because it introduces important modifications to how AF_XDP works.
Building kernel with the XDP support
I decided to install the latest stable version of Linux - 5.4.12:
sudo apt install -y build-essential libncurses-dev bison flex libssl-dev wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.12.tar.xz unxz -v linux-5.4.12.tar.xz wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.4.12.tar.sign gpg --verify linux-5.4.12.tar.sign
The sample output will be:
gpg: assuming signed data in 'linux-5.4.12.tar' gpg: Signature made Sun 12 Aug 2018 04:00:28 PM CDT gpg: using RSA key 647F28654894E3BD457199BE38DBBDC86092693E gpg: Can't check signature: No public key
Copy RSA to the clipboard and run:
gpg --recv-keys 647F28654894E3BD457199BE38DBBDC86092693E gpg --verify linux-5.4.12.tar.sign
Now, we can move to the installation process. Enter linux directory, make config and make sure that following options are enabled:
# Open config editor and save it to .config $ make menuconfig # or you can copy the current one: $ cp -v /boot/config-$(uname -r) .config # Check if following options are enabled: # CONFIG_BPF=y # CONFIG_BPF_SYSCALL=y # CONFIG_XDP_SOCKETS=y
Then, build the kernel:
make -j $(nproc) sudo make modules_install INSTALL_MOD_STRIP=1 sudo make install
The next, important step is to install kernel’s headers. They are used by userspace programs (such as Open vSwitch) to interface with kernel services. In other words, kernel’s headers provide the API for userspace programs. For instance, in case of OVS_AFXDP,
if_xdp.h provides the API to the XDP socket.
sudo make INSTALL_HDR_PATH=/usr headers_install
After system’s reboot you should have up and running Linux 5.4.12 that will allow you to install OVS_AFXDP without any problems.
Firstly, let’s install required tools and dependencies:
sudo apt install -y git make gcc libelf-dev autoconf libtool
Next, clone the OVS repository:
git clone https://github.com/openvswitch/ovs
Then, according to the official documentation, the
libbpf should be installed:
git clone git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git cd bpf-next/ cd tools/lib/bpf/ make sudo make install sudo make install_headers
In my case, above commands were not enough. I had to copy the
libbpf shared library to location, where all libraries are stored and create a symbolic link. I based on the tutorial from Intel.
# from bpf-next/tools/lib/bpf/ sudo cp libbpf.so.0 /usr/lib/ sudo cp libbpf.so.0.0.7 /usr/lib/ sudo ldconfig -v -n /usr/lib sudo ln -sf /usr/lib/libbpf.so.0 /usr/lib/libbpf.so.0.0.7 sudo ldconfig ldconfig -p | grep bpf # You should have libbpf listed in the output of the above command.
Now, let’s configure OVS.
cd ovs/ ./boot.sh ./configure --enable-afxdp
At this stage, if you use older kernel (e.g. 4.18 or even 5.0.0) you will get the following error during configuration process. This error will also appear if the command
sudo make INSTALL_HDR_PATH=/usr headers_install has not been invoked.
configure: WARNING: bpf/xsk.h: present but cannot be compiled configure: WARNING: bpf/xsk.h: check for missing prerequisite headers? configure: WARNING: bpf/xsk.h: see the Autoconf documentation configure: WARNING: bpf/xsk.h: section "Present But Cannot Be Compiled" configure: WARNING: bpf/xsk.h: proceeding with the compiler's result configure: WARNING: ## ----------------------------------- ## configure: WARNING: ## Report this to firstname.lastname@example.org ## configure: WARNING: ## ----------------------------------- ## checking for bpf/xsk.h... no configure: error: unable to find bpf/xsk.h for AF_XDP support
If we would look at
config.log we can see the following message:
configure:18659: checking bpf/xsk.h usability configure:18659: gcc -c -g -O2 conftest.c >&5 In file included from conftest.c:69: /usr/local/include/bpf/xsk.h: In function 'xsk_ring_prod__needs_wakeup': /usr/local/include/bpf/xsk.h:82:21: error: 'XDP_RING_NEED_WAKEUP' undeclared (first use in this function) return *r->flags & XDP_RING_NEED_WAKEUP; ^~~~~~~~~~~~~~~~~~~~ /usr/local/include/bpf/xsk.h:82:21: note: each undeclared identifier is reported only once for each function it appears in /usr/local/include/bpf/xsk.h: In function 'xsk_umem__extract_addr': /usr/local/include/bpf/xsk.h:173:16: error: 'XSK_UNALIGNED_BUF_ADDR_MASK' undeclared (first use in this function) return addr & XSK_UNALIGNED_BUF_ADDR_MASK; ^~~~~~~~~~~~~~~~~~~~~~~~~~~ /usr/local/include/bpf/xsk.h: In function 'xsk_umem__extract_offset': /usr/local/include/bpf/xsk.h:178:17: error: 'XSK_UNALIGNED_BUF_OFFSET_SHIFT' undeclared (first use in this function) return addr >> XSK_UNALIGNED_BUF_OFFSET_SHIFT; ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is caused by the old version of
if_xdp.h that does not provide declaration of
XDP_RING_NEED_WAKEUP and others. If you get this error, come back to the section
Building kernel with the XDP support and install kernel 5.4.12, which comes with
if_xdp.h supporting those declarations.
./configure has been completed successfully, let’s build OVS_AFXDP:
make -j $(nproc) sudo make install
Voila! OVS_AFXDP is ready to be run!
It is recommended to run tests first to check basic AF_XDP functionality:
make check-afxdp TESTSUITEFLAGS='1'
libbpf has not been installed properly the test will fail with the following message:
020-01-17 06:54:19.418142218 +0000 @@ -0,0 +1 @@ +ovsdb-tool: error while loading shared libraries: libbpf.so.0: cannot open shared object file: No such file or directory ./system-afxdp.at:5: exit code was 127, expected 0
In my case I had to fix the installation of
If tests were passed, we can move on and run OVS_AFXDP! Firstly, let’s run OVSDB and
sudo mkdir -p /usr/local/etc/openvswitch sudo ovsdb-tool create /usr/local/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema sudo mkdir -p /usr/local/var/run/openvswitch sudo ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ --pidfile --detach sudo ovs-vswitchd --pidfile --detach
Then, let’s configure OVS with the netdev datapath:
sudo ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev
OVS_AFXDP can be configured with both physical and virutal interfaces. For the test purpose, I use
veth pairs and attach them to OVS:
sudo ip netns add test0 sudo ip link add port0 type veth peer name peer0 sudo ip link set port0 netns test0 sudo ip link set dev peer0 up sudo ovs-vsctl add-port br0 peer0 -- set interface peer0 external-ids:iface-id="port0" type="afxdp" sudo ip netns exec test0 sh << NS_EXEC_HEREDOC ip addr add "10.1.1.1/24" dev port0 ip link set dev port0 up NS_EXEC_HEREDOC sudo ip netns add test1 sudo ip link add port1 type veth peer name peer1 sudo ip link set port1 netns test1 sudo ip link set dev peer1 up sudo ovs-vsctl add-port br0 peer1 -- set interface peer1 external-ids:iface-id="port1" type="afxdp" sudo ip netns exec test1 sh << NS_EXEC_HEREDOC ip addr add "10.1.1.2/24" dev port1 ip link set dev port1 up NS_EXEC_HEREDOC
The final configuration of OVS should looks as follows:
ubuntu@ovsafxdp:~/ovs$ sudo ovs-vsctl show e7c40460-0252-4c98-a225-44416c01ead2 Bridge br0 datapath_type: netdev Port peer1 Interface peer1 type: afxdp Port br0 Interface br0 type: internal Port peer0 Interface peer0 type: afxdp
To test if OVS_AFXDP works, let’s run
ping in both directions:
ubuntu@ovsafxdp:~/ovs$ sudo ip netns exec test0 ping 10.1.1.2 PING 10.1.1.2 (10.1.1.2) 56(84) bytes of data. 64 bytes from 10.1.1.2: icmp_seq=1 ttl=64 time=0.179 ms 64 bytes from 10.1.1.2: icmp_seq=2 ttl=64 time=0.110 ms 64 bytes from 10.1.1.2: icmp_seq=3 ttl=64 time=0.106 ms ubuntu@ovsafxdp:~/ovs$ sudo ip netns exec test1 ping 10.1.1.1 PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data. 64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.820 ms 64 bytes from 10.1.1.1: icmp_seq=2 ttl=64 time=0.104 ms 64 bytes from 10.1.1.1: icmp_seq=3 ttl=64 time=0.086 ms
In this post, I described the installation and configuration process to make OVS_AFXDP up and running. As I encountered some problems on the way to install OVS_AFXDP I thought that it was worth sharing my experience with the community. Hence, this post can be a useful complement to the official documentation of OVS_AFXDP. Enjoy!
The next step is to run our P4rt-OVS prototype with AF_XDP and get rid of DPDK. We also consider to integrate OVS_AFXDP with Mininet to provide researchers and network engineers a simple tool to run and test P4rt-OVS, without the need for DPDK.