EVL networking in a nutshell

EVL features a simple network stack which currently implements the UDP protocol from the PF_INET domain (IPv4), and raw ethernet datagrams from the PF_PACKET domain, both over an ethernet medium. This provides datagram-oriented networking support to applications with real-time communication requirements, through the common networking device infrastructure. Out-of-band communication happens via out-of-band network ports as provided by Dovetail.

Out-of-band network ports

An out-of-band network port as defined by the Dovetail interface is a communication channel hosted by a regular Linux network device (netdev), which provides a way for real-time applications to send and receive packets from the out-of-band execution stage. Both real (physical) devices and virtual (VLAN) interfaces sitting on top of the latter can become out-of-band network ports independently from each other. However, when a VLAN device is turned into an out-of-band port, input diversion is automatically enabled on the underlying real device.

Enabling a network device as an out-of-band port can be done in three ways:

  • Using the evl net -ei <interface> management command.

  • Calling the evl_net_enable_port() service from your application.

  • writing a non-zero value to /sys/class/net/<netdev>/oob_port. Conversely, writing zero disables the port.

Enabling a network device as an out-of-band port does NOT magically make its driver oob-capable. Doing so always requires code changes to the stock kernel driver for the relevant network interface controller. Enabling the port merely allows out-of-band traffic to be submitted to the network device by applications running on the out-of-band stage. If the driver is not oob-capable, ingress packets are (usually) received from the NAPI poll routine running on the in-band stage then forwarded to out-of-band consumers, egress packets are relayed from out-of-band senders to the driver transmit routine running on the in-band stage. In other words, when the driver is not oob-capable, no real-time requirement can be met by EVL, the worst-case latency depends on the real-time capabilities of the in-band kernel.

Redirecting the ingress traffic to EVL

EVL relies on Dovetail which can redirect ingress traffic between NIC drivers and the EVL core directly, either from the in-band or out-of-band stages, depending on whether such drivers have been adapted in order to handle traffic from the out-of-band stage. EVL can recognize the ethernet traffic it should handle based on two types of input filters:

  • VLAN tagging. Packets flowing into an out-of-band VLAN are redirected to the EVL network stack, others are left for handling by the regular / in-band network stack. Any valid VLAN number may be assigned to EVL for accepting traffic from it.

Since the 802.1Q standard has been around for quite some time by now, most ethernet switches should pass on frames with the ethertype information set to the 802.1Q TPID “as is” to some hardware port, and they should also be able to cope with the four additional octets involved in VLAN tagging without having to lower the MTU everywhere (most equipments even support jumbo frames these days).

  • eBPF filtering. An eBPF program can be installed on any network device hosting an out-of-band port to decide whether an ingress packet should be processed by the EVL network stack, the in-band stack, or dropped. The program should be of type BPF_PROG_TYPE_SOCKET_FILTER, receiving a socket buffer.

By returning the appropriate status code, the filter program can decide to either:

  • Accept the packet for handling by the EVL network stack (EVL_RX_ACCEPT).
  • Hand over the packet to the in-band stack instead (EVL_RX_SKIP).
  • Postpone the decision to applying the VLAN matching rule (EVL_RX_VLAN).
  • Drop the packet, which won’t enter any network stack as a consequence (EVL_RX_DROP).

When present, the eBPF program takes precedence over the VLAN matching rule. In absence of eBPF filter, the VLAN matching rule applies. Packets which EVL did not accept are transparently forwarded to the in-band network stack.

Therefore, a transparent filter passing all input to the next selection method (i.e. the VLAN-based one) would look like this:

#include <linux/types.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <evl/net/bpf-abi.h>

SEC("socket")
int bpf_netrx(struct __sk_buff *skb)
{
	return EVL_RX_VLAN;
}

char _license[] SEC("license") = "GPL";

Conversely, a filter entirely bypassing EVL for ingress traffic would look like this:

#include <linux/types.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include <evl/net/bpf-abi.h>

SEC("socket")
int bpf_netrx(struct __sk_buff *skb)
{
	return EVL_RX_SKIP;
}

char _license[] SEC("license") = "GPL";

Obviously, if you can’t rely on the built-in VLAN-based selection of packets EVL implements, you may have to use more elaborate rules in order to divert the appropriate portion of the traffic received from the network interface to EVL.

The following diagram illustrates the flow of an incoming packet from the network interface controller to the service access point in the application, i.e. the socket.

Alt text

Egress traffic handling

EVL is in charge of building the outgoing packets for conveying the data sent by applications which run on the out-of-band execution stage. Analogously to receiving the ingress traffic, EVL can either send the outgoing packets directly from the out-of-band stage, or hand them over to the in-band network stack, depending on whether the NIC driver has been adapted in order to handle traffic from the out-of-band stage directly.

It is fundamental to understand that enabling an out-of-band port on a network device does not per se ensure end-to-end real-time communication through it. For this guarantee to be provided, the NIC driver must have been made oob-capable, so that packet transmit and receive operations to/from the hardware do happen directly from the out-of-band stage. However, in any case, EVL still guarantees that threads calling out-of-band receive/transmit services won’t demote to the in-band execution stage when doing so.

Typical use cases

Dedicating a network interface to out-of-band traffic

When multiple network interfaces are available from the hardware platform, dedicating one or more of them to sending and/or receiving out-of-band traffic is best to decrease network latency for real-time applications.

Sharing a network interface between in-band and out-of-band traffic

The hardware platform might have a single network interface available, in which case VLAN tagging may come in handy to direct incoming packets either to the EVL network stack or the in-band one. Obviously, this comes at a cost with respect to latency, since the in-band traffic might slow down the out-of-band packets at the hardware level. Likewise, the hardware would be shared for transmitting both in-band and out-of-band originated packets. This said, depending on the real-time requirements, that cost may still be within budget for many applications.

Dealing with complex out-of-band selection rules

An eBPF program allows deep inspection of the packet data before issuing any decision about which network stack should handle the traffic. One may rely on this feature to implement complex out-of-band traffic detection rules.

Out-of-band support in NIC drivers

Unlike Xenomai 3 with the RTnet stack, EVL provides a network stack which does not require EVL-specific drivers. Instead, the capabilities of the stock NIC drivers can be extended to supporting out-of-band I/O operations for dealing with ingress and egress traffic, based on facilities provided by Dovetail in that area. If a driver does not advertise out-of-band I/O capabilities, EVL automatically offloads the final I/O operations to the in-band network stack for talking to the network device, allowing the application code to keep running on the out-of-band stage in parallel.

Although EVL does not require the NIC driver code to be oob-capable, i.e. conveying ingress and egress traffic directly from the out-of-band execution stage, having such support in place is the only way to have a complete, end-to-end real-time networking path, from the Ethernet wire to the application code. In other words, one may still use stock ethernet controller drivers along with the EVL network stack, at the expense of the real-time performance which would depend on the low-latency capabilities of the host kernel.

Out-of-band packet routing

EVL does no input routing. At all. All ingress packets accepted by EVL should either be delivered to a receiver on the local host or dropped, there is no forwarding of any sort of such data to remote hosts.

However, EVL does a basic form of output routing for IPv4 over Ethernet, enough to satisfy the requirements of the UDP protocol for determining the proper output device to submit outgoing packets to, along with the MAC address to put into the Ethernet destination field of those packets. However, an out-of-band sender cannot perform direct lookups into the tables maintained by the in-band network stack to find the routing information it needs, this would contradict the stage separation rule. In order to address this issue, EVL maintains two front caches containing the related information:

  • a table of routes indexed on the IPv4 address of the receiving peer. Each entry refers to the routing information determined by the in-band kernel for those outgoing routes (i.e. a struct rtable item).

  • a table of ARP entries indexed on a composite key combining the output device found by the in-band routing process for reaching the peer, and the IPv4 address where it may be reached from that device. Each entry contains the MAC Ethernet address to be written to the destination field of a packet for sending it to the given peer. This cache tracks the updates to its counterpart maintained by the in-band network stack.

In other words, EVL does not calculate any route, does not determine any destination MAC address by itself: it simply collects and learns this information from the in-band kernel when the latter estabishes a new route via some network device. When an out-of-band port is enabled for that device, EVL receives a notification about such route, recording this information into its own caches, which in turn can be accessed directly from the out-of-band stage by applications.

Peer solicitation

As mentioned above, EVL is able to figure out where to channel packets to an IPv4 peer based on the route and Ethernet addressing information it retrieves from its dedicated front caches. If this information is absent when needed for an out-of-band caller to issue an outgoing UDP packet, the sending operation is demoted to the in-band stage, where using the related databases of the in-band network stack is allowed. Obviously, doing so also would drop any real-time guarantee, which would not be that nice. Therefore, we must make sure to have this information directly available from the EVL front caches in order to meet real-time requirements from the out-of-band stage.

The process of obtaining this information involves the peer host which should eventually reply to an ARP request with its MAC address, once we have determined through which network device such request should be sent. By extension, EVL calls the this process along with the logging of the routing data received into the front caches, peer solicitation. There are two ways to solicit a peer:

Sending an UDP packet to a non-solicited peer will cause this packet to be offloaded to the in-band network stack, losing the EVL real-time guarantee in the process.

ARP cache management

Keep in mind that ARP entries are ageing unless made permanent. After some time, the entry is no more usable and the corresponding peer should be probed again to confirm its presence, which might cause outgoing UDP packets to be offloaded to the in-band stage in the meantime. For this reason, assuming the topology and addressing of your real-time has to be stable anyway, it is wise to set the ARP entries of real-time peers as permanent, typically by using the evl net -S command form instead of evl net -s.

Since EVL monitors the updates to the ARP table maintained by the in-band network stack, you can use the common administration tools such as arp(8) to add, remove or set the properties of ARP entries for real-time peers. Those changes will be mirrored into the related EVL front cache.

Flushing the EVL front cache from all ARP entries can be achieved by writing a non-zero value to /sys/class/evl/net/arp, e.g.:

~# echo 1 > /sys/class/evl/net/arp

Route cache management

Flushing the EVL front cache from all IPv4 routes can be achieved by writing a non-zero value to /sys/class/evl/net/ipv4_routes, e.g.:

~# echo 1 > /sys/class/evl/net/ipv4_routes

Special addresses

Local host

When EVL is present, the loopback device (aka ’lo’) is automatically turned into an oob-capable device. This means that ’localhost’ (127.0.0.1) is a perfectly valid peer host for end-to-end real-time communication. However, you still need to enable the out-of-band port on this device, then solicit this address prior to sending it UDP datagrams directly from the out-of-band stage.

~# evl net -ei lo -S localhost

Broadcast

You need to solicit the 'broadcast' address (255.255.255.255) prior to broadcasting UDP datagrams directly from the out-of-band stage. For this operation to be successful, you need to have enabled an out-of-band port on a device which supports broadcasting too. For instance:

~# evl net -S broadcast -i eth0.42
Last modified: Thu, 03 Jul 2025 10:14:18 +0200