Socket

Using out-of-band enabled sockets

The EVL network stack in kernel space communicates with applications running in user-space via the common socket service access point. Passing the SOCK_OOB flag to the socket(2) system call enables out-of-band services for the new socket, provided the EVL network stack implements such support for the address and protocol family specified in the socket creation call. Currently, EVL implements AF_INET,SOCK_DGRAM (i.e. IPv4 UDP protocol) and AF_PACKET,SOCK_RAW (raw ethernet packet). Custom protocols can be implemented in the out-of-band network stack.

Since any oob-enabled socket is primarily a genuine socket with extended services provided by the EVL core, the complete set of standard ioctl(2) requests and socket options are available with oob-enabled sockets. In addition, the EVL core handles its own set of requests and options.

EVL-specific socket options

In-band options should be set and retrieved using the regular setsockopt(2) and getsockopt(2) system calls, which will be redirected to the EVL network stack from the in-band execution stage. Conversely, out-of-band options should be set and retrieved using the oob_setsockopt() and oob_getsockopt() system calls. The following options are recognized:

  • SO_TIMESTAMP_OOB, configures packet timestamping either from the in-band or out-of-band execution stage, indifferently. See the discussion.

Packet timestamping

The EVL network stack can timestamp ingress and/or egress traffic upon request from the application. The general form for configuring at socket level the timestamping feature is as follows:

```
int tsflags = <timestamping-selection-flags>;
int ret = setsockopt(s, SOL_SOCKET, SO_TIMESTAMP_OOB, &tsflags, sizeof(tsflags));
int ret = oob_setsockopt(s, SOL_SOCKET, SO_TIMESTAMP_OOB, &tsflags, sizeof(tsflags));
```

The timestamping information is returned in the C `struct iotimes`,
which is defined as follows:

```
struct evl_net_iotimes {
	/* Time at device<->netstack boundary (monotonic). */
	__u64 device_time;
	/* Time at RX/TX thread dequeuing/queuing point (monotonic). */
	__u64 queuing_time;
	/* Time at kernel/user boundary (monotonic). */
	__u64 delivery_time;
};
```

On TX, timestamps are collected asynchronously, and can be
retrieved by the application by passing the `MSG_TIMESTAMP`
operation flag to [oob_recvmsg()](/core/user-api/io/#oob_recvmsg), instead of receiving
traffic data.

```
#include <evl/net/socket.h>

struct evl_net_iotimes iotimes[16] = { 0 }; /* Collect up to 16 timestamps */
struct oob_msghdr msghdr;
struct iovec iov;
ssize_t ret;

iov.iov_base = iotimes;
iov.iov_len = sizeof(iotimes);
msghdr.msg_iov = &iov;
msghdr.msg_iovlen = 1;
msghdr.msg_control = NULL;
msghdr.msg_controllen = 0;
msghdr.msg_name = NULL;
msghdr.msg_namelen = 0;
msghdr.msg_flags = 0;

ret = oob_recvmsg(s, &msghdr, NULL, MSG_TIMESTAMP | MSG_DONTWAIT);
if (ret > 0)
	 /* Count of TX timestamps received is ret / sizeof(iotimes[0]. */
```

On RX, the timestamp for the received packet is returned by the
[oob_recvmsg()](/core/user-api/io/#oob_recvmsg) call directly, via the
control data area.

```
#include <evl/net/socket.h>

struct evl_net_iotimes iotimes = { 0 };
struct oob_msghdr msghdr;
struct iovec iov[...];
ssize_t ret;

msghdr.msg_iov = iov;
msghdr.msg_iovlen = <number-of-iov-cells>;
msghdr.msg_control = &iotimes;
msghdr.msg_controllen = sizeof(iotimes);
msghdr.msg_name = NULL;
msghdr.msg_namelen = 0;
msghdr.msg_flags = 0;

ret = oob_recvmsg(s, &msghdr, NULL, 0);
if (msghdr.msg_controllen == sizeof(iotimes)) {
	/* RX timestamp available along with the packet data (in iov[]). */
}
```

The EVL network stack timestamps ingress and/or egress packets if
`EVL_SOF_TIMESTAMP_RX` and/or `EVL_SOF_TIMESTAMP_TX` are set in
`tsflags` respectively. Conversely, timestamping is disabled for a
direction if the corresponding flag is cleared in
`tsflags`. Therefore, passing zero in `tsflags` disables
timestamping entirely.

  • if EVL_SOF_TIMESTAMP_DEVICE is set, every incoming packet is timestamped when the NIC driver hands it over to the network stack for receive (EVL_SOF_TIMESTAMP_RX), or when the network stack hands it over to the NIC driver for transmit (EVL_SOF_TIMESTAMP_TX).

  • if EVL_SOF_TIMESTAMP_QUEUING is set along with EVL_SOF_TIMESTAMP_RX, every incoming packet is timestamped when the RX thread in the network stack passes it to the receiving protocol layer. In this case, queuing_time - device_time measures the delay spent waiting in the RX queue. if EVL_SOF_TIMESTAMP_QUEUING is set along with EVL_SOF_TIMESTAMP_TX, every outgoing packet is timestamped when the network stack queues it for transmission by the TX thread, according to the queuing discipline in effect. In this case, device_time - queuing_time measures the delay spent waiting in the TX queue in addition to the qdisc handling.

  • if timestamping is enabled for a direction, a timestamp is collected unconditionally at the kernel <-> user boundary, this information is stored into the delivery_time field of the struct evl_net_iotimes. On RX, the delivery timestamp is taken when a packet is queued by the protocol layer, for consumption by oob_recvmsg() later on, therefore delivery_time - device_time measures the delay between receipt from the hardware and availability to the application. On TX, the delivery timestamp is taken when the network stack starts building the outgoing packet received from the application via a call to oob_sendmsg(), therefore device_time - delivery_time measures the delay between data receipt from the application and packet submission to the hardware for transmit. latmus -E measures the maximum delays observed in both directions.

EVL_SOF_TIMESTAMPS is a shorthand enabling all timestamping locations (RX and TX).