Out-of-band I/O services
Communicating with real-time capable device drivers
Using the EVL kernel API, you can extend an existing driver for supporting out-of-band I/O operations, or even write one from scratch. Both character-based I/O and socket protocol drivers are supported.
On the user side, application can exchange data with, send requests to these real-time capable drivers from the out-of-band execution stage with the a couple of additional services libevl provides.
You may notice that several POSIX file I/O services such as open(2), socket(2), close(2), fcntl(2), mmap(2) and so on have no out-of-band counterpart in the following list. The reason is that we don’t need them: opening, closing or mapping a file are inherently non-deterministic operations, which may block for an unspecified amount of time for a number of reasons, depending on the underlying file and current runtime conditions. Besides, those are hardly useful in a time-critical loop.
However, issuing data transfers and control requests to the driver is definitely something we may want to happen within a bounded time, hence directly from the out-of-band execution stage.
Since the EVL core exports every public element as a character device which can be accessed from /dev/evl, libevl can interface with elements from other processes through the out-of-band I/O requests documented here, which are sent to the corresponding devices.
Opening an out-of-band enabled I/O channel
Since the EVL core does not redefine the open(2) and socket(2) calls, there has to be a way to tell the kernel code managing the device and/or protocol that we want to enable out-of-band operations.
In most cases, we don’t have to do so though, because the purpose of the corresponding device driver is all about providing out-of-band services, so enabling them for any connecting file is implicit. For instance, most of the drivers accessed through the /dev/evl file hierarchy turn on out-of-band services automatically.
However, some drivers might distinguish between out-of-band enabled files and others, providing a different set of services. Typically, a regular in-band driver which is extended in order to handle out-of-band requests too should be told when to do so for any given file.
To meet this requirement, Dovetail introduces the additional open flag O_OOB, which can be passed
to open(2) ORed
into the flags argument. Similarly, it defines the SOCK_OOB flag
which can be passed to
socket(2) ORed
into the type argument for the same purpose. If the receiving driver
implements opt-in out-of-band services, passing this flag when opening
a file/socket should enable them.
Not all devices drivers may support out-of-band operations (the
overwhelming majority does not). Whether passing either O_OOB or
SOCK_OOB to them when opening a file/socket would cause an error, or
the flag would just be ignored depends on the driver code.
Out-of-band I/O services
This is the strict equivalent to the standard read(2) system call, for sending the request from the out-of-band stage to an EVL driver. In other words, oob_read() attempts to read up to count bytes from file descriptor fd into the buffer starting at buf, from the out-of-band execution stage.
The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.
A file descriptor obtained by opening a real-time capable driver which we want to read from.
A buffer to receive the data.
The number of bytes to read at most, which should fit into buf.
oob_read() returns the actual number of bytes read, copied to buf on success. Otherwise, -1 is returned, and errno is set to the error code:
EBADF if fd does not refer to a real-time capable driver, or fd was not opened for reading.
EINVAL if fd does not support the .oob_read operation.
EFAULT if buf points to invalid memory.
EAGAIN fd is marked as non-blocking (O_NONBLOCK), and the read operation would block.
Other driver-specific error codes may be returned, such as:
ENOBUFS fd is a cross-buffer file descriptor, and there is no ring buffer space associated with the outbound traffic (i.e. o_bufsz parameter was zero when creating the cross-buffer).
EINVAL fd is a cross-buffer file descriptor, and count is greater than the size of the ring buffer associated with the inbound traffic. (i.e. the i_bufsz parameter given when creating the cross-buffer).
EFBIG fd is a proxy file descriptor, and count is larger than the size of the input buffer as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).
EINVAL fd is a proxy file descriptor, and count is not a multiple of the input granularity as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).
ENXIO fd is a proxy file descriptor which is not available for input. See EVL_CLONE_INPUT.
This is the strict equivalent to the standard write(2) system call, for sending the request from the out-of-band stage to an EVL driver. In other words, oob_write() attempts to write up to count bytes to file descriptor fd from the buffer starting at buf, from the out-of-band execution stage.
The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.
A file descriptor obtained by opening a real-time capable driver which we want to read from.
A buffer containing the data to be written.
The number of bytes to write starting from buf.
oob_write() returns the actual number of bytes written from buf on success. Otherwise, -1 is returned, and errno is set to the error code:
EBADF if fd does not refer to a real-time capable driver, or fd was not opened for writing.
EINVAL if fd does not support the .oob_write operation.
EFAULT if buf points to invalid memory.
EAGAIN fd is marked as non-blocking (O_NONBLOCK), and the write operation would block.
Other driver-specific error codes may be returned, such as:
ENOBUFS fd is a cross-buffer file descriptor, and there is no ring buffer space associated with the outbound traffic (i.e. o_bufsz parameter was zero when creating the cross-buffer).
EINVAL fd is a cross-buffer file descriptor, and count is greater than the size of the ring buffer associated with the outbound traffic. (i.e. the o_bufsz parameter given when creating the cross-buffer).
EFBIG fd is a proxy file descriptor, and count is larger than the size of the output buffer as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).
EINVAL fd is a proxy file descriptor, and count is not a multiple of the output granularity as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).
ENXIO fd is a proxy file descriptor which is not available for output. See EVL_CLONE_OUTPUT.
This is the strict equivalent to the standard ioctl(2) system call, for sending the I/O control request from the out-of-band stage to an EVL driver. In other words, oob_ioctl() issues request to file descriptor fd from the out-of-band execution stage.
The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.
A file descriptor obtained by opening a real-time capable driver which we want to send a request to.
The I/O control request code.
An optional variable argument list which applies to request.
oob_ioctl() returns zero on success. Otherwise, -1 is returned, and errno is set to the error code:
EBADF if fd does not refer to a real-time capable driver.
ENOTTY if fd does not support the .oob_ioctl operation, or the driver does not implement request.
EFAULT if buf points to invalid memory.
EAGAIN fd is marked as non-blocking (O_NONBLOCK), and the control request would block.
Other driver-specific error codes may be returned.
This is an equivalent to the standard recvmsg(2) system call, for sending the request from the out-of-band stage to an EVL driver with a socket-based interface. In other words, oob_recvmsg() is used to receive messages from an out-of-band enabled EVL socket from the out-of-band execution stage.
The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.
A socket descriptor obtained from a regular
socket(2) call,
with the SOCK_OOB flag set in the type argument, denoting that
out-of-band services are enabled for the socket.
A pointer to a structure containing the multiple arguments to this call, which is described below.
A time limit to wait for a message before the call returns on error. The built-in clock EVL_CLOCK_MONOTONIC is used for tracking the elapsed time. If NULL is passed, the call is allowed to wait indefinitely for a message.
A set of flags further qualifying the operation. Only the following flags should be recognized for out-of-band requests:
-
MSG_DONTWAITcauses the call to fail with the error EAGAIN if no message is immediately available at the time of the call.MSG_DONTWAITis implied if O_NONBLOCK was set for the socket descriptor via the fcntl(2) F_SETFL operation. -
MSG_PEEKcauses the receive operation to return data from the beginning of the receive queue without removing that data from the queue. Thus, a subsequent receive call will return the same data.
oob_recvmsg() returns the actual number of bytes received on success. Otherwise, -1 is returned, and errno is set to the error code:
EBADF if s does not refer to a valid socket opened with the
SOCK_OOB type flag set, or s was not opened for reading.
EINVAL if s does not support the .oob_ioctl operation.
EFAULT if msghdr, or any buffer it refers to indirectly points to invalid memory.
EAGAIN s is marked as non-blocking (O_NONBLOCK), or
MSG_DONTWAIT is set in flags, and the receive operation would
block.
ETIMEDOUT the timeout fired before the operation could complete successfully.
This call is equivalent to the standard sendmsg(2) system call, for sending the request from the out-of-band stage to an EVL driver with a socket-based interface. In other words, oob_sendmsg() is used to send messages to an out-of-band enabled EVL socket from the out-of-band execution stage.
The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.
A socket descriptor obtained from a regular
socket(2) call,
with the SOCK_OOB flag set in the type argument, denoting that
out-of-band services are enabled for the socket.
A pointer to a structure containing the multiple arguments to this call, which is described below.
A time limit to wait for an internal buffer to be available for sending the message before the call returns on error. The built-in clock EVL_CLOCK_MONOTONIC is used for tracking the elapsed time. If NULL is passed, the call is allowed to wait indefinitely for a buffer.
A set of flags further modifying the operation:
-
MSG_DONTWAITcauses the call to fail with the error EAGAIN if no buffer is immediately available at the time of the call for sending the message.MSG_DONTWAITis implied if O_NONBLOCK was set for the socket descriptor via the fcntl(2) F_SETFL operation. -
If s is an out-of-band socket,
MSG_DONTROUTEtells the EVL core to check that the destination is directly reachable from the current host, i.e. no more than one hop away (aka on-link routing scope). This is a different interpretation of this flag compared to the in-band counterpart, since it won’t actually affect routing, but only make sure that the device and routing configuration you set up does allow sending data directly to the destination host. See the documentation about soliciting hosts for details.
oob_sendmsg() returns the actual number of bytes sent on success. Otherwise, no message was sent, -1 is returned, and errno is set to the error code:
EBADF if s does not refer to a valid socket opened with the
SOCK_OOB type flag set, or s was not opened for writing.
EINVAL if s does not support the .oob_ioctl operation.
EFAULT if msghdr, or any buffer it refers to indirectly points to invalid memory.
EAGAIN s is marked as non-blocking (O_NONBLOCK), or
MSG_DONTWAIT is set in flags, and the send operation would
block.
EINPROGRESS s is an out-of-band UDP socket, and no routing information is available from the EVL front caches to reach the destination host directly from the out-of-band stage. This status indicates that the request was offloaded to the in-band stage, therefore delivery time may be affected.
ETIMEDOUT the timeout fired before the operation could complete successfully.
EMULTIHOP MSG_DONTROUTE is set in flags, and the destination for the message
is more than one hop away from the local host.
This call is equivalent to the standard setsockopt(2) system call, for setting the out-of-band related options of a socket. For this reason, the set of options recognized by this call is EVL-specific.
A socket descriptor obtained from a regular
socket(2) call,
with the SOCK_OOB flag set in the type argument, denoting that
out-of-band services are enabled for the socket.
The interface level the option is meaningful at. Currently, only
SOL_SOCKET is supported.
The name of the socket-level option to set.
An opaque pointer to the new value for optname.
The length in bytes of the value pointed to by optval.
This call is equivalent to the standard getsockopt(2) system call, for retrieving the out-of-band related options of a socket. For this reason, the set of options recognized by this call is EVL-specific.
A socket descriptor obtained from a regular
socket(2) call,
with the SOCK_OOB flag set in the type argument, denoting that
out-of-band services are enabled for the socket.
The interface level the option is meaningful at. Currently, only
SOL_SOCKET is supported.
The name of the socket-level option to get.
An opaque pointer to the memory location where the value of optname
should be copied to.
The length in bytes of the value copied to optval.
The out-of-band message header
The structure oob_msghdr which is passed to the oob_recvmsg() and oob_sendmsg() calls is defined as follows:
struct oob_msghdr {
void *msg_name; /* Optional address */
socklen_t msg_namelen; /* Size of address */
struct iovec *msg_iov; /* Scatter/gather array */
size_t msg_iovlen; /* # elements in msg_iov */
void *msg_control; /* Ancillary data, see below */
size_t msg_controllen; /* Ancillary data buffer len */
int msg_flags; /* Flags on received message */
struct timespec msg_time; /* Optional time, see below */
};
struct iovec { /* Scatter/gather array items */
void *iov_base; /* Starting address */
size_t iov_len; /* Number of bytes to transfer */
};
The msg_name field points to a caller-allocated buffer that is used
to return the source address if the socket is unconnected. The caller
should set msg_namelen to the size of this buffer before this call.
On success, oob_recvmsg() updates
msg_namelen to contain the length of the returned address. If the
application does not need to know the source address, msg_name can
be specified as NULL.
The fields msg_iov and msg_iovlen describe scatter-gather
locations pointing at the message data being sent or received, as
discussed in
readv(2).
The field msg_control points to a buffer for other protocol
control-related messages or miscellaneous ancillary data. When either
oob_recvmsg() or oob_sendmsg() is called, msg_controllen should contain
the length of the available buffer in msg_control. On success,
oob_recvmsg() updates
msg_controllen to contain the actual length of the control message
sequence returned by the call.
The msg_flags field is only set on return of oob_recvmsg(). It can contain any of the flags which may
be returned by
recvmsg(2).
msg_time may be used to send or receive timestamping information
to/from the protocol driver implementing out-of-band operations.
Protocol drivers should no attach any meaning to MSG_OOB when
operating in out-of-band mode, so that no additional confusion arises
with the common usage of this flag with
recvmsg(2).
