The ‘evl’ umbrella utility can run the set of base commands available for controlling, inspecting and testing the state of the EVL core and any command matching the ‘evl-*’ glob pattern which may be reachable from the shell $PATH variable. The way the ‘evl’ utility centralizes access to a variety of EVL-related commands is very similar to that of git on purpose. Each of the EVL commands is implemented by an external plugin, which can be a mere executable, or a script in whatever language. The only requirement is that the caller must have execute permission on such file to run it.
The general syntax is as follows:
evl [-V] [-P <cmddir>] [-h] [<command> [command-args]]
<command> may be any command word listed by ‘evl -h’, such as: ps which reports a snapshot of the current EVL threads test for running the EVL test suite trace which is a simple front-end to the ftrace interface for EVL
-P switches to a different installation path for base command
plugins, which is located at $prefix/libexec by default.
-V displays the version information then exits. The information is
extracted from the
libevl library the EVL command depends on,
displayed in the following format:
evl.<serial> -- #<git-HEAD-commit> (<git-HEAD-date>) [ABI <revision>]
where <serial> is the
libevl serial release number, the
<git-HEAD> information refers to the topmost GIT commit
which is present in the binary distribution the ‘evl’ command is
part of, and <revision> refers to the kernel ABI this binary
distribution is compatible with. For instance:
~ # evl -V evl.0 -- #1c6115c (2020-03-06 16:24:00 +0100) [requires ABI 19]
The information following the double dash may be omitted if the built sources were not version-controlled by GIT.
-h or without any argument, ‘evl’ displays this general
help, along with a short help string for each of the supported
commands found in <cmddir>, such as:
~ # evl usage: evl [options] [<command> [<args>]] -P --prefix=<path> set command path prefix -V --version print library and required ABI versions -h --help this help available commands: gdb debug EVL command plugin with GDB ps report a snapshot of the current EVL threads test run EVL tests trace ftrace control front-end for EVL
When you need to know which threads are currently present in your system, evl ps comes in handy. The command syntax - which supports short and long options formats - is as follows:
$ evl ps [-c --cpu=<cpu>[,<cpu>...]] [-s --state] [-t --times] [-p --policy] [-l --long] [-n --numeric] [-S --sort=<key>] [-h --help]
This command fetches the information it needs from the /sysfs attributes the EVL core exports for every thread it manages. The output is organized in groups of values, representing either runtime parameters or statistics for each displayed thread:
NAME reports the thread name, as specified in the evl_attach_self() library call.
CPU is the processor id. the thread is currently pinned to.
PID is the process id. inband-wise, since any EVL thread is originally a regular Linux [k]thread. This value belongs to the global namespace (i.e. task_pid_nr()).
SCHED is the current scheduling policy for the thread.
PRIO is the priority level in the scheduling policy for the thread.
ISW counts the number of inband switches. Under normal circumstances, this count should remain stable over time once the thread has entered its work loop. As the only exception, a thread which undergoes the SCHED_WEAK policy may see this counter progress as a result of calling out-of-band services. For all other scheduling policies, observing any increase in this value after the time-critical loop was entered is a sure sign of a problem in the application code, which might be calling real-time unsafe services when it should not.
CTXSW counts the number of context switches performed by the EVL core for the thread. This value is incremented each time the thread resumes from preemption or suspension in the EVL core. CAUTION: this has nothing to do with the context switches performed by the main kernel logic.
SYS is the number of out-of-band system calls issued by the thread to the EVL core. Basically, this is the number of out-of-band I/O requests the thread has issued so far, since everything is a file in the EVL core.
RWA counts the number of Remote WAkeup signals the EVL core had to send so far for waking up the thread whenever it was sleeping on a remote CPU. For instance, this would happen if two threads running on different CPUs were to synchronize on an EVL event. Waking up a remote thread entails sending an inter-processor interrupt to the CPU that thread sleeps on for kicking the rescheduling procedure, which entails more overhead than a local wakeup. If this counter increases like crazy when your application runs, you might want to check the situation with respect to CPU affinity, to make sure the current distribution of threads over the available CPUs is actually what you want.
STAT gives an abbreviated runtime status of the thread as follows:
TIMEOUT is the remaining time before some timer which was started specifically for the thread fires. Which timer was started depends on the undergoing operation for such thread, which may block until a resource is available, wait for the next period in a timeline and so on. See STAT for details.
%CPU is the current amount of CPU horsepower consumed by the thread over the last second. When an out-of-band interrupt preempts a thread, the time spent handling it is charged to that thread.
CPUTIME reports the cumulated CPU time already consumed by the thread, using a minutes:milliseconds.microseconds format.
The command options allow to select which threads and which data should be displayed:
-c option filters the output on the CPU the threads are
pinned on. The argument is a comma-separated list of CPU
numbers. Ranges are also supported via the usual dash separator. For
instance, the following command would report threads pinned on CPU0,
and all CPUs from CPU3 to CPU15.
$ evl ps -c 0,3-15
-s includes information about the thread state, which is ISW,
CTXSW, SYS, RWA and STAT.
-t includes the thread times, which are TIMEOUT, CPU% and CPUTIME.
-p includes the scheduling policy information, which is SCHED and
-l enables the long output format, which is a combination of all
-n selects a numeric output for the STAT field, instead of the
one-letter flags. This actually dumps the 32-bit value representing
all aspects of a thread status in the EVL core, which contains more
information than reported by the abbreviated format. EVL hackers
-S sorts the output according to a given sort key in increasing
order. The following sort keys are understood:
For instance, the following command would list the times of all threads from CPU14 by decreasing CPU time consumption:
$ evl ps -t -Srt -c14 CPU PID TIMEOUT %CPU CPUTIME NAME 14 2603 - 0.6 00:435.952 rtup_ufpp14-5:2069 14 2604 - 0.6 00:430.147 rtup_ufpp14-6:2069 14 2599 - 0.5 00:423.118 rtup14-3:2069 14 2600 - 0.5 00:420.293 rtup14-4:2069 14 2595 - 0.3 00:207.143 [rtk1@14:2069] 14 2597 - 0.3 00:204.301 [rtk2@14:2069] 14 2619 - 0.2 00:186.139 rtuo_ufpp14-14:2069 14 2617 - 0.2 00:185.497 rtuo_ufpp14-13:2069 14 2623 - 0.2 00:184.812 rtuo_ufpp_ufps14-18:2069 14 2622 - 0.2 00:184.772 rtuo_ufpp_ufps14-17:2069 14 2621 - 0.2 00:181.692 rtuo_ufps14-16:2069 14 2616 - 0.2 00:181.329 rtuo14-12:2069 14 2615 - 0.2 00:181.230 rtuo14-11:2069 14 2620 - 0.2 00:180.604 rtuo_ufps14-15:2069 14 2572 - 0.0 00:000.006 rtus_ufps13-9:2069 14 2125 - 0.0 00:000.006 rtus1-7:2069 14 2646 - 0.0 00:000.005 rtus15-8:2069 14 2650 - 0.0 00:000.005 rtus_ufps15-10:2069 14 2310 - 0.0 00:000.005 rtus6-7:2069
-hdisplays the command help string.
The trace command provides a simple front-end for controlling the function tracer which is part of the FTRACE kernel framework, in a way which is EVL-aware.
In order to use this tracer, make sure to enable the following features in your kernel build:
The command syntax is as follows:
evl trace [-e [-s <buffer_size>]] [-d] [-p] [-f] [-h] [-c <cpu>] [-h]
The command options allow for a straightforward use of the function tracer:
-e enables the tracer in the kernel. From this point, FTRACE
starts logging information about a set of kernel functions which may
be traversed while the system executes. By default, this switch only
enables tracing for out-of-band IRQ events, CPU idling events, and
all (in-kernel) EVL core routines. If a particular CPU is mentioned
-c along with
-e, then per-CPU tracing is enabled for
-f is mentioned, all kernel functions traversed in the course
of execution are logged, not only the minimal subset enabled by
-e. CAUTION: enabling full tracing may cause a massive
-s changes the size of the FTRACE buffer on each tracing CPU to
<buffer_size>. If a particular CPU is mentioned with
-c along with
-s, then the change is applied to the snapshot buffer of <cpu>
-d fully disables the tracer which stops logging events on all
-p prints out the contents of the trace buffer. If a particular
CPU is mentioned with
-c along with
-p, then only the snapshot
buffer of <cpu> is dumped.
-h displays the command help string.
For instance, the following command starts tracing all kernel routines:
$ evl trace -ef
This command is a short-hand for running the EVL test suite. The usage is as follows:
$ evl test [-l][-L][-k] [test-list]
With no argument, this command runs all of the tests available from the default installation path at $prefix/tests:
$ evl test duplicate-element: OK monitor-pp-dynamic: OK monitor-pi: OK clone-fork-exec: OK ...
You can also chose to run a specific set of tests by mentioning them as arguments to the command, such as:
$ evl test duplicate-element monitor-pi duplicate-element: OK monitor-pi: OK
You may ask for listing the available tests instead of executing them,
by using the
$ evl test -l duplicate-element monitor-pp-dynamic monitor-pi clone-fork-exec ...
In a variant aimed at making scripting easier, you can ask for the absolute paths instead:
$ evl test -L proxy-pipe mapfd /usr/evl/tests/proxy-pipe /usr/evl/tests/mapfd
If some test goes wrong, the command normally stops
-k would allow it to keep going until the end
of the series.
You can implement your own ‘evl’ command plugins, which may be located anywhere provided it is reachable from the shell PATH variable with the proper execute permission bit set. EVL comes with a set of base plugins available from $prefix/libexec (*). The latter directory is implicitly searched for the command after the PATH variable was considered, which means that you may override any base command with your own implementation whenever you see fit.
Crash course: adding the ‘foo’ command script to ~/tools
$ mkdir ~/tools $ cat > ~/tools/evl-foo #! /bin/sh echo "this is your 'evl foo' command" ^D $ chmod +x ~/tools/evl-foo $ export PATH=$PATH:~/tools $ evl foo this is your 'evl foo' command
In addition, ‘evl’ sets a few environment variables before calling a plugin. Your plugin executable/script can retrieve them using getenv(3) from a C program, or directly dereference those variables from a shell:
|EVL_CMDDIR||Where to find the base plugins||$prefix/libexec|
|EVL_TESTDIR||Where to find the tests||$prefix/tests|
|EVL_SYSDIR||root of the /sysfs hierarchy for EVL devices||/sys/devices/virtual|
|EVL_TRACEDIR||root of ftrace hierarchy||/sys/kernel/debug/tracing|
(*) may be overriden using the
Last modified: Fri, 13 Mar 2020 12:44:32 CET