Harness real-time performance and the rich features of Linux.

Embedded systems usually fall into one of two categories: those that require hard real-time performance and those that don’t. In the past, we had to pick our poison—the performance of our “go to” real-time operating system or the rich feature set of our favorite Linux distribution—and then struggle with its shortcomings.

Today, embedded developers no longer need to choose between the two. Asymmetric multiprocessing (AMP) offers the best of both worlds.

Several modern system-on-chip (SoC) product offerings integrate multiple CPUs, a broad variety of standard I/O peripherals and programmable logic. The Xilinx® Zynq-7000® All Programmable SoC family, for example, includes a dual-core ARM® Cortex™-A9, standard peripherals (such as Gigabit Ethernet MACs, USB, DMA, SD/MMC, SPI and CAN) and a large programmable logic array.

We can use these SoC products as the basis of a Linux/RTOS AMP system that provides considerable flexibility. In many ways, the typical AMP configuration is similar to a PCI-based system, with the Linux domain functioning as the host, the RTOS domain functioning as an adapter, and one or more shared memory regions used for communication between the two domains.

Unlike PCI, however, an AMP configuration can more conveniently—and dynamically—assign resources (both the standard peripherals and custom logic) to one domain or the other. In addition, a Linux/RTOS AMP system can dynamically reconfigure programmable logic based on runtime requirements, such as the presence or absence of various external devices.

This level of flexibility is often coupled with concerns about complexity and the degree of difficulty involved in bringing up an AMP system. Rest assured that the Linux development community has introduced many features into the kernel that greatly simplify AMP configuration and use.

LINUX MULTIPROCESSING IN A NUTSHELL

With respect to multiprocessing, the Linux kernel comes in two flavors: the uniprocessor (UP) kernel and the symmetric multiprocessor (SMP) kernel. The UP kernel can only run on a single core, regardless of the number of available cores. AMP systems can incorporate two or more instances of the uniprocessor kernel.

The SMP kernel, however, can run on one core or simultaneously on multiple cores (Figure 1). An optional kernel command line parameter controls the number of cores used by the SMP kernel following system initialization. Once

the kernel is running, various command line utilities control the number of cores assigned to the kernel. The ability to dynamically control the number of cores used by the kernel is a primary reason AMP developers prefer the SMP kernel over the UP kernel.

The Remote Processor Framework

The Remote Processor (remoteproc) Framework is the Linux component that is responsible for starting and stopping individual cores (remote processors), as well as for loading a core’s software in an AMP system. For example, we can dynamically reconfigure the SMP system shown in Figure 1 into the AMP system shown in Figure 2, and then back again to SMP, using the capabilities of remoteproc.

We can fully control reconfiguration via a user-space application or system initialization script. This allows user applications to stop, reload and run a variety of RTOS applications based on the dynamic needs of the system.

The core’s software (in our example, the RTOS and user application) is loaded from a standard Executable and Linkable Format (ELF) file that contains a special section known as the resource table. The resource table is analogous to the PCI configuration space in that it describes the resources that the RTOS requires. Among those resources is the memory needed for the RTOS code and data.

Trace buffers

Trace buffers are regions of memory that automatically appear as files in a Linux file system. As their name suggests, trace buffers provide basic tracing capabilities to the remote processor. A remote processor writes trace, debug and status messages to the buffers, where the messages are available for inspection via the Linux command line or by custom applications.

One or more trace buffers may be requested via entries in the resource table. Although they typically contain plain text, trace buffers may also contain binary data such as application state information or alarm indications.

Virtual I/O devices

The resource table also supports the definition virtual input/output devices (VDEVs), which are basically pairs of shared memory queues that support message transfer between the Linux kernel and the remote processor. The VDEV definition includes fields that negotiate the size of the queues as well as the interrupts used to signal between the processors.

The Linux kernel handles initialization of the virtual I/O queues. The software running on a remote processor need only include a VDEV description in its resource table and then use the queues once it begins execution; the kernel handles the rest.

Remote Processor Messaging Framework

The Remote Processor Messaging (rpmsg) Framework is a messaging “bus” based on the Linux kernel’s virtual I/O system. The messaging bus is similar to a local area subnetwork in which individual processors can create addressable endpoints and exchange messages, all via shared memory.

The kernel’s rpmsg framework acts as a switch, routing messages to the appropriate endpoint based on the destination address contained in the message. Because the message header includes a source address, ad hoc connections can be established between various processors.

Naming service

Processors can dynamically announce a particular service by sending a message to the rpmsg framework’s naming service. By itself, the naming service feature is only marginally useful. The rpmsg framework, however, allows service names to be bound to device drivers to support the automatic loading and initialization of specific drivers. For example, if a remote processor announces the service dlinx-h323-v1.0, the kernel can search for, load and initialize the driver bound to that name. This greatly simplifies driver management in systems where services are dynamically installed on remote processors.

Managing interrupts

Interrupt management can be a little tricky, especially when starting and stopping cores. Ultimately, the system needs to redirect specific interrupts dynamically to the remote processor domain when the remote processor is started, then reclaim those interrupts when the remote processor is stopped. In addition, the system must protect the interrupts from inadvertent allocation by potentially misconfigured drivers. In short, interrupts must be managed systemwide.

For the Linux SMP kernel, this is a routine matter—and a further reason that the SMP kernel is preferred in AMP configurations. The remote processor framework conveniently manages interrupts with only minimal support from the device driver.

Device drivers

Device driver development is always a concern because it requires a skill set that may not be readily available. Fortunately, the Linux kernel’s remoteproc and rpmsg frameworks do most of the heavy lifting; drivers need only implement a handful of standard driver routines. A fully functional driver may only require a few hundred lines of code. The kernel source tree includes sample drivers that embedded developers can adapt to their requirements.

Generic open-source device drivers are also available from vendors. DesignLinx Hardware Solutions provides generic rpmsg drivers for both Linux and FreeRTOS. Since the generic driver makes no assumptions about the format of the messages that are exchanged, embedded developers can use it for a variety of AMP applications without any modifications.

MOVING INSIDE THE PINS

The kernel’s multiprocessing support is not limited to homogeneous multiprocessing systems (systems using only the same kind of processor). All of the features described above can also be used in heterogeneous systems (systems with different kinds of processors). These multiprocessing features are especially useful when migrating existing designs “inside the pins.”

Modern SoC products let designers conveniently move various hardware designs from a printed-circuit board to a system-on-chip (Figure 3). What was once implemented as a collection of discrete processors and components on a PCB can be implemented entirely inside the pins of an SoC.

For example, we can implement the original PCB hardware architecture of Figure 3 with a Xilinx Zynq-7000 family SoC using one of the ARM processors as the control CPU and soft processors (such as Xilinx MicroBlaze™ processors) in the programmable logic to replace the discrete microprocessors. We can use the remaining ARM processor to run the Linux SMP kernel (Figure 4).

The addition of Linux to the original design provides all of the standard multiprocessing features described above for both the ARM cores and the soft core processors (such as start, stop, reload, trace buffers and remote messaging). But it also brings the broad Linux feature set, which supports a variety of network interfaces (Ethernet, Wi-Fi, Bluetooth), networking services (Web servers, FTP, SSH, SNMP), file systems (DOS, NFS, cramfs, flash memory) and other interfaces (PCIe, SPI, USB, MMC, video), to name just a few. These features offer a convenient pathway to new capabilities without significantly altering tried-and-true architectures.

THE CORES KEEP COMING

The past several years have seen an increase in multicore SoC offerings that target the embedded market and are well suited for AMP configurations.

The Xilinx UltraScale+™ MPSoC architecture, for example, includes a 64-bit quad-core ARM Cortex-A53, a 32-bit dual-core ARM Cortex-R5, a graphics processing unit (GPU) and a host of other peripherals—and, of course, a healthy helping of programmable logic. This is fertile ground for designers who understand how to harness the performance of real-time operating systems coupled with the rich feature set of the Linux kernel.

For more information on designing a Linux/RTOS AMP system, contact DesignLinx Hardware Solutions. A premier member of the Xilinx Alliance Program, DesignLinx specializes in FPGA design and support, including systems design, schematic capture and electronic packaging/mechanical engineering design, and signal integrity.