DIAPM-RTAI

(Dipartimento di Ingegneria Aerospaziale, Politecnico di Milano - Real Time Application Interface)

A Hard Real Time support for LINUX

  
to RTAI  
home page 
  

History and motivation for a Real Time Application Interface (RTAI)

It all started with a variant to NMT_RTLinux that grew out of the need of having a periodic scheduler to enhance efficiency in control applications, when one can work with a basic period and integer multiples of it. In fact the original RTLinux release had just a oneshot timer that, while very flexible in getting close to microsecond timing resolutions, required much overhead in keeping the clock time and in being reprogrammed. That is related to the use of the 8254 timer, which is connected to the ISA bus, that is becoming more and more a dragging weight in relation to the CPU power of today's PCs. Moreover the initial release had no real time support functions, i.e. semaphore, timing functions, messages etc., and that made impractical the implementation of relatively complex control applications requiring a few cooperating tasks.
Ironically at some time we found ourselves in the need of using oneshot precise heterogeneous timers to implement PWM control systems at medium frequencies that, if done within the RTLinux tasks, would have allowed much more flexibility than an hardwired implementation. Again however the original release maintained its architecture and its overhead remained excessive. So we implemented the oneshot timers in a different way by using the CPU TSC (Time Stamp Clock) with far great efficiency, but with earlier than Pentium machines, and compatibles, no more usable.
Another important reason for a variant was the fact it solved the problem of a bug free FPU support that , for some time, has been lacking in the official release.
Right now the official RTLinux has solved most of these problems and has a periodic timer too, along with semaphores and mailboxes. However the oneshot timing is still less efficient.
With the releases for Linux 2.2.xx we went back to what we were thinking before RTLinux appeared, i.e. develop  Real Time Hardware Abstraction Layer (RTHAL)  onto which  a Real Time Application Interface (RTAI) could be mounted to make Linux usable for hard real time applications.
Unfortunately kernel 2.0.25, on which we were working at the very beginning of the all story was to much unclean in design, for the idea we had in mind.  Thus  we were lucky that people at NMT introduced RTLinux, as it gave us the possibility of gradually learning about the kernel and its relation to the hardware.
So when 2.2.xx appeared to have a cleaner interface to the hardware we could go back to our original idea, but with a somewhat more deeper understanding of what was behind.
What has come out is a comprehensive Real Time Application Interface (RTAI), usable both for  uniprocessors (UP) and for symmetric multi processors (SMP),  that allows the use Linux kernel 2.2.xx for many hard real time applications.
SMP tasks are defaulted to work on any cpu but you can assign them to any subset,  or even to a single cpu, by using the function "rt_set_runnable_on_cpus".  It is also possible to assign any real time interrupt service to a specific  cpu by using "rt_assign_irq_to_cpu" and "rt_reset_irq_to_sym_mode".
Thus a user can statically optimize his/her application if he/she believes that it can be better done than by using a symmetric load distribution.  The possibility of forcing any interrupts to a specific cpu is clearly not  related to the smpscheduler and can be used also with interrupt handlers alone.
Note that only the real time interrupt handling is forced to a specific cpu.  That means that if you check this feature by using "cat /proc/interrupts" for  a real time interrupt that is chained to Linux, e.g the timer when rtl_sched is  installed, you can still see some interrupts distributed to all the cpus, even  if they are mostly on the assigned one. That is because Linux interrupts are  kept symmetric by the RTAI dispatcher of Linux irqs.
The schedulers allow to chose between a periodic and a oneshot timer, not to be used together. The periodic ticking is less flexible but, with the usual PC  hardware much more efficient. So it is up to you which one to choose in  relation to the applications at hand.
It should be noted that in the oneshot mode the time is measured on the base of the CPU time stamp clock (TSC) and not on the 8254 chip, which is used only  to generate oneshot interrupts. The periodic mode is instead timed by the 8254 chip only.
In this way slow I/Os to the ISA bus are limited as much as possible with a sizeable gain in efficiency. The oneshot mode has just about 15-20% more  overhead than the periodic one. It is likely that local APIC timers could lead  to a further improvement.
Right now local APIC timers are hard disabled on UPs and a preliminary experience  with with a single SMP local APIC timer, to be released soon for SMP, shows that there is no performance improvement for a periodic scheduling while the oneshot case gain is sizeable, but not so large with respect to the already available  solution.
In fact by using the TSC just two outb are required to reprogram the 8254,  i.e 2.5 us approximately, against almost nothing for the APIC timer.  However you have to broadcast a message to all the cpus in any case, and that is about 2 us, the APIC bus is an open drain 2 wires one and is not lightning like.  Note that the performance loss of the 8254 is just a fraction of the overall task switching procedure, which is always substantially heavier in the oneshot case than in periodic mode.
Since the TSC is not available on 486 machines, for them we use a form of  emulation of the "read time stamp clock" (rdtsc) assembler instruction based on counter2 of the 8254. So you can use RTAI also on such machines. Be warned that  the oneshot timer on 486 is a performance overkill because of the need of  reading the tsc, i.e. 8254 counter2 in this case, twice. That can take 6-8 us,  i.e. more than it takes for a full switch among many tasks while using a  periodic timer. Thus only a few Khz period is viable for real time  tasks if you want to keep Linux alive.
No similar problems exist for the periodic timer that needs not to use any TSC at all.  So, compared to the 20% cited above, the real time performance ratio of the  oneshot/periodic timer efficiency ratio can be very low on 486 machines.  Moreover it will produce far worse jitters than those caused on Pentiums and upward machines. If you really need a oneshot timer buy a Pentium, at least.  Instead, for a periodic timing,  486s can be still more than adequate for  many applications.
A feature of our RTAI implementation is that interrupt handlers preambles take  care natively of the task switched (TS) flag. Thus you can freely use floating point operations in your interrupt handlers, without causing a trap fault whatever thing Linux is doing.
RTAI is thus very suitable to trap interrupts without minding of Linux so that you can effectively interact with the bare PC hardware much in the same  way I, and maybe many of you, liked the old good time of DOS.
With RTAI you have the added advantage that Linux maintains all of its features  untouched so that you can pass to it whatever you get from your handler for  logging, displaying and postprocessing, by using fifos and/or shared memory.
Imagine a remote controller at 10 Khz, +-5 us average interrupt uncertainty, connected  through the internet, with  all the bells and whistles of X and its  applications. It is an application we simulated easily. See the "examples/timer" directory in examples for some clues, without remote  control however.
Note that RTAI has also some very useful system services,  including: timings, semaphores, messages and remote procedure calls (RPC). That  makes it easier to develop complex real time applications. RPCs are a limited  form of QNX messages as they pass either just an unsigned integer or a pointer  to an unsigned integer for reason of efficiency. They can be easily changed to  be fully compatible with QNX if you'd like.
There are many significant examples that allow you to experiment with forcing  tasks and timer interrupts to any cpu, a summary of cpu usage is printed at  examples module rmmoding.

Many warm thanks are due to Emanuele Bianchi, Davide Martini, Lorenzo Dozio,  Maurizio Quadrio and Daniele Danlugli for their cooperation, suggestions and  precious help in various jobs using Linux for hard real time control systems.
Emanuele, Davide and Lorenzo did a nice work to verify and test our RTLinux  variant of NMT-RTLinux in production work for their thesis and following research works at DIAPM.
This helpful html documentation is almost entirely due to Emanuele, Davide and Lorenzo.
Lorenzo played also a very important role in retrofitting our efficient oneshot timers to the then purely periodic timing control, because of the need to use them to easily pulse width modulate air jets emulating on-off control rockets.
Maurizio is one of the top Linux experts at DIAPM and has been  our "great saver"  during some deadly disk losses occurred while developing and using RTL and RTAI
Daniele has been very fundamental in the past scheduler development,  especially in relation to fpu support. He was the first one to point out  unbearable jitters when the fpu was used in RTLinux and in its DIAPM variant.  Without his convincing cries many people, including ourselves, would never have  known that they were using the fpu just to generate troubles in their  applications. It is because of that that the fpu support is included natively  also for RTAI interrupt handlers.
For other information on RTAI see the files INSTALL and README in the rtai distribution.