Ideas in Wiring

Tracing Software During Hardware Test

In my last post, I mentioned a need to find software faults in microprocessor-controlled embedded systems. Since I'm a software guy, I'm curious about this angle and started doing some reasearch.

1) The faults that were mentioned (timing, memory leaks, crashes) are basically software domain problems but they may only be triggered by certain I/O inputs the system receives (and the sw ends up dealing with). So, it looks like the main tools to diagnose the fault will need to be ones that deal with the software domain but the system may need to be driven from the hardware I/O domain.

2) These challenges are similar to those found in other applications that use microprocessor-driven control systems. For example, NI makes hardware that can be used for real-time process control (such as controlling valves in a paper mill). The controllers need to run 24x7, just like a radio, and the entire system is controlled by the microprocessor. NI needs to make those controllers robust and therefore we need to test them extensively.

3) The automotive industry also uses microprocessor-based high speed control systems, especially to control the "powertrain" (engine & transmission). They use a technique called Hardware in the loop testing. The real controller (ECM) that is to be mounted in the car is hooked to a "simulator". The ECM sends out real output signals using a real wiring harness to the simulator, the simulator then looks at the outputs from the ECM and calculates what the real engine would do and then sends signals back to the ECM that make it think it is talking to a real engine. You then "drive" the ECM as if you were driving the car through various scenarios and the simulator checks to make sure all of the outputs from the ECM are valid. At the same time as the simulator is running, they monitor the software running on the ECM to verify that it is operating properly. If the ECM crashes, they have tools to tell them what happened in the software. It would seem like you could do exactly the same thing with software defined radio test.

4) In the auto industry, they typically don't have an OS and they do the tracing at a low level. In the old days, this was done with a chip emulator. The emulator (or logic analyzer) would snoop the address and data bus to monitor execution. Now that caches are so prevalent, the external bus is not a good place to monitor. Chips that Freescale makes for automotive such as the MPC565 have a port called NEXUS. This port works in conjunction with information coming out on the address/data bus to spit out information while the processor is running that gives visibility into the processor pipeline. You can tell that a branch was taken, not taken, etc.. even if the instruction stream is all in the cache. There are boxes such as the Lauterbach ICD Trace that hook up to the chip for this purpose. Their tools can then show the full execution history, what led up to the processor crash, etc. WindRiver, GreenHills, and other emulator vendors also have this capability.

It would seem like this concept could be applied to Software Defined Radio. A Radio Emulator could drive the radio under test while the radio is being monitored with a tool like this. In the automotive case, I don't know how much this trace data ends up getting synchronized to the operation of the HIL emulator. It might be critical to share control or timing information between the radio emulator and the trace tool so that you can correlate the waveform signals precisely to the line of code being executed. This is where I think LabVIEW would come in. HIL simulators provided by NI partners use LabVIEW as their base and are open to be modified by the end user. LabVIEW has the ability to work with other software tools via ActiveX and other interfaces. It should be relatively easy to get LabVIEW to control both the HIL simulator and the trace tool.

5) Since SDRs typically run high function OSes like GreenHills Integrity or WindRiver VxWorks, there are some additional tools that would be useful during debug. WindRiver has a tool called MemScope that will monitor the heap so you can see memory leaks. LabVIEW could control this tool also. GreenHills has a leak checker but I haven't had any luck finding out more details on what it does.

I'm looking into setting up this type of system in our lab. We have a reliability lab here for testing our real time controllers and we're looking at adding a tool like MemScope (since we use VxWorks in our controllers) to our testing arsenal. We may add on a trace tool as the next step after that.

Labels: hardware in the loop, Hardware Test, Software Test, SoftwareDefinedRadio

posted by Joel @ 9:00 AM