An overview of Embedded Systems Debugging
“A debugger is a computer program which may or may not be assisted by some hardware and is used to test and debug other Programs.”
Embedded Systems are resource limited complex hardware running software on some deep embedded processor with no USER interface normally. Debuggers have very old memories with software development processes. Where there is some software development, there will be bugs and when there are bugs, there will be some debuggers. Debugging in Embedded Systems emerged from native Desktop application debugging. In Desktop Computers, applications run under an OS where the program execution and debugging is under the control of OS Kernel and managed by System Calls.
When it comes to debugging Embedded Systems, the whole process can’t be seen as one big picture like in most UNIX like Systems where the link between a Tracer (program used for debugging) and a Tracee (program being debugged) is simply a matter of few System Calls like
ptrace. In Embedded Systems, due to lack of
ptrace like systems calls, the link is achieved by various techniques ranging from debug stubs/monitors to onchip debug silicon hardware.
In Embedded Systems there are two main bright sides of debuggers and debugging process – Figure-1. The focus can be narrowed down to either side depending from which perspective the debugging is analyzed. The focus of this tutorial is on left side i.e. from embedded software developer perspective though we will touch the right side briefly. In the next tutorial we will discuss how Debug events are handled by Systems Calls in UNIX like Operating Systems.
Embedded Processors usually comes with two states, i.e. Debug State and Program Execution State. In the Debug State, Processors allow to:
- Trace Execution
- Inspect Memory Contents
- Observe Core Registers
- Halt/Resume Execution
- Reset Processor
- Set Break Points and Watch Points
In Debug State there are three main agents involved and the whole debugging process is all about their inter-working.
- The Front End Program – Running on Host Computer Side
- The Back End Program – Running on Target Processor
- The Communication Link
Out of the above three, the various combination and their variants of last two has made embedded systems debugging techniques evolved over the past decades. All these techniques share same principal (See bellow “The big Picture”) and are based on the above three main agents. The main difference are improvements that made the debugging process more efficient over time.
1. Debugging Process – The big Picture
As we mentioned earlier and shown in Figure-2, the debugging process in embedded systems involves the Front End, the Back End and the communication channel between the two. The debugging process starts with the Front-End Program running on Host Computer. The Front-End program is human interface GUIs consisting of various sub-windows providing various informations like Registers contents, variables, current program execution etc. etc. on the target hardware. Common examples of such programs are Eclipse CDT, DDD, STrace etc.
The Front-End is not just a GUI, its equipped with some background debugging script/programs on top of which it runs. For example a Single-Step button will invoke Single-Stepping script/command which will be sent over the Communication Channel to the Back-End agent running on remote target – Figure-4. Before Embedded Systems, most of these Front-End assisting scripts/programs were meant to be used for native debugging i.e. debugging programs on the same computer via System Calls. Later they were equipped with cross-platform and remote debugging capabilities. One such script/program example is GNU debugger (gdb).
The Back-End is what evolved over time. It can be either in the form of a tiny software placed in internal onchip memory (either by you or manufacturer) e.g. Debug Monitor / Debug Stub (early ages approach) or it can be some debug assisting on chip hardware e.g. onchip-debugger (new approach). The purpose of Back-End is to provide low-level control of target processor debug functionalities like setting breakpoints, reading registers/memory contents, reseting processor, single-stepping etc.
Both the Front-End and Back-End agrees on a common communication protocol and set of commands/ responses which they use to perform platform specific debug operations. The purpose of Back-End is to respond to Front-End commands. For example a user may want to examine contents of a specific register. The Front-End may sends command like read register x and the Back-End will respond with register x data which will subsequently be displayed in Front-End GUI.
The Communication Medium seems to be the simples of all three main components involved in debug process but as a matter of fact this is the most complicated part especially in case where the Back-End is some Onchip Dubugging Hardware with platform specific registers and commands. In such cases it becomes very difficult to link/establish communication between two agents speaking different language. Let’s take an example of ARM onchip Coresight debug technology. If we want to debug ARM processors with GDB as Front-End, we need to do two things. First, we have to find some medium like JTAG, SWD to connect ARM Processor to Host PC. Second we will require some intermediate guy acting like a translator to translate commands back and forth between GDB and Coresight. The communication medium is discussed further later in the tutorial.
Luckily silicon vendors provide Front-Ends capable of communicating directly to the Onchip Dubugging Hardware. For example ARM Keil debugging interface provides support for various Communication Medium to the target chip like SWD, JTAG etc.
Before we can move ahead to various implementation of above “The big Picture”, let’s discuss various Debug Events that make processor to HALT / enter debug state.
2. Debug Events:
A Debug Event is any event that causes the CPU to enter debug mode thus passing control to the attached debugger for further action. This is different from UNIX like systems where upon such events control is transfered to Tracer via System calls. As embedded systems debugging emerged from Native debugging so in embedded systems the control is passed to attached debugger instead which acts like Tracer for them.
Following are some common Debug Events that cause processors to halt execution and pass control to attached debugger.
- Software Break Points
- Hardware Break Points
- Data Watchpoints
- External Debug Request
- Implementation dependent Events
Let’s be honest, what’s debugging all about??
Any Embedded Software developer who has used debuggers, of all the reasons the following two are the main ones.
- To Trace Program Execution
- To analyze processor registers and variable values at breakpoints
The first one i.e. The Trace Capability may or may not be provided by SOC onchip debugger but analysis at breakpoints is what provided by all SoC that facilitate debugging. Unfortunately many Embedded Software developers normally don’t care (or don’t have any idea) about the breakpoint they are inserting at a specific location apart from double clicking that left side of Front-End GUI that brings a red balloon sign indicating successful breakpoint insertion. In this portion we will discuss in details different types of breakpoints, their purpose and suitability.
2.1 Software Breakpoints:
As the name indicates, software breakpoints are implemented purely in software. Some people call them memory breakpoints as it is implemented by modifying memory contents as explained later. Most processor architectures either have a special purpose instruction or reserved bit in OPCODE for the debug/breakpoint purpose. For example ARMv5 has
BKPT instruction for Software breakpoints. The later case i.e. reserved bit in OPCODE is very rare as it limits the flexibility and density of instruction set. This is also not possible with variable length instructions (e.g. Thumb-2). An example of the later case implementation is one of TI C6000 architectures in which 28th bit is reserved for software breakpoint.
Whenever a breakpoint is required to be inserted (normally double clicking the left side of your debugger Front-End), the debugger pin point the location and replace the OPCODE with the BreakPoint Instruction OPCODE Figure-5 (a),(b). Whenever the breakpoint is encountered, the CPU halts and inform the Front-End/debugger. The Front-End/debugger checks if the halt point location is one on which a break was recently inserted? If the condition matches, the OPCODE at the location is replaced by the original instruction OPCODE and execution is resumed when the USER press the resume button on the Front-End. Once the CPU passes the breakpoint location, the OPCODE is again replaced with breakpoint OPCODE for halting the processor next time when it will execute instruction at the location.
The advantage of using Software breakpoints is that there is no limit on the number of breakpoint to be inserted in code. The software breakpoints can be placed as many time places as you want. The main issue is; as the Software breakpoints require modification of memory thus it can only be placed on instruction in memory with write access. For example Software breakpoints can’t be put in non-volatile memory like ROM/Flash etc. which pretty much restrict its usage to programs executing from RAM memory only (and may be hybrid memories as well like NVRAM, MRAM, FRAM etc.). Another disadvantage of Software breakpoints is that they slow downs program execution as the breakpoint OPCODE has to be replaced temporarily with original OPCODE and once the instruction is executed, the OPCODE is again replaced with breakpoint instruction OPCODE.
2.2 Hardware Breakpoints:
Hardware breakpoints are implemented into debug hardware logic. Hardware breakpoints are basically a set of programmable comparators connected to Program Address bus. Whenever a breakpoint is placed in code, the address is programmed/placed into one of these comparators. While executing code, whenever the value of Program Counter (PC)/address bus matches the comparator value, the processor is Halted and attached debugger is given control.
The advantage of using Hardware breakpoint is that it can be used on any type of memory (unlike Software breakpoints which can only be used on RAM memory) as no complex swapping of OPCODEs is required. All that is needed is just to set the breakpoint address in comparator which has nothing to do with the type of memory. Another advantage is that Hardware breakpoints are faster compared to Software breakpoints because they are hardware based bits comparison and no OPCODE swapping is required. Hardware breakpoint greatly simplify the complex method used by Software breakpoint for single-stepping.
The main limitation of Hardware breakpoints is that as they are implemented in debug hardware that’s why they are limited in number as implementing unlimited number of comparators is not possible. Usually there are 2,3,4, 8 and 16 may be if you are lucky.
2.3 Data Watchpoints:
Data watchpoints are no more different than Hardware breakpoints. They use the same comparator logic in debug hardware but this time instead of comparing with Program Counter/address bus value, the logic is connected to data bus. The main goal of watchpoints is to stop processor when the value on certain location changes i.e. situations where you are not sure where a variable value is changing. Watchpoints are very useful for tracking variable values modification.
As watchpoints uses the same logic as Hardware breakpoints thats why they inherit all the advantages and limitations of Hardware breakpoints. For example like Hardware breakpoints the watchpoints are also limited in number.
3. The big Picture Implementations:
In the above we discussed various agents involved in debugging process and described how they work together. In this portion, we will discuss various efforts made till now to make the debug process more efficient.
Note: This portion is all about the Back-End program/hardware (on the Embedded target side) discussed earlier. The Front-End is just a computer program running on host machine whose only job is to speak/communicate to the Back-End program/hardware on embedded target.
The three main debugging agents i.e. Front-End, Back-End, and the Communication Link discussed in “Debugging Process- The Big Picture” have been evolved to two main debugging techniques.
- Debugging with Debug Software
- Debugging with Debug Hardware
- On Chip Debugger
3.1 Debugging with Debug Software:
In the early days of embedded computing, embedded system didn’t had any support for embedded software debugging. Infect the hardware was so simple that it only consist of CPU, ROM/EPROM, and SRAM. The program once burned to ROM/EPROM can only be expected to be accurate if it behaved as desired, if not the developers had to analyze the whole code, recompile it, reburn it and test the behavior again. The most common debugging technique was a Blinky LED. e.g. say if variable i == 10 then blink the attached LED on a GPIO.
As time passed the memories became more and more cheap and larger in size as well. This give the embedded developers room for writing more complex software. With more complex software the conventional method of analyzing software behavior i.e. recompiling it and burning it became more tedious and new more sophisticated debugging methods development became obvious.
As Serial port made its way to embedded systems, debugging started to catch phase as developers found its way to interact with embedded software from external World. One of the first efforts made towards debugging was to place tiny piece of code called “Debug Monitors/Stub” side by side with application program code (remember Back-End Program we discussed in here).
The purpose of “Debug Monitors/Stub” was to listen to serial port commands from Front-End and respond with internal software/hardware status. For example upon reception of say “m 1234,10”, the “Debug Monitors/Stub” will send back contents of 10 memory locations starting from address 0x1234. Similarly command “g” will respond with processor internal registers values.
As time passed more and more capabilities were added to debug monitors like inserting Software Break Points, single stepping, flash programming etc. A very practical example is that of “GNU gdb stub”. The source code of debug monitors is included in gdb source code which can be ported to the target processor. The Front-End for it is gdb Front-End and has support for Serial Port, TCP, and UDP as Communication Link.
3.2 Debugging with Debug Hardware:
The Debug Monitors did served good for long time but it had few drawbacks. The most prominent one was they were slow especially the way they use to handle single-stepping. Also they require additional RAM/ROM memory and one dedicated UART which indeed was difficult on resource limited SoCs. Another drawback was that they couldn’t operate on ROM memory because they only support Software breakpoint and operate on RAM memory only due to the way Software breakpoint are inserted. Almost all modern SoCs execute code from some permanent memory like EEPROM/Flash etc. For such situations debug monitors fails or perform very poorly. As time passed the Silicon manufacturer felt the necessity of adding debug assisting silicon along with core. This give birth to two Debug Hardware methods.
In order to cope with the limitations of Debug Monitors and to make processor hardware more debug friendly, debug assisting silicon (onchip circuitry) was added to processor cores. By the time silicon and fabrication processes were expensive so two versions of processor cores were proposed… The one which will go into final product running debugged/final version of software and the other which will be used for embedded software debugging and testing. The later was called In-Circuit Emulator (ICE). ICEs were equipped onchip with proper debugging hardware while the actual processor had only processing core onchip.
The idea was that the ICE would repace the CPU in the target system at the time of debugging to debug the embedded software on ICE using its debug friendly hardware and once debugging is done, the ICE will be replaced with the original processor core.
The In-Circuit Emulators (ICEs) performed very well by providing complex breakpoints and tracing functionalities. ICEs supported Hardware breakpoints, data watchpoints and tracing facilities. The problem with ICEs was cost… They were very costly and not every once could afford them, also they were bulky in size as well. ICEs couldn’t chase the clock speed and the large number of peripherals that started integrating on processor cores. The following figure shows Motorola Exorciser ICE .
3.2.2 On-Chip Debuggers:
With the advancement of technology the chip manufacturing cost greatly reduced. The large number of low cost embedded processor made their way to local market, hobbyists, and academic research. The common availability of embedded processor cores and the cost of ICEs couldn’t remained in phase with each other. The cost of ICEs remained high while the actual processor cores remained flooded in market with on low price.
Looking at the huge demand in market, it was highly recommended that processor core designers and silicon manufacturer make debugging hardware low cost and available to common end users. The only way to reach every processor core user was to fabricate debugging hardware on each processor core (as manufacturing cost had reduced significantly). Silicon manufacturer agreed and thus the era of on chip debugger started.
Almost all processor core architectures (except few) provide on chip debug hardware facility which can be easily interfaced with debugger Front-Ends via various hardware links like JTAG, SWD, JLink etc. A well known on chip debug architecture is ARM Coresight.
The on chip debugger allows much more features like ICEs including Hardware breakpoints (limited to few), data watch points and program tracing facility.
4. Debug Information:
So Let’s say a processor hits a breakpoint, how does the Front-End knows where the breakpoint matches in C-Code? How does the Front-End brings the cursor back to the exact position in C-Code as shown in Figure-3 main window!
Well as we discussed in the Build Process tutorial that Object files contains Symbol table which correlates Symbols with LINK addresses (VMA). Based on this symbol table, debugging information is generated by compiler together with machine code and is embedded into Object file in a separate section usually named
.debug_info. These information are encoded into a predefined format depending upon Object file and is passed to the Front-End when the debug session is started. The machine code resides on the target system / or burned to the target flash while the debug information are loaded into Front-End only. When processor hits a breakpoint, the Front-End is sent information about the memory address of breakpoint. Using the debug information, the Front-End then looks for the matching High-Level code statement and bring the cursor to it to indicate the matched breakpoint location in C-code.
The debug information encoded in object files are of many formats evolved over time with various Object file formats as discussed in Build Process tutorial. This is worth mentioning that not every Object file format is capable of holding debug information. For example the old champ, the a.out is so simple that is can’t hold anything else then the basic section i.e. .text, .data, .bss and symbol table.
Out of many encoding formats the most common one is DWARF which is almost ubiquitously used today as debugging information format for ELF executables. Though DWARF is mainly designed for ELF, it can be embedded in other Object formats that allows embedded debug informations.
We will not go into inner details of DWARF internal encoding which is well explained on DWARF website . A sample DWARF encoding is shown in the bellow figure.
Another tricky Question: Can we debug embedded Software without debugging information such as DWARF and Symbol addresses?.
The answer is YES – the debug information are for human convenience to debug in High level language they find easy to understand. Without debug infromatoin and symbol addresses, the program on target processor can still be debugged but you will see your application the way the processor see it i.e. binary instruction (and assembly mnemonics which are converted from binary information by Front-End).
 – Dwarf Debugging Standard
 – Old Computers