Build Process – From ‘C’ to Machine Code 0’s 1’s

Keywords: Embedded systems, Build Process, Toolchain, Compiler, Object Files


One of the major difference between Desktop application programming and embedded system programming is the assumptions that the toolchains/build tools make about target hardware. In case of Desktop application, the assumptions made by toolchain is due to high consistency and legacy support in both Desktop computer Processors (CISC) and Operating Systems. And most importantly its the OS build (keeping in view of target hardwares) that hides the low level details of underlaying hardware thus providing an abstract layer for the upper applications running on the hardware. This provides sort of relaxation to desktop application developer to rarely think about underlaying hardware or the memory available in the system. Desktop programmers usually take into account the target OS on which the application is going to run but again the underlaying toolchain takes care of these details thus making the build process more automated.

Embedded Systems hardware are usually not only resource limited but also tightly coupled with the processor i.e. fixed and non expendable. Embedded System programming is not substantially different from normal/desktop programming. The only thing that makes the difference is that in case of embedded systems each target hardware is unique [1].

Due to the uniqueness in terms of architecture and silicon manufacturing implementation, the software development tools make almost 0 to least assumption about the target hardware! The programmer usually provides very low level details in order for the development tool to generate correct code!

The Job of a compiler is to translate high level language to machine executable code. In this tutorial we will discuss build process steps in general and some additional steps (used in embedded systems) in particular. The path from source code to binary executable image is a bit steep in case of embedded systems.

Figure-1: Compiler Job

The Compilation process is a multi stage process. Though we will not go into the history of how each step in compilation process evolved, we will surely explain what each step exactly does.


1. Compilation Stages:

Generally ‘C’ compilation is a 5-stages process with the 5th stage normally merged with 4th-stage. At each stage only one Compiler COMPONENT is active. The output of One stage is input to another stage. The code flow is unidirectional i.e. once a stage finishes its job, its never called back for the same piece of Code. In Compiler terms the stages are called PASSES e.g. Assembler Pass, Linker Pass etc.

Remember, the Build Process (not the target binaries) is independent of underlying Operating System BUT the Output format of the final Pass is actually OS dependent, unless otherwise  specifically mentioned . For example on Windows the common file formats are .exe, .dll etc. while on Unix like systems the output format will be like .out, .elf, or simply no extension.

All modern IDEs normally hide compilation stages and other minor details. They come pre-targets’ configured or require very minor configuration to reduce whole build process to a single button press. The following figure summarizes all Compilation Stages.

Figure-2: Compilation Stages

Let’s get into details of each stage. The toolchain considered for tutorial is opensource GCC ‘C’ Compiler.


1.1 Preprocessor:

This is the first pass of Compilation process. The input to this stage is off course ‘C’ source code. The output of this pass is PURE ‘C’ code with the following properties:

  • All ‘C’ MACROS and Pre-Processors #define#if etc. are replaced by their substitutions.
  • All included header files (#include) source code is added to the source file.
  • All comments are removed.
  • The output file of Preprocessor is of .i extension.

Tip: If you ever come across some strange compilation error like expected ‘;’ at line #XYZ and you go to that line and see – wow every thing is okay at line #XYZ. In such cases remember to check your Preprocessor OUTPUT. You might have defined a macro mistakenly ending it with ‘;’.

Let’s take an example of hello.c file with the ‘C’ code given below.

/*
   @file name:  hello.c
*/
#define    CONST_NUM    100

int main() {
    /*
      define variables
    */
    int num = 10;
    int sum = 0;
	
    /*
      sum of variable with constant macro
    */
    sum = num + CONST_NUM;
	
    return 0;
}

In GCC the preprocessor output can be generated via the -E switch. The pre-processor output of above hello.c file code is given bellow.

gcc -E hello.c -o hello.i

The -E switch invokes only the pre-processor stage, the -o switch specify the name of output file to be generated.

int main() {

    int num = 10;
    int sum = 0;

    sum = num + 100;
	
    return 0;
}

As can be seen in above code all the comments are removed and pre-processor macros are replaced by their numerical value.

The way input files are processed by preprocessor is dependent on source file extension. The following Table summarizes preprocessor behavior to input source file [3].

*.ii Description
*.c C source code which must be preprocessed.
*.i C source code which should not be preprocessed.
*.ii C++ source code which should not be preprocessed.
*.h C header file (not to be compiled or linked).
*.cc
*.cp
*.cpp
*.c++
*.C
C++ source code which must be preprocessed.
*.S Assembler code which must be preprocessed.

Table-1: Preprocessor response to various file Extensions

1.2 Compiler:

This is the second pass of compilation process. It takes the output of the preprocessor and generates assembly language source code. This PASS need to be informed for which Architecture Assembly instructions the source code needs to be converted to i.e. x86, x386, ARM, PowerPC etc. This is normally set to the underlaying hardware on which it runs (Native Compilation) or set as a default for specific Processor architecture or simply configured by end user as per requirements. When the Compiler is configured to generated assembly instruction for the Processor Architecture that is not the one on which the compiler is running, the compilation is called Cross Compilation.

In case of GNU Compiler, the compiler can be generated via -S switch. The Compiler output of the above hello.i is given bellow.

gcc -S hello.i -o hello.s
        .file   "hello.c"                                                                                                                              
        .text                                                                                                                                          
        .globl  main                                                                                                                                   
        .type   main, @function                                                                                                                        
main:                                                                                                                                                  
.LFB0:                                                                                                                                                 
        .cfi_startproc                                                                                                                                 
        pushq   %rbp                                                                                                                                   
        .cfi_def_cfa_offset 16                                                                                                                         
        .cfi_offset 6, -16                                                                                                                             
        movq    %rsp, %rbp                                                                                                                             
        .cfi_def_cfa_register 6                                                                                                                        
        movl    $10, -4(%rbp)                                                                                                                          
        movl    $0, -8(%rbp)                                                                                                                           
        movl    -4(%rbp), %eax                                                                                                                         
        addl    $100, %eax                                                                                                                             
        movl    %eax, -8(%rbp)                                                                                                                         
        movl    $0, %eax                                                                                                                               
        popq    %rbp                                                                                                                                   
        .cfi_def_cfa 7, 8                                                                                                                              
        ret                                                                                                                                            
        .cfi_endproc                                                                                                                                   
.LFE0:                                                                                                                                                 
        .size   main, .-main                                                                                                                           
        .ident  "GCC: (GNU) 7.2.1 20170915 (Red Hat 7.2.1-2)"                                                                                          
        .section        .note.GNU-stack,"",@progbits

1.3 Assembler:

This is the third pass of compilation process. The job of assembler is to convert assembly language to binary Object file containing machine language 0’s and 1’s. The GNU C/C++ assembler is called (as).

What are Object Files?

The code normally contains both instructions to be executed by Processor and data (variables etc.) to be processed. Normally instruction are executed from permanent memory like EEPROM/Flash while data is placed in volatile memory like RAM as it changes during program execution. The two type memories i.e. (ROM, RAM) doesn’t overlap in Processor MEMORY MAP. So for efficient storage, analysis and execution (making it easy for OS/run time libraries to load it in RAM).The whole program is divided into various sections and organized in a file Data Structure called “Object File”.

Object files and executable come in several formats each evolved over time for different Operating Systems (OS). The following table [3] shows common Object file formats.

Object file FormatsDescription
a.outThe a.out format is the original file format for Unix. It consists of three sections: text, data, and bss, which are for program code, initialized data, and uninitialized data, respectively. This format is so simple that it doesn’t have any reserved place for debugging information. The only debugging format for a.out is stabs, which is encoded as a set of normal symbols with distinctive attributes.
COFFThe COFF (Common Object File Format) format was introduced with System V Release 3 (SVR3) Unix. COFF files may have multiple sections, each prefixed by a header. The number of sections is limited. The COFF specification includes support for debugging but the debugging information was limited. There is no file extension for this format.
ECOFFA variant of COFF. ECOFF is an Extended COFF originally introduced for Mips and Alpha workstations.
XCOFFThe IBM RS/6000 running AIX uses an object file format called XCOFF (eXtended COFF). The COFF sections, symbols, and line numbers are used, but debugging symbols are dbx-style stabs whose strings are located in the .debug section (rather than the string table). The default name for an XCOFF executable file is a.out.
PEWindows 9x and NT use the PE (Portable Executable) format for their executables. PE is basically COFF with additional headers. The extension normally .exe.
ELFThe ELF (Executable and Linking Format) format came with System V Release 4 (SVR4) Unix. ELF is similar to COFF in being organized into a number of sections, but it removes many of COFF’s limitations. ELF used on most modern Unix systems, including GNU/Linux, Solaris and Irix. Also used on many embedded systems.
SOM/ESOMSOM (System Object Module) and ESOM (Extended SOM) is HP’s object file and debug format (not to be confused with IBM’s SOM, which is a cross-language Application Binary Interface – ABI).
Table-2: Common Object File formats

As mentioned earlier, Object Files contains various section. The sections allowed depends on the Object File format mentioned in Table-2. The most common sections and ones shared by Object Files are given bellow.

Figure-1: Object file Sections/Segments
Section NameDescription
.textThis section contains the executable instruction codes and is shared among every process running the same binary.
.bssBSS stands for ‘Block Started by Symbol’. It holds un-initialized global and static variables.
.dataContains the initialized global and static variables and their values.
Symbol TableA symbol is basically a name and an address. Symbol table holds information needed to locate and relocate a program’s symbolic definitions and references. A symbol table index is a subscript into this array. Index 0 both designates the first entry in the table and serves as the undefined symbol index. The symbol table contains an array of symbol entries.
Table-3: Object file Sections

1.4 Linker/Linking:

In compilation process each file is compiled and assembled separately and substantially each file produces separate Object file. The job of linker is:

  • To merge all input object files into one single Object File.
  • To resolve all unresolved symbols.

The Output of Linker is a single file that contains all the sections merged into respective sections i.e. by merging all .text, .data, .bss etc. sections of all input files into one single Object file contains one .text, one .data and one .bss section respectively – Figure-3,4. The output of Linker is relocatable image/Object file. Relocatable image means all the input object files have been successfully merged but no REAL addresses have been assigned so far to variables and instructions in symbol table. The symbol table contains entries for symbols like: foo variable is located at offset 18 of the output .data section. i.e. all entries are relative to sections beginning or some reference address. The relocatable image is not executable.

Figure-3: Compilation Process – Linker Job
Figure-4: Linker Output – Relocatable Image

The GNU C/C++ compiler linker tool is called (ld).

1.5 Locator:

This is the last step of compilation process. Locator main job is to assign load address to the relocatable image generated by Linker. Locator is normally a part of Linker (e.g. like in case of GNU C/C++ Compiler) but can also be a separate entity of toolchain. This is the part of toolchain that almost assume nothing about the target hardware memory. This Part of toolchain mainly deals memory map rather than the architecture itself. Locator need to be informed how to assign addresses to various sections. If no information is given to Locator, it assigns addresses based on Object file type discussed in Table-1.

In case of embedded systems, due to huge versatility of hardware interfaces with memories and memory mapped devices, a script called Linker Script (in case of GNU C/C++ Compiler) is used to instruct Locator how to assign addresses. We have a detail tutorial on GNU Linker script with link given bellow.

The output of Locator is executable images which can be directly loaded into target hardware memory for running on target machine.

References:

[1] – Programming Embedded Systems in C and C++ by Michael Barr

[2] – Linker and Loader by John R. Levine

[3] – C Build Process Stages



4,903 thoughts on “Build Process – From ‘C’ to Machine Code 0’s 1’s”