8 Starting a process

We mentioned before that simply saying the program starts with the main() function is not quite true. Below we examine what happens to a typical dynamically linked program when it is loaded and run (statically linked programs are similar but different XXX should we go into this?).

Firstly, in response to an exec system call the kernel allocates the structures for a new process and reads the ELF file specified from disk.

We mentioned that ELF has a program interpreter field, PT_INTERP, which can be set to 'interpret' the program. For dynamically linked applications that interpreter is the dynamic linker, namely ld.so, which allows some of the linking process to be done on the fly before the program starts.

In this case, the kernel also reads in the dynamic linker code, and starts the program from the entry point address as specified by it. We examine the role of the dynamic linker in depth in the next chapter, but suffice to say it does some setup like loading any libraries required by the application (as specified in the dynamic section of the binary) and then starts execution of the program binary at its entry point address (i.e. the _init function).

8.1 Kernel communication to programs

The kernel needs to communicate some things to programs when they start up; namely the arguments to the program, the current environment variables and a special structure called the Auxiliary Vector or auxv (you can request the the dynamic linker show you some debugging output of the auxv by specifying the environment value LD_SHOW_AUXV=1).

The arguments and environment at fairly straight forward, and the various incarnations of the exec system call allow you to specify these for the program.

The kernel communicates this by putting all the required information on the stack for the newly created program to pick up. Thus when the program starts it can use its stack pointer to find the all the startup information required.

The auxiliary vector is a special structure that is for passing information directly from the kernel to the newly running program. It contains system specific information that may be required, such as the default size of a virtual memory page on the system or hardware capabilities; that is specific features that the kernel has identified the underlying hardware has that userspace programs can take advantage of.

8.1.1 Kernel Library

We mentioned previously that system calls are slow, and modern systems have mechanisms to avoid the overheads of calling a trap to the processor.

In Linux, this is implemented by a neat trick between the dynamic loader and the kernel, all communicated with the AUXV structure. The kernel actually adds a small shared library into the address space of every newly created process which contains a function that makes system calls for you. The beauty of this system is that if the underlying hardware supports a fast system call mechanism the kernel (being the creator of the library) can use it, otherwise it can use the old scheme of generating a trap. This library is named linux-gate.so.1, so called because it is a gateway to the inner workings of the kernel.

When the kernel starts the dynamic linker it adds an entry to the auxv called AT_SYSINFO_EHDR, which is the address in memory that the special kernel library lives in. When the dynamic linker starts it can look for the AT_SYSINFO_EHDR pointer, and if found load that library for the program. The program has no idea this library exists; this is a private arrangement between the dynamic linker and the kernel.

We mentioned that programmers make system calls indirectly through calling functions in the system libraries, namely libc. libc can check to see if the special kernel binary is loaded, and if so use the functions within that to make system calls. As we mentioned, if the kernel determines the hardware is capable, this will use the fast system call method.

8.2 Starting the program

Once the kernel has loaded the interpreter it passes it to the entry point as given in the interpreter file (note will not examine how the dynamic linker starts at this stage; see Chapter 9, Dynamic Linking for a full discussion of dynamic linking). The dynamic linker will jump to the entry point address as given in the ELF binary.

$ cat test.c

int main(void)
{
	return 0;
}

$ gcc -o test test.c

$ readelf --headers ./test | grep Entry
  Entry point address:               0x80482b0

$ objdump --disassemble ./test

[...]

080482b0 <_start>:
 80482b0:       31 ed                   xor    %ebp,%ebp
 80482b2:       5e                      pop    %esi
 80482b3:       89 e1                   mov    %esp,%ecx
 80482b5:       83 e4 f0                and    $0xfffffff0,%esp
 80482b8:       50                      push   %eax
 80482b9:       54                      push   %esp
 80482ba:       52                      push   %edx
 80482bb:       68 00 84 04 08          push   $0x8048400
 80482c0:       68 90 83 04 08          push   $0x8048390
 80482c5:       51                      push   %ecx
 80482c6:       56                      push   %esi
 80482c7:       68 68 83 04 08          push   $0x8048368
 80482cc:       e8 b3 ff ff ff          call   8048284 <__libc_start_main@plt>
 80482d1:       f4                      hlt
 80482d2:       90                      nop
 80482d3:       90                      nop

08048368 <main>:
 8048368:       55                      push   %ebp
 8048369:       89 e5                   mov    %esp,%ebp
 804836b:       83 ec 08                sub    $0x8,%esp
 804836e:       83 e4 f0                and    $0xfffffff0,%esp
 8048371:       b8 00 00 00 00          mov    $0x0,%eax
 8048376:       83 c0 0f                add    $0xf,%eax
 8048379:       83 c0 0f                add    $0xf,%eax
 804837c:       c1 e8 04                shr    $0x4,%eax
 804837f:       c1 e0 04                shl    $0x4,%eax
 8048382:       29 c4                   sub    %eax,%esp
 8048384:       b8 00 00 00 00          mov    $0x0,%eax
 8048389:       c9                      leave
 804838a:       c3                      ret
 804838b:       90                      nop
 804838c:       90                      nop
 804838d:       90                      nop
 804838e:       90                      nop
 804838f:       90                      nop

08048390 <__libc_csu_init>:
 8048390:       55                      push   %ebp
 8048391:       89 e5                   mov    %esp,%ebp
 [...]

08048400 <__libc_csu_fini>:
 8048400:       55                      push   %ebp
 [...]
Example 8.2.1 Disassembley of program startup

Above we investigate the very simplest program. Using readelf we can see that the entry point is the _start function in the binary. At this point we can see in the disassembley some values are pushed onto the stack. The first value, 0x8048400 is the __libc_csu_fini function; 0x8048390 is the __libc_csu_init and then finally 0x8048368, the main() function. After this the value __libc_start_main function is called.

__libc_start_main is defined in the glibc sources sysdeps/generic/libc-start.c. The file function is quite complicated and hidden between a large number of defines, as it needs to be portable across the very wide number of systems and architectures that glibc can run on. It does a number of specific things related to setting up the C library which the average programmer does not need to worry about. The next point where the library calls back into the program is to handle init code.

init and fini are two special concepts that call parts of code in shared libraries that may need to be called before the library starts or if the library is unloaded respectively. You can see how this might be useful for library programmers to setup variables when the library is started, or to clean up at the end. Originally the functions _init and _fini were looked for in the library; however this became somewhat limiting as everything was required to be in these functions. Below we will examine just how the init/fini process works.

At this stage we can see that the __libc_start_main function will receive quite a few input paramaters on the stack. Firstly it will have access to the program arguments, environment variables and auxiliary vector from the kernel. Then the initalization function will have pushed onto the stack addresses for functions to handle init, fini, and finally the address of the main function itself.

We need some way to indicate in the source code that a function should be called by init or fini. With gcc we use attributes to label two functions as constructors and destructors in our main program. These terms are more commonly used with object oriented languages to describe object life cycles.

$ cat test.c
#include <stdio.h>

void __attribute__((constructor)) program_init(void)  {
  printf("init\n");
}

void  __attribute__((destructor)) program_fini(void) {
  printf("fini\n");
}

int main(void)
{
  return 0;
}

$ gcc -Wall  -o test test.c

$ ./test
init
fini

$ objdump --disassemble ./test | grep program_init
08048398 <program_init>:

$ objdump --disassemble ./test | grep program_fini
080483b0 <program_fini>:

$ objdump --disassemble ./test 

[...]
08048280 <_init>:
 8048280:       55                      push   %ebp
 8048281:       89 e5                   mov    %esp,%ebp
 8048283:       83 ec 08                sub    $0x8,%esp
 8048286:       e8 79 00 00 00          call   8048304 <call_gmon_start>
 804828b:       e8 e0 00 00 00          call   8048370 <frame_dummy>
 8048290:       e8 2b 02 00 00          call   80484c0 <__do_global_ctors_aux>
 8048295:       c9                      leave
 8048296:       c3                      ret
[...]

080484c0 <__do_global_ctors_aux>:
 80484c0:       55                      push   %ebp
 80484c1:       89 e5                   mov    %esp,%ebp
 80484c3:       53                      push   %ebx
 80484c4:       52                      push   %edx
 80484c5:       a1 2c 95 04 08          mov    0x804952c,%eax
 80484ca:       83 f8 ff                cmp    $0xffffffff,%eax
 80484cd:       74 1e                   je     80484ed <__do_global_ctors_aux+0x2d>
 80484cf:       bb 2c 95 04 08          mov    $0x804952c,%ebx
 80484d4:       8d b6 00 00 00 00       lea    0x0(%esi),%esi
 80484da:       8d bf 00 00 00 00       lea    0x0(%edi),%edi
 80484e0:       ff d0                   call   *%eax
 80484e2:       8b 43 fc                mov    0xfffffffc(%ebx),%eax
 80484e5:       83 eb 04                sub    $0x4,%ebx
 80484e8:       83 f8 ff                cmp    $0xffffffff,%eax
 80484eb:       75 f3                   jne    80484e0 <__do_global_ctors_aux+0x20>
 80484ed:       58                      pop    %eax
 80484ee:       5b                      pop    %ebx
 80484ef:       5d                      pop    %ebp
 80484f0:       c3                      ret
 80484f1:       90                      nop
 80484f2:       90                      nop
 80484f3:       90                      nop


$ readelf --sections ./test
There are 34 section headers, starting at offset 0xfb0:

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .interp           PROGBITS        08048114 000114 000013 00   A  0   0  1
  [ 2] .note.ABI-tag     NOTE            08048128 000128 000020 00   A  0   0  4
  [ 3] .hash             HASH            08048148 000148 00002c 04   A  4   0  4
  [ 4] .dynsym           DYNSYM          08048174 000174 000060 10   A  5   1  4
  [ 5] .dynstr           STRTAB          080481d4 0001d4 00005e 00   A  0   0  1
  [ 6] .gnu.version      VERSYM          08048232 000232 00000c 02   A  4   0  2
  [ 7] .gnu.version_r    VERNEED         08048240 000240 000020 00   A  5   1  4
  [ 8] .rel.dyn          REL             08048260 000260 000008 08   A  4   0  4
  [ 9] .rel.plt          REL             08048268 000268 000018 08   A  4  11  4
  [10] .init             PROGBITS        08048280 000280 000017 00  AX  0   0  4
  [11] .plt              PROGBITS        08048298 000298 000040 04  AX  0   0  4
  [12] .text             PROGBITS        080482e0 0002e0 000214 00  AX  0   0 16
  [13] .fini             PROGBITS        080484f4 0004f4 00001a 00  AX  0   0  4
  [14] .rodata           PROGBITS        08048510 000510 000012 00   A  0   0  4
  [15] .eh_frame         PROGBITS        08048524 000524 000004 00   A  0   0  4
  [16] .ctors            PROGBITS        08049528 000528 00000c 00  WA  0   0  4
  [17] .dtors            PROGBITS        08049534 000534 00000c 00  WA  0   0  4
  [18] .jcr              PROGBITS        08049540 000540 000004 00  WA  0   0  4
  [19] .dynamic          DYNAMIC         08049544 000544 0000c8 08  WA  5   0  4
  [20] .got              PROGBITS        0804960c 00060c 000004 04  WA  0   0  4
  [21] .got.plt          PROGBITS        08049610 000610 000018 04  WA  0   0  4
  [22] .data             PROGBITS        08049628 000628 00000c 00  WA  0   0  4
  [23] .bss              NOBITS          08049634 000634 000004 00  WA  0   0  4
  [24] .comment          PROGBITS        00000000 000634 00018f 00      0   0  1
  [25] .debug_aranges    PROGBITS        00000000 0007c8 000078 00      0   0  8
  [26] .debug_pubnames   PROGBITS        00000000 000840 000025 00      0   0  1
  [27] .debug_info       PROGBITS        00000000 000865 0002e1 00      0   0  1
  [28] .debug_abbrev     PROGBITS        00000000 000b46 000076 00      0   0  1
  [29] .debug_line       PROGBITS        00000000 000bbc 0001da 00      0   0  1
  [30] .debug_str        PROGBITS        00000000 000d96 0000f3 01  MS  0   0  1
  [31] .shstrtab         STRTAB          00000000 000e89 000127 00      0   0  1
  [32] .symtab           SYMTAB          00000000 001500 000490 10     33  53  4
  [33] .strtab           STRTAB          00000000 001990 000218 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)

$ objdump --disassemble-all --section .ctors ./test

./test:     file format elf32-i386

Contents of section .ctors:
 8049528 ffffffff 98830408 00000000           ............
Example 8.2.2 Constructors and Destructors

The last value pushed onto the stack for the __libc_start_main was the initialisation function __libc_csu_init. If we follow the call chain through from __libc_csu_init we can see it does some setup and then calls the _init function in the executable. The _init function eventually calls a function called __do_global_ctors_aux. Looking at the disassembley of this function we can see that it appears to start at address 0x804952c and loop along, reading an value and calling it. We can see that this starting address is in the .ctors section of the file; if we have a look inside this we see that it contains the first value -1, a function address (in big endian format) and the value zero.

The address in big endian format is 0x08048398, or the address of program_init function! So the format of the .ctors section is firstly a -1, and then the address of functions to be called on initialisation, and finally a zero to indicate the list is complete. Each entry will be called (in this case we only have the one function).

Once __libc_start_main has completed with the _init call it finally calls the main() function! Remember that it had the stack setup initially with the arguments and environment pointers from the kernel; this is how main gets its argc, argv[], envp[] arguments. The process now runs and the setup phase is complete.

A similar process is enacted with the .dtors for destructors when the program exits. __libc_start_main calls these when the main() function completes.

As you can see, a lot is done before the program gets to start, and even a little after you think it is finished!