Skip to content

Reverse engineering

Quick References

  • Introduction to RE

  • Why RE is necessary (no-support/source/malware)

  • IA-32 CPU architecture
    • General Purpose Registers (V1 C3.4.1):
      • EAX - accumulator
      • EBX - pointer to data in DS segment
      • ECX - counter for string and loop operations
      • EDX - I/O pointer
      • ESI
        • Pointer to data in the segment pointed to by the DS register
        • Source pointer for string operations
      • EDI
        • Pointer to data or destination in the segment pointed to by the ES register
        • Destination pointer for string operations
      • ESP - Stack pointer, in the SS segment
      • EBP - Pointer to data on the stack, in the SS segment
    • Accessing parts of registers (EAX AH AL)
    • EFLAGS (V1 C3.4.3)
      • Arithmetic: OF SF ZF AF PF CF
      • ZF
        • SCAS
      • DF (direction flag)
        • 1: String instructions auto-decrement (processed from higher to lower address)
        • 0: String instructions auto-increment (processed from from lower to higher address)
        • cld to **cl**ear **d**irection flag
        • std to s**e**t **d**irection flag
        • String operation
          • ESI/EDI: must point to either to start/end of the string
          • ECX must contain number of bytes to compare
      • IOPL (IO **p**rivilege **l**evel) - OS operations
      • Trap Flag - Enable single step mode (debugging)
    • Segment Registers (V1 C3.4.2)
      • CS - code
      • DS - data
      • SS - stack
      • ES - data
      • FS - data
      • GS - data
    • EIP - Instruction Pointer Register = PC (V1 C3.5)
      • Cannot access directly
    • DR0 - DR7: Debug Registers (V3 C317.2)
      • DR0 - DR3 are important: Used to store hardware breakpoints (addresses)
      • Cannot be accessed directly from userland (Ring-3)
      • In Windows use Ring 3 API and transfer execution to kernel level to update the register
    • Machine/Model Specific Registers (V1 C3.2)
      • Not accessible to applications, except from Time-Stamp Counter
      • 64-bit
      • Read using RDTSC (Read time-stamp counter)
        • Low read to EAX
        • High read to EDX
      • Incremented with every clock cycle (reset when processor reset)
  • CPU has a thread-wise context
  • Software breakpoints work by 0xCC (int 3h) instruction. Core modification, is independent from thread context.
  • Functions
    • Why functions are required in a program
    • Execution flow changes and it's required to remember where the execution was left off (and registers)
    • Introduce stack (SS)
  • Stack:
    • Process:
      • Contain threads
      • Created by OS and provided with a virtual address space
      • OS creates at least one thread, ready to be executed
    • Thread:
      • Has it's own stack
      • Share the same virtual address space (of process)
      • Time slice assigned by processor to that specific process
      • Singe processor system: One thread at a time
      • Multi processor system: Simultaneous threads
    • Multi-tasking: Illusion of simultaneous execution of multiple application at the same time
    • Multi-threading: Multiple threads that actually run in parallel
    • PUSH POP - LIFO and bottom to top nature
      • ESP always point to TOS is decremented to point to new place
        • Addition decrement
        • Removal increment
  • Function calling (V1 C6):
    • Parameters pushed to stack
    • CALL
      • Push return address to (return IP) ti stack
      • Load the address of the function's entry-point (to EIP)
    • RET
      • POP address from TOS to EIP and resume execution
      • RET 4. RET + increment ESP 4 bytes (used in stdcall and fastcall)
    • Stack frames (V1 C6)
      • Each function need a way to store return/local variables etc.
      • Function prologue create a frame for each function
        • After this some GP registers stored in stack
      • Function epilogue re-balance the stack (free memory) after execution
        • Before this GP registers are restored
    • prologue:
      push ebp      # save base pointer
      mov ebp, esp  # set the base-pointer, so that it points to the top of the stack
      sub esp, N    # allocate memory for current stack frame
      enter   N, 0   # More complex prologues can be obtained using different values (other than 0)
    • function invocation
      push eax    # save registers
      push ebx
      # Function body 
      pop ebx    # restore saved registers
      pop eax
    • epilogue:
      mov esp, ebp   # free memory allocated for current stack frame 
      pop ebp        # restore previous base pointer 
      ret            # exit function
    • Calling conventions:
      • cdecl
        • Arguments are passed on the stack in Right-to-Left order, and return values are passed in eax.
        • The calling function cleans the stack.
          • CDECL functions to have variable-length argument lists (aka variadic functions)
          • Number of arguments is not appended to the name of the function by the compiler, and the assembler and the linker are therefore unable to determine if an incorrect number of arguments is used.
        • CDECL functions are almost always prepended with an _.
          _cdecl int MyFunction1(int a, int b)
          return a + b;
          x = MyFunction1(2, 3);
          push ebp              # prologue
          mov ebp, esp          # prologue
          mov eax, [ebp + 8]    # read 1st param into eax
          mov edx, [ebp + 12]   # read 2nd param into abx
          add eax, edx          
          pop ebp               # epilogue
          ret                   # epilogue
          push 3                # push rightmost param into stack (b)
          push 2                # push next-rightmost param into stack (a)
          call _MyFunction1     # call
          add esp, 8            # calling function cleans (free memory)
    • stdcall (WINAPI)
      • Exclusively by Microsoft as the standard calling convention for the Win32 API
      • Arguments are passed on the stack in Right-to-Left order, and return values are passed in eax. (same as __cdecl)
      • The called function cleans the stack, unlike CDECL.
        • STDCALL doesn't allow variable-length argument lists.
      • Name-decorated with a leading _, followed by an @, and then the number (in bytes) of arguments passed on the stack. This number will always be a multiple of 4, on a 32-bit aligned machine.
      • Resulting code is smaller because clean up instructions are not repeated with each invocation.
        _stdcall int MyFunction2(int a, int b)
        return a + b;
        x = MyFunction2(2, 3);
        :_MyFunction2@8       # Function name contains info about number of bytes to clean
        push ebp              # prologue
        mov ebp, esp          # prologue
        mov eax, [ebp + 8]    # read 1st param into eax
        mov edx, [ebp + 12]   # read 2nd param into abx
        add eax, edx          
        pop ebp               # epilogue
        ret 8                 # epilogue + called function cleans (free memory). 8 is how many bytes to pop off.
        push 3                # push rightmost param into stack (b)
        push 2                # push next-rightmost param into stack (a)
        call _MyFunction2@8   # call
    • fastcall
      • not completely standard across all compilers
      • first 2 or 3 32-bit (or smaller) arguments are passed in registers, with the most commonly used registers being edx, eax, and ecx
      • Additional arguments, or arguments larger than 4-bytes are passed on the stack, often in Right-to-Left order (similar to CDECL)
      • Calling function cleans most frequently
      • prepends an @ to the function name, and follows the function name with @x, where x is the number (in bytes) of arguments
      • FASTCALL function doesn't need a stack frame
      • Commonly gcc and Windows FASTCALL convention pushes parameters one and two into ecx and edx, respectively, before pushing any remaining parameters onto the stack.
        _fastcall int MyFunction3(int a, int b)
        return a + b;
        x = MyFunction3(2, 3);
        push ebp           # prologue
        mov ebp, esp       # prologue
        add ecx, edx       # a is in ecx, b is in edx
        pop ebp            # epilogue
        ret                # epilogue
        mov ecx, 2         # move rightmost param to eax 
        mov edx, 3         # move next-rightmost param to edx
        call @MyFunction3@8 
  • Read EIP (used in relocatable code)
    • Move return address (EIP) to the EAX register
      GetEIP proc 
      mov eax, dword ptr ss:[esp]
      GetEIP endp
    • Put the address of previous instruction executed into EAX. Usable to locate other parts of code in memory (by adding / subtracting to EIP value):
      Call _getEIP
      _getEIP: pop eax
  • Heaps - Dynamically & runtime allocated memory (used to store data that doesn't fit in stack)
  • Handles - References to resources. Used by OS to control resource access.
    • Example: To access a file need to create a file-handle using Windows API.
  • Exceptions: Events occurred during runtime. and handlers to handle events.
    • Hardware exceptions: Bad sequence (div by zero). OS map to error codes.
    • Software exceptions:
    • Windows implements: Structured Exception Handling (SEH) to handle both software and hardware exceptions.

  • The CPU ring0, ring1, ring2, ring3:
  • Tracking Ring3 - Ring0 of running processes:;jsessionid=B89F397D53E2BA65418324AA33A1437D
  • The CPU privilege level is divided into four levels: RING0, RING1, RING2, RING3.
  • Windows uses only one of the two levels RING0 and RING3 (userland).
  • If the application attempts to execute RING0general instructions, then Windows will display Illegal Instruction error message.
  • Windows Ring3 Internal Structures
      • Address of the top and bottom of current thread's stack
      • Thread identifier
      • Process thread belongs to
      • Code of the last error
      • Address of the Thread Local Storage (TLS)
      • Address of the PROCESS_ENVIRONMENT_BLOCK (PEB)
      • Image base of the process
      • Address of Loader data structure (PEB_LDR_DATA)
      • NtGlobalFlag value (useable in detecting debugging)
      • Major and minor version of Windows OS
      • Number of processors
      • BeingDebugged flag (useable in detecting debugging)
    • CONTEXT contains:
      • All CPU state info for thread (during internal operations)
  • Windows APIs
    • Ring3 OS functions
    • Communicate with kernel function in a safe way (app to OS communication)
    • Categories:
      • Administration and Management
      • Diagnostics
      • Graphics and Multimedia
      • Networking
      • System Services
      • Windows User Interface
  • Tool categories
  • Immunity Practice
    • Level 1: App that branch based on password. Function that return if password is valid using two params.
      • Change ZF
      • New origin
      • Substitute JMP with NOPs (Binary -> Fill with NOPs)
      • Substitute JMP to desired location (Binary -> Edit and set offset to X)
      • Inverse JMP (double click on instruction and change JNE to JE, etc.)
      • Check params sent to check function and extract password.
      • Save changes with right-click, save to executable. right-click save-file.
    • Looking at memory regions
    • PE information: Right click "PE header" memory section -> dump in cpu -> right-click in dump -> special -> PE header
      • Check offset value and go to that address
      • Beginning of section: from memory view
      • Actual beginning of section within binary: PointerToRawData
    • Find offset within binary:
      • offset within code section = (address of the instruction - start address of code section)
      • offset within code section + PointerToRawData of the code section = exact location within binary


  • Hex Editors
  • Decompilers
  • Disassemblers
  • Debuggers
    • Ring0
    • Ring3
  • System Monitoring Tools
  • Windows API Monitoring Tools


Language/OS Specific



  • Disable ASLR: sudo sysctl -w kernel.randomize_va_space=0
  • Allow ptrace processes: sudo sysctl -w kernel.yama.ptrace_scope=0
  • Installing 32bit Libraries
    dpkg --add-architecture i386
    apt-get install libc6:i386
  • Run GBD with env variables: env - gdb /bin/lcars
  • Display Information
    info registers
    info all-registers
  • Display memory map: vmmap
  • Display Registers / Memory: display /x $eax x/50c $eax x/s $eax
  • Disassemble-flavor: set disassembly-flavor intel
  • Disassemble: disassemble $eip
  • Print Type Information: ptype Student
  • Check security information: checksec



Quick Reference

  • Check DEP policy: bcdedit /enum | findstr nx


  • Process Explorer: to show the processes’ DEP and ASLR status, I added the following columns to the view: View -> Select Columns -> DEP Status and View -> Select Columns -> ASLR Enabled
    • Set the lower pane to view DLLs for a process and added the “ASLR Enabled” column
  • BinScope Binary analyzer, analyzes binaries for a wide variety of security protections:
  • LookingGlass is a handy tool to scan a directory structure or the running processes to report which binaries do not make use of ASLR and NX:







  • Kernel and the user space of a process share the same zero page
  • “Attacking the Core: Kernel Exploiting Notes” by twiz & sgrakkyu, which can be found at
  • Each user-mode address space is unique to a particular process, while the kernel address space is shared across all processes. Mapping the NULL page in one process only causes it to be mapped in that pro- cess’s address space only.



Stack Based BOF

jmp reg

Null pointer dereference



  • Never trust user input (this includes file data, network data, etc.).
  • Never use unvalidated length or size values.
  • Always define proper error conditions.
  • Always validate return values correctly.





  • Toold, JADX, JD-GUI, Procyon, CFR, Fernflower etc., Bytecodeviewer, dex2jar, APK tool

Mobile Apps