Iczelion's Win32 Assembly Tutorial 30: Win32 Debug API part 3

Tutorial 30: Win32 Debug API part 3

In this tutorial, we continue the exploration of win32 debug api. Specifically, we will learn how to trace the debuggee.

Download the example.

Theory:

If you have used a debugger before, you would be familiar with tracing. When you "trace" a program, the program stops after executing each instruction, giving you the chance to examine the values of registers/memory. Single-stepping is the official name of tracing.
The single-step feature is provided by the CPU itself. The 8th bit of the flag register is called trap flag. If this flag(bit) is set, the CPU executes in single-step mode. The CPU will generate a debug exception after each instruction. After the debug exception is generated, the trap flag is cleared automatically.
We can also single-step the debuggee, using win32 debug api. The steps are as follows:

Call getthreadcontext, specifying context_control in ContextFlags, to obtain the value of the flag register.
Set the trap bit in regflag member of the context structure
call setthreadcontext
Wait for the debug events as usual. The debuggee will execute in single-step mode. After it executes each instruction, we will get exception_debug_event with exception_single_step value in u.exception.pexceptionrecord.exceptioncode
If you need to trace the next instruction, you need to set the trap bit again.

Example:

.386
.model FLAT,STDCALL 
OPTION casemap:none 

include    \masm32\include\windows.inc 
include    \masm32\include\kernel32.inc 
include    \masm32\include\comdlg32.inc 
include    \masm32\include\user32.inc 
includelib \masm32\lib\kernel32.lib 
includelib \masm32\lib\comdlg32.lib 
includelib \masm32\lib\user32.lib 

.data 
AppName db "Win32 Debug Example no.4",0 
ofn OPENFILENAME <;> 
FilterString db "Executable Files",0,"*.exe",0 
             db "All Files",0,"*.*",0,0 
ExitProc     db "The debuggee exits",0Dh,0Ah 
             db "Total Instructions executed: %lu",0 
TotalInstruction dd 0

.data? 
buffer    db 512 dup(?) 
startinfo STARTUPINFO <;> 
pi        PROCESS_INFORMATION <;> 
DBEvent   DEBUG_EVENT <;> 
context   CONTEXT <;> 

.code 
start: 
        mov    ofn.lStructSize,SIZEOF ofn 
        mov    ofn.lpstrFilter, OFFSET FilterString 
        mov    ofn.lpstrFile, OFFSET buffer 
        mov    ofn.nMaxFile,512 
        mov    ofn.Flags, OFN_FILEMUSTEXIST or OFN_PATHMUSTEXIST \
                       or OFN_LONGNAMES or OFN_EXPLORER          \
                       or OFN_HIDEREADONLY 
        invoke GetOpenFileName, ADDR ofn 
.IF eax==TRUE 
        invoke GetStartupInfo,ADDR startinfo 
        invoke CreateProcess, ADDR buffer, NULL, NULL, NULL, FALSE,
                              DEBUG_PROCESS+ DEBUG_ONLY_THIS_PROCESS,
                              NULL, NULL, ADDR startinfo, ADDR pi
.WHILE TRUE 
        invoke WaitForDebugEvent, ADDR DBEvent,INFINITE 
.IF DBEvent.dwDebugEventCode==EXIT_PROCESS_DEBUG_EVENT
        invoke wsprintf, ADDR buffer, ADDR ExitProc, TotalInstruction 
        invoke MessageBox, 0, ADDR buffer, ADDR AppName, \
                           MB_OK+MB_ICONINFORMATION 
.BREAK 
.ELSEIF DBEvent.dwDebugEventCode==EXCEPTION_DEBUG_EVENT
.IF DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_BREAKPOINT
        mov    context.ContextFlags, CONTEXT_CONTROL 
        invoke GetThreadContext, pi.hThread, ADDR context 
        or     context.regFlag,100h 
        invoke SetThreadContext,pi.hThread, ADDR context 
        invoke ContinueDebugEvent, DBEvent.dwProcessId, \
                                   DBEvent.dwThreadId,  \
                                   DBG_CONTINUE 
.CONTINUE
.ELSEIF DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_SINGLE_STEP
        inc    TotalInstruction 
        invoke GetThreadContext,pi.hThread,ADDR context
        or     context.regFlag,100h 
        invoke SetThreadContext,pi.hThread,ADDR context 
        invoke ContinueDebugEvent, DBEvent.dwProcessId, \
                                   DBEvent.dwThreadId,  \
                                   DBG_CONTINUE 
.CONTINUE
.ENDIF 
.ENDIF 
        invoke ContinueDebugEvent, DBEvent.dwProcessId, \
                                   DBEvent.dwThreadId,  \
                                   DBG_EXCEPTION_NOT_HANDLED 
.ENDW 
.ENDIF 
        invoke CloseHandle,pi.hProcess 
        invoke CloseHandle,pi.hThread 
        invoke ExitProcess, 0
         
END start

Analysis:

The program shows the openfile dialog box. When the user chooses an executable file, it executes the program in single-step mode, couting the number of instructions executed until the debuggee exits.

.ELSEIF DBEvent.dwDebugEventCode==EXCEPTION_DEBUG_EVENT
.IF DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_BREAKPOINT

We take this opportunity to set the debuggee into single-step mode. Remember that Windows sends an EXCEPTION_BREAKPOINT just before it executes the first instruction of the debuggee.

        mov    context.ContextFlags, CONTEXT_CONTROL
        invoke GetThreadContext, pi.hThread, ADDR context

We call getthreadcontext to fill the context structure with the current values in the registers of the debuggee. More specifically, we need the current value of the flag register.

or context.regFlag,100h

We set the trap bit (8th bit) in the flag register image.

        invoke SetThreadContext,pi.hThread, ADDR context
        invoke ContinueDebugEvent, DBEvent.dwProcessId, \
                                   DBEvent.dwThreadId, DBG_CONTINUE
.CONTINUE

Then we call setthreadcontext to overwrite the values in the context structure with the new one(s) and call continuedebugevent with dbg_continue flag to resume the debuggee.

.ELSEIF DBEvent.u.Exception.pExceptionRecord.ExceptionCode==EXCEPTION_SINGLE_STEP
        inc    TotalInstruction

When an instruction is executed in the debuggee, we receive an exception_debug_event. We must examine the value of u.exception.pexceptionrecord.exceptioncode. If the value is exception_single_step, then this debug event is generated because of the single-step mode. In this case, we can increment the variable totalinstruction by one because we know that exactly one instruction was executed in the debuggee.

        invoke GetThreadContext,pi.hThread,ADDR context
        or     context.regFlag,100h
        invoke SetThreadContext,pi.hThread, ADDR context
        invoke ContinueDebugEvent, DBEvent.dwProcessId,             \
                                   DBEvent.dwThreadId,DBG_CONTINUE 
.CONTINUE

Since the trap flag is cleared after the debug exception is generated, we must set the trap flag again if we want to continue in single-step mode.
Warning: Don't use the example in this tutorial with a large program: tracing is SLOW. You may have to wait for ten minutes before you can close the debuggee.

Tutorial 29: Win32 Debug API part 2

Overview

Tutorial 31: ListView Control