Iczelion's Win32 Assembly Tutorial 20: Window Subclassing

Tutorial 20: Window Subclassing

In this tutorial, we will learn about window subclassing, what it is and how to use it to your advantage.

Download the example here.

Theory:

If you program in Windows for some time, you will find some cases where a window has nearly the attributes you need in your program but not quite. Have you encountered a situation where you want some special kind of edit control that can filter out some unwanted text ? The straightforward thing to do is to code your own window. But it's really hard work and time-consuming. Window subclassing to the rescue.

In a nutshell, window subclassing allows you to "take over" the subclassed window. You will have absolute control over it. Let's take an example to make this clearer. Suppose you need a text box that accepts only hex numbers. If you use a simple edit control, you have no say whatsoever when your user types something other than hex numbers into your text box, ie. if the user types "zb+q*" into your text box, you can't do anything with it except rejecting the whole text string. This is unprofessional at least. In essence, you need the ability to examine each character the user typed into the text box right at the moment he typed it.

We will examine how to do that now. When the user types something into a text box, Windows sends WM_CHAR message to the edit control's window procedure. This window procedure resides inside Windows itself so we can't modify it. But we can redirect the message flow to our own window procedure. So that our window procedure will get first shot at any message Windows sends to the edit control. If our window procedure chooses to act on the message, it can do so. But if it doesn't want to handle the message, it can pass it to the original window procedure. This way, our window procedure inserts itself between Windows and the edit control. Look at the flow below:

Before Subclassing

Windows ==> edit control's window procedure

After Subclassing

Windows ==> our window procedure -----> edit control's window procedure

Now we put our attention on how to subclass a window. Note that subclassing is not limited to controls, it can be used with any window.

Let's think about how Windows knows where the edit control's window procedure resides. A guess ? ...... lpfnWndProc member of WNDCLASSEX structure. If we can replace this member with the address of our own window procedure, Windows will send messages to our window PROC instead.

We can do that by calling SetWindowLong.

SetWindowLong PROTO hWnd:DWORD, nIndex:DWORD, dwNewLong:DWORD

hwnd handle of the window to change the value in the WNDCLASSEX structure.
nindex value to change.

gwl_exstyle Sets a new extended window style.

gwl_style Sets a new window style.

gwl_wndproc Sets a new address for the window procedure.

gwl_hinstance Sets a new application instance handle.

gwl_id Sets a new identifier of the window.

gwl_userdata Sets the 32-bit value associated with the window. Each window has a corresponding 32-bit value intended for use by the application that created the window.

dwnewlong the replacement value.

So our job is easy: We code a window PROC that will handle the messages for the edit control and then call SetWindowLong with GWL_WNDPROC flag, passing along the address of our window PROC as the third parameter. If the function succeeds, the return value is the previous value of the specified 32-bit integer, in our case, the address of the original window procedure. We need to store this value for use within our window procedure.
Remember that there will be some messages we don't want to handle, we will pass them to the original window procedure. We can do that by calling CallWindowProc function.

CallWindowProc PROTO lpPrevWndFunc:DWORD, \
      hWnd:DWORD,          \
      Msg:DWORD,           \
      wParam:DWORD,        \
      lParam:DWORD

lpprevwndfunc the address of the original window procedure.

The remaining four parameters are the ones passed to our window procedure. We just pass them along to CallWindowProc.

Code Sample:

.386
.model FLAT,STDCALL
option casemap:none

include    \masm32\include\windows.inc
include    \masm32\include\user32.inc
include    \masm32\include\kernel32.inc
include    \masm32\include\comctl32.inc
includelib \masm32\lib\comctl32.lib
includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib

WinMain     PROTO :DWORD,:DWORD,:DWORD,:DWORD
EditWndProc PROTO :DWORD,:DWORD,:DWORD,:DWORD

.data
ClassName  db "SubclassWinClass",0
AppName    db "Subclassing Demo",0
EditClass  db "EDIT",0
Message    db "You pressed Enter in the text box!",0

.data?
hInstance  HINSTANCE ?
hwndEdit   dd ?
OldWndProc dd ?

.code
start:
        invoke GetModuleHandle, NULL
        mov    hInstance,eax
        invoke WinMain, hInstance,NULL,NULL, SW_SHOWDEFAULT
        invoke ExitProcess,eax
        
WinMain PROC hInst:HINSTANCE,hPrevInst:HINSTANCE,CmdLine:LPSTR,
             CmdShow:DWORD
LOCAL wc:WNDCLASSEX
LOCAL msg:MSG
LOCAL hwnd:HWND
        mov    wc.cbSize,SIZEOF WNDCLASSEX
        mov    wc.style, CS_HREDRAW or CS_VREDRAW
        mov    wc.lpfnWndProc, OFFSET WndProc
        mov    wc.cbClsExtra,NULL
        mov    wc.cbWndExtra,NULL
        push   hInst
        pop    wc.hInstance
        mov    wc.hbrBackground,COLOR_APPWORKSPACE
        mov    wc.lpszMenuName,NULL
        mov    wc.lpszClassName,OFFSET ClassName
        invoke LoadIcon,NULL,IDI_APPLICATION
        mov    wc.hIcon,eax
        mov    wc.hIconSm,eax
        invoke LoadCursor,NULL,IDC_ARROW
        mov    wc.hCursor,eax
        invoke RegisterClassEx, ADDR wc
        invoke CreateWindowEx,WS_EX_CLIENTEDGE,ADDR ClassName,
                                               ADDR AppName,           \
                              WS_OVERLAPPED+WS_CAPTION+WS_SYSMENU      \
                             +WS_MINIMIZEBOX+WS_MAXIMIZEBOX+WS_VISIBLE,\
                              CW_USEDEFAULT,CW_USEDEFAULT,             \
                              350,200,                                 \
                              NULL,NULL,hInst,NULL
        mov    hwnd,eax
.WHILE TRUE
        invoke GetMessage, ADDR msg,NULL,0,0
.BREAK .IF (!eax)
        invoke TranslateMessage, ADDR msg
        invoke DispatchMessage, ADDR msg
.ENDW
        mov    eax,msg.wParam
        ret
WinMain ENDP

WndProc PROC hWnd:HWND, uMsg:UINT, wParam:WPARAM, lParam:LPARAM
.IF uMsg==WM_CREATE
        invoke CreateWindowEx,WS_EX_CLIENTEDGE,ADDR EditClass,NULL,\
                              WS_CHILD+WS_VISIBLE+WS_BORDER,       \
                              20,20,                               \
                              300,25,                              \
                              hWnd,NULL,                           \
                              hInstance,NULL
        mov    hwndEdit,eax
        invoke SetFocus,eax
;-----------------------------------------
; Subclass it!
;-----------------------------------------
        invoke SetWindowLong,hwndEdit,GWL_WNDPROC,ADDR EditWndProc
        mov    OldWndProc,eax
.ELSEIF uMsg==WM_DESTROY
        invoke PostQuitMessage,NULL
.ELSE
        invoke DefWindowProc,hWnd,uMsg,wParam,lParam
        ret
.ENDIF
        xor    eax,eax
        ret
WndProc ENDP

EditWndProc PROC hEdit:DWORD,uMsg:DWORD,wParam:DWORD,lParam:DWORD
.IF uMsg==WM_CHAR
        mov    eax,wParam
.IF (al>="0" &;& al<="9") || (al>="A" && al<="F")
 || (al>="a" &;& al<="f") || al==VK_BACK
.IF al>="a" &;& al<="f"
        sub    al,20h
.ENDIF
        invoke CallWindowProc,OldWndProc,hEdit,uMsg,eax,lParam
        ret
.ENDIF
.ELSEIF uMsg==WM_KEYDOWN
        mov    eax,wParam
.IF al==VK_RETURN
        invoke MessageBox,hEdit,ADDR Message,ADDR AppName,
                          MB_OK+MB_ICONINFORMATION
        invoke SetFocus,hEdit
.ELSE
        invoke CallWindowProc,OldWndProc,hEdit,uMsg,wParam,lParam
        ret
.ENDIF
.ELSE
        invoke CallWindowProc,OldWndProc,hEdit,uMsg,wParam,lParam
        ret
.ENDIF
        xor    eax,eax
        ret
EditWndProc ENDP 

END start

Analysis:

        invoke SetWindowLong,hwndEdit,GWL_WNDPROC,ADDR EditWndProc
        mov    OldWndProc,eax

After the edit control is created, we subclass it by calling SetWindowLong, replacing the address of the original window procedure with our own window procedure. Note that we store the address of the original window procedure for use with CallWindowProc. Note the EditWndProc is an ordinary window procedure.

.IF uMsg==WM_CHAR
        mov    eax,wParam
.IF (al>="0" && al<="9") || (al>="A" && al<="F")
 || (al>="a" && al<="f") || al==VK_BACK
.IF al>="a" && al<="f"
        sub    al,20h
.ENDIF
        invoke CallWindowProc,OldWndProc,hEdit,uMsg,eax,lParam
        ret
.ENDIF

Within EditWndProc, we filter WM_CHAR messages. If the character is between 0-9 or a-f, we accept it by passing along the message to the original window procedure. If it is a lower case character, we convert it to upper case by adding it with 20h. Note that, if the character is not the one we expect, we discard it. We don't pass it to the original window PROC. So when the user types something other than 0-9 or a-f, the character just doesn't appear in the edit control.

.ELSEIF uMsg==WM_KEYDOWN
        mov    eax,wParam
.IF al==VK_RETURN
        invoke MessageBox,hEdit,ADDR Message,ADDR AppName, \
                          MB_OK+MB_ICONINFORMATION
        invoke SetFocus,hEdit
.ELSE
        invoke CallWindowProc,OldWndProc,hEdit,uMsg,wParam,lParam
        ret
.END

I want to demonstrate the power of subclassing further by trapping Enter key. EditWndProc checks WM_KEYDOWN message if it's VK_RETURN (the Enter key). If it is, it displays a message box saying "You pressed the Enter key in the text box!". If it's not an Enter key, it passes the message to the original window procedure.
You can use window subclassing to take control over other windows. It's one of the powerful techniques you should have in your arsenal.

Tutorial 19: Tree View Control

Overview

Tutorial 21: Pipe