Tutorial 24: Windows Hooks

We will learn about Windows hooks in this tutorial. Windows hooks are very powerful. With them, you can poke inside other processes and sometimes alter their behaviors.

Theory:

Windows hooks can be considered one of the most powerful features of Windows. With them, you can trap events that will occur, either in your own process or in other processes. By "hooking", you tell Windows about a function, filter function also called hook procedure, that will be called everytime an event you're interested in occurs. There are two types of them: local and remote hooks. When you install hooks, remember that they affect system performance. System-wide hooks are the most notorious. Since ALL related events will be routed through your filter function, your system may slow down noticeably. So if you use a system-wide hook, you should use it judiciously and unhook it as soon as you don't need it. Also, you have a higher chance of crashing the other processes since you can meddle with other processes and if something is wrong in your filter function, it can pull the other processes down to oblivion with it. Remember: Power comes with responsibility.
You have to understand how a hook works before you can use it efficiently. When you create a hook, Windows creates a data structure in memory, containing information about the hook, and adds it to a linked list of existing hooks. New hook is added in front of old hooks. When an event occurs, if you install a local hook, the filter function in your process is called so it's rather straightforward. But if it's a remote hook, the system must inject the code for the hook procedure into the address space(s) of the other process(es). And the system can do that only if the function resides in a DLL. Thus , if you want to use a remote hook, your hook procedure must reside in a DLL. There is two exceptions to this rule: journal record and journal playback hooks. The hook procedures for those two hooks must reside in the thread that installs the hooks. The reason why it must be so is that: both hooks deal with the low-level interception of hardware input events. The input events must be recorded/playbacked in the order they appeared. If the code of those two hooks is in a DLL, the input events may scatter among several threads and it is impossible to know the order of them. So the solution: the hook procedure of those two hooks must be in a single thread only i.e. the thread that installs the hooks.
There are 14 types of hooks: Now that we know some theory, we can move on to how to install/uninstall the hooks.
To install a hook, you call SetWindowsHookEx which has the following syntax:
SetWindowsHookEx proto HookType:DWORD, pHookProc:DWORD, hInstance:DWORD, ThreadID:DWORD If the call is successful, it returns the hook handle in eax. If not, NULL is returned. You must save the hook handle for unhooking later.
You can uninstall a hook by calling UnhookWindowsHookEx which accepts only one parameter, the handle of the hook you want to uninstall. If the call succeeds, it returns a non-zero value in eax. Otherwise, it returns NULL.
Now that you know how to install/uninstall hooks, we can examine the hook procedure.
The hook procedure will be called whenever an event that is associated with the type of hook you have installed occurs. For example, if you install WH_MOUSE hook, when a mouse event occurs, your hook procedure will be called. Regardless of the type of hook you installed, the hook procedure always has the following prototype: HookProc is actually a placeholder for the function name. You can name it anything you like so long as it has the above prototype. The interpretation of nCode, wParam and lParam is dependent on the type of hook you install. So as the return value from the hook procedure. For example:
WH_CALLWNDPROC WH_MOUSE
The bottom line is: you must consult your win32 api reference for details about the meanings of the parameters and return value of the hook you want to install.
Now there is a little catch about the hook procedure. Remember that the hooks are chained in a linked list with the most recently installed hook at the head of the list. When an event occurs, Windows will call only the first hook in the chain. It's your hook procedure's responsibility to call the next hook in the chain. You can choose not to call the next hook but you'd better know what you're doing. Most of the time, it's a good practice to call the next procedure so other hooks can have a shot at the event. You can call the next hook by calling CallNextHookEx which has the following prototype:
CallNextHookEx proto hHook:DWORD, nCode:DWORD, wParam:DWORD, lParam:DWORD
An important note about remote hooks: the hook procedure must reside in a DLL which will be mapped into other processes. When Windows maps the DLL into other processes, it will not map the data section(s) into the other processes. In short, all processes share a single copy of code but they will have their own private copy of the DLL's data section! This can be a big surprise to the unwary. You may think that when you store a value into a variable in the data section of a DLL, that value will be shared among all processes that load the DLL into their process address space. It's simply not true. In normal situation, this behavior is desirable since it provides the illusion that each process has its own copy of the DLL. But not when Windows hook is concerned. We want the DLL to be identical in all processes, including the data. The solution: you must mark the data section as shared. You can do this by specifying the section(s) attribute in the linker switch. For MASM, you need to use this switch:
/SECTION:<section name>, S
The name of the initialized data section is .data and the uninitialized data is .bss. For example if you want to assemble a DLL which contains a hook procedure and you want the uninitialized data section to be shared amoung processes, you must use the following line:
link /section:.bss,S  /DLL  /SUBSYSTEM:WINDOWS ..........
S attribute marks the section as shared.

Example:

There are two modules: one is the main program which will do the GUI part and the other is the DLL that will install/uninstall the hook.

;--------------------------------------------- This is the source code of the main program --------------------------------------

Main:
    call 'Kernel32.GetModuleHandleA' 0  | mov D$hInstance eax
    call 'USER32.DialogBoxParamA'  0 IDD_MAINDLG 0 DialogFunc 0
    call 'Kernel32.ExitProcess' 0
 
____________________________________________________________________________________________
____________________________________________________________________________________________

Proc DialogFunc:
    Arguments @Adressee, @Message, @wParam, @lParam

    pushad

    ...If D@message = &WM_CLOSE
        On D$HookFlag = &TRUE, call 'mousehook.UninstallHook'
        call 'user32.EndDialog' D@Adressee 0
 
    ...Else_If D@message = &WM_INITDIALOG
        ; Nop.

    ...Else_If D@Message = WM_MOUSEHOOK
        call 'user32.GetDlgItemTextA' D@Adressee IDC_HANDLE buffer1 128
        call 'user32.wsprintfA' buffer template D@wParam | add esp 12
        call 'kernel32.lstrcmpi' buffer buffer1
        On eax <> 0, call 'user32.SetDlgItemTextA' D@Adressee IDC_HANDLE buffer
 
        call 'user32.GetDlgItemTextA' D@Adressee IDC_CLASSNAME buffer1 128
        call 'user32.GetClassNameA' D@wParam buffer 128
        call 'kernel32.lstrcmpi' buffer buffer1
        On eax <> 0, call 'user32.SetDlgItemTextA' D@Adressee IDC_CLASSNAME buffer

        call 'user32.GetDlgItemTextA' D@Adressee IDC_WNDPROC buffer1 128
        call 'user32.GetClassLongA' D@wParam &GCL_WNDPROC
        call 'user32.wsprintfA' buffer template eax | add esp 12
        call 'kernel32.lstrcmpi' buffer buffer1
        On eax <> 0, call 'user32.SetDlgItemTextA' D@Adressee IDC_WNDPROC buffer
 
    ...Else_If D@message = &WM_COMMAND
        ..If D@lParam = 0
            ; No Handle >>> nop.
 
        ..Else_If W@wParam = IDC_EXIT
            call 'user32.SendMessageA' D@Adressee &WM_CLOSE 0 0 ;| jmp L2>>
 
        ..Else_If W@wParam+2 = &BN_CLICKED
 
            .If D$HookFlag = 0
                call 'mousehook.InstallHook' D@Adressee
                If eax <> 0
                    mov D$HookFlag &TRUE
                    call 'user32.SetDlgItemTextA' D@Adressee IDC_HOOK UnhookText
                End_If
 
            .Else
                call 'mousehook.UninstallHook'
                call 'user32.SetDlgItemTextA' D@Adressee IDC_HOOK HookText
                mov D$HookFlag &FALSE
                call 'user32.SetDlgItemTextA' D@Adressee IDC_CLASSNAME 0
                call 'user32.SetDlgItemTextA' D@Adressee IDC_HANDLE 0
                call 'user32.SetDlgItemTextA' D@Adressee IDC_WNDPROC 0
 
            .End_If
 
        ..End_If
 
    ...Else
L8:     popad | mov eax &FALSE | Exit
 
    ...End_If

L9: popad | mov eax &TRUE
EndP

;----------------------------------------------------- This is the source code of the DLL --------------------------------------
Proc Main:
    Arguments @Instance, @Reason, @Reserved
 
        If D@Reason = &DLL_PROCESS_ATTACH
           move D$hInstance D@Instance
        Else_If D@Reason = &DLL_PROCESS_DETACH
            ; ...

        Else_If D@Reason = &DLL_THREAD_ATTACH
            ; ...

        Else_If D@Reason = &DLL_THREAD_DETACH
            ; ...

        End_If

        mov eax &TRUE
Endp
____________________________________________________________________________________________

proc MouseProc::
    Arguments @nCode, @wParam, @lParam
 
    call 'user32.CallNextHookEx' D$hHook D@nCode D@wParam D@lParam
    mov edx D@lParam
 
    ; edx Points to a Point structure
    ; first dword = x second = y
 
    call 'user32.WindowFromPoint' D$edx D$edx+4
    call 'user32.PostMessageA' D$hWnd WM_MOUSEHOOK eax 0
    xor eax eax
EndP
 

Proc InstallHook::
    Arguments @hwnd2
 
        push D@hwnd2 | pop D$hWnd
 
        call 'user32.SetWindowsHookExA' &WH_MOUSE MouseProc D$hInstance &NULL
        mov D$hHook eax
EndP
 

Proc UninstallHook::
        call 'user32.UnhookWindowsHookEx' D$hHook
EndP
 

Analysis:

The example will display a dialog box with three edit controls that will be filled with the class name, window handle and the address of the window procedure associated with the window under the mouse cursor. There are two buttons, Hook and Exit. When you press the Hook button, the program hooks the mouse input and the text on the button changes to Unhook. When you move the mouse cursor over a window, the info about that window will be displayed in the main window of the example. When you press Unhook button, the program removes the mouse hook.
The main program uses a dialog box as its main window. It defines a custom message, WM_MOUSEHOOK which will be used between the main program and the hook DLL. When the main program receives this message, wParam contains the handle of the window that the mouse cursor is on. Of course, this is an arbitrary arrangement. I decide to send the handle in wParam for the sake of simplicity. You can choose your own method of communication between the main program and the hook DLL.

 
            .If D$HookFlag = 0
                call 'mousehook.InstallHook' D@Adressee
                If eax <> 0
                    mov D$HookFlag &TRUE
                    call 'user32.SetDlgItemTextA' D@Adressee IDC_HOOK UnhookText
                End_If

The program maintains a flag, HookFlag, to monitor the state of the hook. It's FALSE if the hook is not installed and TRUE if the hook is installed.
When the user presses Hook button, the program checks if the hook is already installed. If it is not, it call InstallHook function in the hook DLL to install it. Note that we pass the handle of the main dialog as the parameter of the function so the hook DLL can send the WM_MOUSEHOOK messages to the right window i.e. our own.
When the program is loaded, the hook DLL is loaded too. Actually, DLLs are loaded immediately after the program is in memory. The DLL entrypoint function is called before the first instruction in the main program is execute even. So when the main program executes the DLL(s) is/are initialized. We put the following code in the DLL entrypoint function of the hook DLL:
 

    If D@Reason = &DLL_PROCESS_ATTACH
           move D$hInstance D@Instance

The code just saves the instance handle of the hook DLL itself to a global variable named hInstance for use within the InstallHook function. Since the DLL entrypoint function is called before other functions in the DLL are called , hInstance is always valid. We put hInstance in .data section so that this value is kept on per-process basis. Since when the mouse cursor hovers over a window, the hook DLL is mapped into the process. Imagine that there is already a DLL that occupies the intended load address of the hook DLL, the hook DLL would be remapped to another address. The value of hInstance will be updated to those of the new load address. When the user presses Unhook button and then Hook button, SetWindowsHookEx will be called again. However, this time, it will use the new load address as the instance handle which will be wrong because in the example process, the hook DLL's load address hasn't been changed. The hook will be a local one where you can hook only the mouse events that occur in your own window. Hardly desirable.
 

Proc InstallHook::
    Arguments @hwnd2
 
        push D@hwnd2 | pop D$hWnd
 
        call 'user32.SetWindowsHookExA' &WH_MOUSE MouseProc D$hInstance &NULL
        mov D$hHook eax
EndP

The InstallHook function itself is very simple. It saves the window handle passed as its parameter to a global variable named hWnd for future use. It then calls SetWindowsHookEx to install a mouse hook. The return value of SetWindowsHookEx is stored in a global variable named hHook for use with UnhookWindowsHookEx.
After SetWindowsHookEx is called, the mouse hook is functional. Whenever a mouse event occurs in the system, MouseProc ( your hook procedure) is called.

proc MouseProc::
    Arguments @nCode, @wParam, @lParam
 
    call 'user32.CallNextHookEx' D$hHook D@nCode D@wParam D@lParam
    mov edx D@lParam
 
    ; edx Points to a Point structure
    ; first dword = x second = y
 
    call 'user32.WindowFromPoint' D$edx D$edx+4
    call 'user32.PostMessageA' D$hWnd WM_MOUSEHOOK eax 0
    xor eax eax
EndP

The first thing it does is to call CallNextHookEx to give the other hooks the chance to process the mouse event. After that, it calls WindowFromPoint function to retrieve the handle of the window at the specified screen coordinate. Note that we use the POINT structure in the MOUSEHOOKSTRUCT structure pointed to by lParam as the current mouse coordinate. After that we send the window handle to the main program via PostMessage with WM_MOUSEHOOK message. One thing you should remember is that: you should not use SendMessage inside the hook procedure, it can cause message deadlock. PostMessage is recommended. The MOUSEHOOKSTRUCT structure is defined below:

[MOUSEHOOKSTRUCT: D§
  ptX: ?   ptY: ?
  hwnd: ?
  wHitTestCode:  ?
  dwExtraInfo:  ?]
 

When the main window receives WM_MOUSEHOOK message, it uses the window handle in wParam to retrieve the information about the window.

     ...Else_If D@Message = WM_MOUSEHOOK
        call 'user32.GetDlgItemTextA' D@Adressee IDC_HANDLE buffer1 128
        call 'user32.wsprintfA' buffer template D@wParam | add esp 12
        call 'kernel32.lstrcmpi' buffer buffer1
        On eax <> 0, call 'user32.SetDlgItemTextA' D@Adressee IDC_HANDLE buffer
 
        call 'user32.GetDlgItemTextA' D@Adressee IDC_CLASSNAME buffer1 128
        call 'user32.GetClassNameA' D@wParam buffer 128
        call 'kernel32.lstrcmpi' buffer buffer1
        On eax <> 0, call 'user32.SetDlgItemTextA' D@Adressee IDC_CLASSNAME buffer

        call 'user32.GetDlgItemTextA' D@Adressee IDC_WNDPROC buffer1 128
        call 'user32.GetClassLongA' D@wParam &GCL_WNDPROC
        call 'user32.wsprintfA' buffer template eax | add esp 12
        call 'kernel32.lstrcmpi' buffer buffer1
        On eax <> 0, call 'user32.SetDlgItemTextA' D@Adressee IDC_WNDPROC buffer

To avoid flickers, we check the text already in the edit controls and the text we will put into them if they are identical. If they are, we skip them.
We retrieve the class name by calling GetClassName, the address of the window procedure by calling GetClassLong with GCL_WNDPROC and then format them into strings and put them into the appropriate edit controls.

                call 'mousehook.UninstallHook'
                call 'user32.SetDlgItemTextA' D@Adressee IDC_HOOK HookText
                mov D$HookFlag &FALSE
                call 'user32.SetDlgItemTextA' D@Adressee IDC_CLASSNAME 0
                call 'user32.SetDlgItemTextA' D@Adressee IDC_HANDLE 0
                call 'user32.SetDlgItemTextA' D@Adressee IDC_WNDPROC 0

When the user presses Unhook button, the program calls UninstallHook function in the hook DLL. UninstallHook just calls UnhookWindowsHookEx. After that, it changes the text of the button back to "Hook", HookFlag to FALSE and clears the content of the edit controls.
Note the linker switch in the makefile.

The DLL .Data must be specified as a shared section to make all processes share the same section of the hook DLL. Without this switch, your hook DLL will not function correctly.


[Iczelion's Win32 Assembly Homepage]