The Complete Idiot's Guide to Writing Shell Extensions in Asm - Part II
By Maurice Montgenie (Original Version in ATL by Michael Dunn) 

A tutorial on writing a shell extension that operates on multiple files at once. 
 

The original version of this tute can be found in the ATL part of the site
  • Download source files - 40 Kb

    In Part I of the Guide, I gave an introduction to writing shell extensions, and demonstrated a simple context menu extension that operated on a single file at a time. In Part II, I'll demonstrate how to operate on multiple files in a single operation. The extension is a utility that can register and unregister COM servers. It also demonstrates how to use the ATL dialog class CDialogImpl. I will wrap up Part II by explaining some special registry keys that you can use to have your extension invoked on any file, not just preselected types.

    Part II assumes that you've read Part I so you know the basics of context menu extensions. You should also understand the basics of COM.

    Beginning the context menu extension what should it do?

    This shell extension will let you register and unregister COM servers in DLLs, and OCXs. Unlike the extension we did in Part I, this extension will operate on all the files that are selected when the right-click event happens. In the original tutorial Michael made a mistake, he said that this extension coul register Exes too. In fact Local servers don't have DllRegister() and DllUnregister().

  • Using AppWizard to get started

    Run the AppWizard and make a new ATL COM app. We'll call it DllReg. Keep all the default settings in the AppWizard, and click Finish. To add a COM object to the DLL, go to the ClassView tree, right-click the DllReg classes item, and pick New ATL Object.

    In the ATL Object Wizard, the first panel already has Simple Object selected, so just click Next. On the second panel, enter DllRegShlExt in the Short Name edit box and click OK. (The other edit boxes on the panel will be filled in automatically.) This creates a class called CDllRegShlExt that contains the basic code for implementing a COM object. We will add our code to this class.

    The initialization interface

    Our IShellExtInit::Initialize() implementation will be quite different in this shell extension, for two reasons. First, we will enumerate all of the selected files. Second, we will test the selected files to see if they export registration and unregistration functions. We will consider only those files that export both DllRegisterServer() and DllUnregisterServer(). All other files will be ignored.

    We will need a few member variables:

        m_hRegBmp   DWORD ?
        m_hUnRegBmp DWORD ?
        m_lsFiles   strList < ? >
        m_szDir     BYTE MAX_PATH DUP(?)   
    

    The CreateDllReg constructor loads two bitmaps for use in the context menu:

    CreateDllReg proc this_:DWORD   
    
      invoke LoadBitmap, g_hModule, IDB_REGISTERBMP  
    
      ;save m_hRegBmp
      pObjectData this_, ecx  ; cast this_ to object data
      lea ecx, (asmDllRegExtObjData ptr [ecx]).m_hRegBmp
      mov [ecx], eax
    
      invoke LoadBitmap, g_hModule, IDB_UNREGISTERBMP
    
      ;save m_hUnRegBmp
      pObjectData this_, ecx  ; cast this_ to object data
      lea ecx, (asmDllRegExtObjData ptr [ecx]).m_hUnRegBmp
      mov [ecx], eax
               
      ret
    CreateDllReg endp
    

    After you add IShellExtInit to the list of interfaces implemented by DllRegShlExt(see Part I for the instructions on doing this), we'll be ready to write the Initialize() function.

    Initialize() will perform these steps:

    1. Change the current directory to the directory being viewed in the Explorer window.
    2. Enumerate all of the files that were selected.
    3. For each file, try to load it with LoadLibrary().
    4. If LoadLibrary() succeeded, see if the file exports DllRegisterServer() and DllUnregisterServer().
    5. If both exports are found, add the filename to our list of files we can operate on, m_lsFiles.
    Initialize proc this_:DWORD, pidlFolder:DWORD, pDataObj:DWORD, hProgID:DWORD   
      LOCAL szFile[MAX_PATH]:BYTE
      LOCAL szFolder[MAX_PATH]:BYTE
      LOCAL szCurrDir[MAX_PATH]:BYTE
      LOCAL pszLastBackslash:DWORD
      LOCAL uNumFiles:DWORD
      LOCAL hdrop:DWORD
      LOCAL hLib:DWORD
      LOCAL bChangedDir:BYTE
      LOCAL fmt:FORMATETC
      LOCAL stg:STGMEDIUM
      LOCAL hDrop:DWORD
      LOCAL hResult:DWORD
      LOCAL pfn:DWORD
      
      ;Initialization of fmt
      mov fmt.cfFormat, CF_HDROP 
      mov fmt.ptd, NULL
      mov fmt.dwAspect, DVASPECT_CONTENT
      mov fmt.lindex, -1
      mov fmt.tymed, TYMED_HGLOBAL 
    
      ;Initialization of stg
      mov stg.tymed, TYMED_HGLOBAL
      
      mov bChangedDir, FALSE
    

    Tons of boring local variables! The first step is to get an HDROP from the pDataObjpassed in. This is done just like in the Part I extension.

      ;Look for CF_HDROP data in the data object.
      coinvoke pDataObj, IDataObject, GetData, addr fmt, addr stg
      .IF_FAILED
        ;Nope! return an "invalid argument" error back to Explorer.
        mov hResult, E_INVALIDARG
        jmp @F
      .endif
      
      .if stg.hGlobal == NULL
        mov hResult, E_INVALIDARG
        jmp @F
      .endif
    
      ;Initialization of our list
      pObjectData this_, ecx  ; cast this_ to object data
      push ecx
      
      lea ecx, (asmDllRegExtObjData ptr [ecx]).m_lsFiles
      invoke memfill, ecx, sizeof strList, 0
      
      ;Initialization of our boolean
      pop ecx
      
      ;Sanity check  make sure there is at least one filename.
      invoke DragQueryFile, stg.hGlobal, 0FFFFFFFFh, NULL, 0
      mov uNumFiles, eax
    

    Next comes a for loop that gets the next filename (using DragQueryFile()) and tries to load it with LoadLibrary(). The real shell extension in the sample project does some directory-changing beforehand, which I have omitted here since it's a bit long.

      xor ecx, ecx
        
      .while ecx < eax
        push ecx
        push eax
    
        ;Get the next filename.
      
        invoke DragQueryFile, stg.hGlobal, ecx, addr szFile, MAX_PATH
        .if !eax
          jmp continue
        .endif
          
        ;Try to load the DLL.  
        invoke LoadLibrary, addr szFile
        mov hLib, eax
        
        .if (!eax)
          jmp continue
        .endif
    

    Next, we'll see if the module exports the two required functions.

        ;Get the address of DllRegisterServer();
        dsText szDllReg, "DllRegisterServer"
        invoke GetProcAddress, hLib, addr szDllReg
        mov pfn, eax
      
        ;If it wasn't found, skip the file. 
        .if !eax
          invoke FreeLibrary, hLib
          jmp continue
        .endif
      
        ;Get the address of DllUnregisterServer();
        dsText szDllUnreg, "DllUnregisterServer"
        invoke GetProcAddress, hLib, addr szDllUnreg
        mov pfn, eax
      
        ;If it was found, we can operate on the file, so add it to
        ;our list o' files (m_lsFiles).
      
        .if eax
          pObjectData this_, ecx  ; cast this_ to object data
          lea ecx, (asmDllRegExtObjData ptr [ecx]).m_lsFiles
          invoke strlist_add, g_hHeap, ecx, ADDR szFile       
        .endif
      
        invoke FreeLibrary, hLib
        
    continue:
        pop eax
        pop ecx
        inc ecx
      .endw
    

    The last step (in the last if block) adds the filename to m_lsFiles, which is a list that holds strings. That list will be used later, when we iterate over all the files and register or unregister them.

    The last thing to do in Initialize() is free up resources and return the right value back to Explorer.

       ;Release resources.
    
      invoke ReleaseStgMedium, addr stg
    
      ;If we found any files we can work with, return S_OK.  Otherwise,
      ;return E_INVALIDARG so we don't get called again for this right-click
      ;operation.
    
      pObjectData this_, ecx  ; cast this_ to object data
      lea ecx, (asmDllRegExtObjData ptr [ecx]).m_lsFiles
      strlist_getCount ecx
      .if eax
        mov hResult, S_OK
      .else
        mov hResult, E_INVALIDARG
      .endif
    
    @@:
      return hResult
    Initialize endp
    

    If you take a look at the sample project's code, you'll see that I have to figure out which directory is being viewed by looking at the names of the files. You might wonder why I don't just use the pidlFolder parameter, which is documented as "the item identifier list for the folder that contains the item whose context menu is being displayed." Well, during my testing on Windows 98, this parameter was always NULL, so it's useless.

    Adding our menu items

    Next up are the IContextMenu methods. As before, you'll need to add IContextMenu to the list of interfaces that DllRegShlExtimplements. And once again, the steps for doing this are in Part I of the Guide.

    We'll add two items to the menu, one to register the selected files, and another to unregister them. The items look like this:

     [context menu - 5K]

    Our QueryContextMenu() implementation starts out like in Part I. We check uFlags, and return immediately if the CMF_DEFAULTONLY flag is present.


    QueryContextMenu proc this_:DWORD, hmenu:DWORD, uMenuIndex:DWORD,\
      uidFirstCmd:DWORD, uidLastCmd:DWORD, uFlags:DWORD
      LOCAL hResult:DWORD
      LOCAL uCmdId:DWORD
      
      mov eax, uidFirstCmd
      mov uCmdId, eax
      
      ;If the flags include CMF_DEFAULTONLY then we shouldn't do anything.
      .if uFlags & CMF_DEFAULTONLY
         MAKE_HRESULT SEVERITY_SUCCESS, FACILITY_NULL, 0
         mov hResult, eax
      .endif
    

    Next up, we add the "Register servers" menu item. There's something new here: we set a bitmap for the item. This is the same thing that WinZip does to have the little folder-in-a-vice icon appear next to its own menu items.


      ;Add our register/unregister items.
      dsText szRegister, "Register Server(s)"
      invoke InsertMenu, hmenu, uMenuIndex, MF_BYPOSITION, uCmdId, offset szRegister
      inc uCmdId
      
      ;Set the bitmap for the register item.
      pObjectData this_, ecx  ; cast this_ to object data
      mov ecx, (asmDllRegExtObjData ptr [ecx]).m_hRegBmp
      .if ecx
        invoke SetMenuItemBitmaps, hmenu, uMenuIndex, MF_BYPOSITION, ecx, NULL
      .endif
      
      inc uMenuIndex
    

    The SetMenuItemBitmaps() API is how we show our little gears icon next to the "Register servers" item. Note that uCmdID is incremented, so that the next time we call InsertMenu(), the command ID will be one more than the previous value. At the end of this step, uMenuIndex is incremented so our second item will appear after the first one.

    And speaking of the second menu item, we add that next. It's almost identical to the code for the first item.


      ;Set the bitmap for the unregister item.
      dsText szUnRegister, "Unregister Server(s)"
      invoke InsertMenu, hmenu, uMenuIndex, MF_BYPOSITION, uCmdId, offset szUnRegister
      inc uCmdId
    
      pObjectData this_, ecx  ; cast this_ to object data
      mov ecx, (asmDllRegExtObjData ptr [ecx]).m_hUnRegBmp
      .if ecx
        invoke SetMenuItemBitmaps, hmenu, uMenuIndex, MF_BYPOSITION, ecx, NULL
      .endif
    

    And at the end, we tell Explorer how many items we added.


      ;The return value tells the shell how many top-level items we added.
      MAKE_HRESULT SEVERITY_SUCCESS, FACILITY_NULL, 2
      mov hResult, eax
        
      return hResult
    QueryContextMenu endp
    

    Providing fly-by help and a verb

    As before, the GetCommandString() method is called when Explorer needs to show fly-by help or get a verb for one of our commands. This extension is different than the last one in that we have 2 menu items, so we need to examine the uCmdID parameter to tell which item Explorer is calling us about.


    GetCommandString proc this_:DWORD, idCmd:DWORD, uFlags:DWORD,\
      pwReserved:DWORD, pszName:DWORD, cchMax:DWORD
      LOCAL hResult
      LOCAL wcszHelpString[MAX_PATH]:WCHAR
      LOCAL szHelpString[MAX_PATH]:BYTE
    
      dsText szRegServers, "Register all selected COM servers"
      dsText szUnRegServers, "Unregister all selected COM servers"
      
      dsText szDllRegSvr, "DllRegSvr"
      dsText szDllUnRegSvr, "DllUnregSvr"
    
      ;If Explorer is asking for a help string, copy our string into the
      ;supplied buffer.
      .if uFlags & GCS_HELPTEXT
    
        ;Copy the help text into the supplied buffer.  If the shell wants
        ;a Unicode string, we need to case szName to an LPCWSTR.
        .if idCmd == 0
          invoke lstrcpyn, pszName, addr szRegServers, cchMax
        .elseif idCmd==1
          invoke lstrcpyn, pszName, addr szUnRegServers, cchMax
        .else
          mov hResult, E_INVALIDARG
          jmp @F
        .endif
    

    If uCmdID is 0, then we are being called for our first item (register). If it's 1, then we're being called for the second item (unregister). After we determine the help string, we copy it into the supplied buffer, converting to Unicode first if necessary.


        .if uFlags & GCS_HELPTEXTW
          invoke MultiByteToWideChar, CP_ACP, 0, pszName, -1, addr wcszHelpString, cchMax
          invoke lstrcpynW, pszName, addr wcszHelpString, cchMax
        .endif
    

    For this extension, I also wrote code that provides a verb. However, when testing on Windows 98, Explorer never called GetCommandString() to get a verb. I even wrote a test app that called ShellExecute() on a DLL and tried to use a verb, but that didn't work either. I don't know if the behavior on NT is different. I have omitted the verb-related code here, but you can check it out in the sample project if you're interested.

    Carrying out the user's selection

    When the user clicks one of our menu items, Explorer calls our InvokeCommand() method. InvokeCommand() first checks the high word of lpVerb. If it's non-zero, then it is the name of the verb that was invoked. Since we know verbs aren't working properly (at least on Win 98), we'll bail out. Otherwise, if the low word of lpVerb is 0 or 1, we know one of our two menu items was clicked.


    InvokeCommand proc this_:DWORD, pCmdInfo:DWORD
      LOCAL hResult:DWORD
      LOCAL szMsg[MAX_PATH+32]:BYTE
      LOCAL MyDlgParam:dlgParam
    
      ;If lpVerb really points to a string, ignore this function call and bail out.
      mov ecx, pCmdInfo
      mov ecx, (CMINVOKECOMMANDINFO PTR[ecx]).lpVerb
      push ecx
      HIWORD ecx
      .if eax
        mov hResult, E_INVALIDARG
        jmp @F
      .endif
    
      ;Get the command index - the only valid one is 0.
      pop ecx
      LOWORD ecx
      .if eax==0 || eax==1
        ;fill our private struct
        pObjectData this_, ecx  ; cast this_ to object data
        lea ecx, (asmDllRegExtObjData ptr [ecx]).m_lsFiles
        mov MyDlgParam.plsFiles, ecx
    
        mov eax, pCmdInfo
        mov MyDlgParam.pCmdInfo, eax
        
        mov ecx, pCmdInfo
        mov ecx, (CMINVOKECOMMANDINFO PTR[ecx]).hwnd
    		invoke DialogBoxParam, g_hModule, IDD_PROGRESSDLG, ecx, OFFSET DlgProc, addr MyDlgParam
    
        mov hResult, S_OK
      .else
        mov hResult, E_INVALIDARG
        jmp @F
      .endif
      
    @@:        
      return hResult
    InvokeCommand endp
    

    If lpVerb is 0 or 1, we create a progress dialog, and pass it the list of filenames and pCmdInfo.

    All of the real work happens in the progress dialog. Its dlgInit function initializes the list control, and then calls DoWork. DoWork() iterates over the string list that was built in Initialize(), and calls the appropriate function in each file. The basic code is below; it is not complete, since I've left out the error-checking code, and the parts that fill the list control. It's just enough to demonstrate how to iterate over the list of filenames and act on each one.


    DoWork PROC hWnd:DWORD, hList:DWORD, pdlgParam:DWORD
      LOCAL pfn:DWORD, hinst:DWORD, pszFnName:DWORD, hr:DWORD
      LOCAL pvMsgBuf:DWORD, nIndex:DWORD
      LOCAL szMsg[512]:BYTE
      LOCAL wCmd:WORD
      LOCAL rItem:LVITEM
      LOCAL msg:MSG
      LOCAL bSuccess:BYTE
      LOCAL pActuel:DWORD, dwLastErr:DWORD
    
      invoke memfill, addr lsStatusMessages, sizeof strList, 0
      
      mov nIndex, 0
      
      mov ecx, pdlgParam
      mov ecx, (dlgParam PTR[ecx]).pCmdInfo
      mov ecx, (CMINVOKECOMMANDINFO PTR[ecx]).lpVerb
    
      LOWORD ecx
    
      ;We only support 2 commands, so check the value passed in lpVerb.
      .if eax!=0 &&  eax!=1
        jmp fin
      .endif
      
      ;Determine which function we'll be calling.  
      .if eax
        mov pszFnName, offset szDllUnreg
      .else
        mov pszFnName, offset szDllReg
      .endif
    
      mov ecx, pdlgParam
      mov ecx, (dlgParam PTR[ecx]).plsFiles
      
      ;Get the first filename
      mov eax, (strList PTR[ecx]).pFirst
      mov pActuel, eax
    
      ;Get the number of filenames in the list
      strlist_getCount ecx
      xor ecx, ecx
      .while ecx < eax
        push ecx 
        push eax
        
        mov bSuccess, FALSE
        mov pvMsgBuf, NULL
        mov hinst, NULL
        mov szMsg, 0
    
        ;We will print a status message into szMsg, which will eventually
        ;be stored in the LPARAM of a listview control item.
          ;Try to load the next file.
          strlist_getData pActuel      
          mov ecx, eax        
    
          invoke LoadLibrary, ecx
          mov hinst, eax
    
          ;If it failed, construct a friendly error message.
          .if !eax
            jmp continue
          .endif
    
          ;Get the address of the register/unregister function.
          invoke GetProcAddress, hinst, pszFnName
          mov pfn, eax
    
          ;If it wasn't found, construct an error message.
          .if !eax
            jmp continue
          .endif
    
          ;Call the function!
          call eax
    

    Registering the shell extension

    The DllReg extension operates on executable files, so let's register it to be invoked on EXE, DLL, and OCX files. As in Part I, we can do this through the RGS script, DllRegShlExt.rgs. Here's the necessary script to register our DLL as a context menu handler for each of those extensions.

    HKCR
    {
        NoRemove dllfile
        {
            NoRemove shellex
            {
                NoRemove ContextMenuHandlers
                {
                    ForceRemove DLLRegSvr = s '{8AB81E72-CB2F-11D3-8D3B-AC2F34F1FA3C}'
                }
            }
        }
        NoRemove ocxfile
        {
            NoRemove shellex
            {
                NoRemove ContextMenuHandlers
                {
                    ForceRemove DLLRegSvr = s '{8AB81E72-CB2F-11D3-8D3B-AC2F34F1FA3C}'
                }
            }
        }
    }

    The format of the RGS file, and the keywords NoRemove and ForceRemove are explained in Part I, in case you've forgotten their meaning.

    As in our previous extension, on NT/2000 we need to add our extension to the list of "approved" extensions. The code to do this is in the DllRegisterServer() and DllUnregisterServer() functions. I won't show the code here, since it's just simple registry access, but you can find the code in this article's sample project.

    What does it all look like?

    When you click one of our menu items, the progress dialog is displayed and shows the results of the operations:

     [progress dialog - 21K]

    The list control shows the name of each file, and whether the function call succeeded or not. When you select a file, a message is shown beneath the list that gives more details, along with a description of the error if the function call failed.

    Other ways to register the extension

    So far, our extensions have been invoked only for certain file types. It's possible to have the shell call our extension for any file by registering as a context menu handler under the HKCR\* key:

    HKCR
    {
        NoRemove *
        {
            NoRemove shellex
            {
                NoRemove ContextMenuHandlers
                {
                    ForceRemove DLLRegSvr = s '{8AB81E72-CB2F-11D3-8D3B-AC2F34F1FA3C}'
                }
            }
        }
    }

    The HKCR\* key lists shell extensions that are called for all files. Note that the docs say that the extensions are also invoked for any shell object (meaning files, directories, virtual folders, Control Panel items, etc.), but that was not the behavior I saw during my testing. The extension was only invoked for files in the file system.

    In shell version 4.71+, there is also a key called HKCR\AllFileSystemObjects. If we register under this key, our extension is invoked for all files and directories in the file system, except root directories. (Extensions that are invoked for root directories are registered under HKCR\Drive.) However, I saw some strange behavior when registering under this key. The SendTo menu uses this key as well, and the DllReg menu items ended up being mixed in with the SendTo item:

     [context menu - 7K ]

    Finally, in shell version 4.71+, you can have a context menu extension invoked when the user right-clicks the background of an Explorer window that's viewing a directory (including the desktop). To have your extension invoked like this, register it under HKCR\Directory\Background\shellex\ContextMenuHandlers. Using this method, you can add your own menu items to the desktop context menu, or the menu for any other directory. The parameters passed to Initialize() are a bit different, though, so I may cover this topic in a future article.


    The component in asm weigth only 21Ko, the same one coded with ATL is 36Ko !!!

    [COMMENTS]

    Copyright © Maurice MONTGENIE - http://www.com4me.net