Establishing a Debugging Environment

Debugging IIS5 ISAPI Applications with VC++

Rick Strahl - http://www.west-wind.com/

This document is based on input from Wade Hilmo of Microsoft which enabled me to debug my ISAPI extensions with IIS 5.0. I've amended some extra notes that were required to get things rolling. I have not verified his notes on debugging Filters or InProcess applications as described in the Notes section and below.

For debugging IIS 4 ISAPI extensions click here.

IIS 5 Background

IISRestart Manager
First off, even before you try and debug, you should be aware that IIS 5 works with Windows 2000 Service Control Manager to automatically recover from failures. By default the IISRestart service is running which causes IIS to be automatically restarted if it crashes or hangs. Because of this, any time the inetinfo process terminates unexpectedly (like when a debugger stops it), SCM will restart it. You can either turn off that service permanently in the Service Manager on your debug machine, or you can disable the service temporarily with "iisreset /disable" from a console command prompt. Use "iisreset /enable" to have it start monitoring again.

IIS 5 and Web Application Processes
The next suggestion is to not try to debug IIS inside of the IIS process. IIS 5 extends the concept of an Out of Process environmen for hosting ISAPI extensions introduced in IIS 4 with virtual applications. The idea is to protect the the core Web Service from crashing when an ISAPI extension is ill-behaved. These Out of Process components are controlled through Microsoft Transaction Server using the dllhost.exe MTS containers.

IIS 5 supports three methods of Application Protection:

Low (IIS Process)
Any ISAPI extensions run in the IIS process as part of the Web Service. Crash the extension there's a good chance the Web server goes with it.
Medium (Pooled ISAPI)
All ISAPI extensions run in a single, shared external module. This is IIS's new default. This means all medium applications run in the same MTS application. Crashing one could crash all the others in the medium pool, but won't crash the Web server. MTS most likely would catch the crash and restart the application.
High (Isolated)
This is similar to virtual applications created in IIS 4 where each application gets its own MTS application. The application is totally isolated and when it crashes it doesn't affect the rest of the Web server.

The higher the isolation the less problematic a crash or hang becomes for the rest of the Web server, but more performance overhead there is.

Debugging an ISAPI extension

What does this have to do with debugging? Well, it's much easier and recommended by Microsoft that you debug the Out of Process component rather than the InProcess service. Here are the steps to set this up with Visual C++.

Shut down the Web server
Shut down IIS by running "net stop iisadmin /y".

Run IIS in Out-of-Process mode
Set up IIS to run in Pooled or Isolated mode. To do this bring up your Web site in the IIS Management Console, click on the Home Directory tab and use the Application Protection combo to select this value.

Configure the MTS Application settings
Use the "Component Services" tool to configure the OOP pool to start under a debugger. To do this use the Start Menu | Administrative Tools | Component Services. Navigate to Console Root -> Component Services -> Computers - > My Computer -> COM + Applications. Right click on "IIS Out-Of-Process Pooled Applications" and pick "Properties". On the Advanced page, check Launch in debugger. In the Debugger Path text box type in the path to msdev.exe and the path to the process to be debugged and the Processid to be debugged as identified by a GUID. This will look something like this (one line):

"D:\vstudio\MSDev98\Bin\msdev.exe" dllhost.exe /ProcessID:{3D14228D-FBE1-11D0-995D-00C04FD919C1}
This will be the default in the textbox so this really should never have to change.

Configure the MTS Security context for debugging
Still in the Component Configuration dialog, configure the security of the Out of Process server. By default the server will run under the IWAM_ account when accessed through IIS. Typically you cannot launch the debugger under this user context and even when you do give IWAM_ admin rights you can't get the debugger visible as it can't interact with the desktop. The only way that I could get the debugger to work correctly is by switching the security context to the Interactive User. To do so go to the Identity tab and select the Interactive User radio.
NOTE: Once you do this you won't be able to reset the security context to IWAM_ unless you manually change the password for this account. IIS assigns a random password for this account so you can't simply reenter this account.

Set up your project
Now you're ready to start debugging. The above steps set up IIS so that the first time an ISAPI extension launches it pops into the debugger, which is your hook into VC++. In order to set breakpoints you have to compile your project in debug mode and put hard breakpoints into your code like so:

_asm int 3

When the code hits this line the debugger should pop up with your source code ready for the debug session.

Note that the debugger will pop up on any ISAPI extension firing not just your own. The exception appears to be ASP which is apparently running in another context altogether (not sure on this one).

Shutting down and restarting IIS
Once you're done debugging you can shut down IIS either by stopping the service or with the Resource Kit's KILL utility. Restart the Web service for another debugging session.

When you're done debugging
When you're done debugging you probably want undo the settings in reverse order. Most importantly remove the Launch in Debugger checkbox. As mentioned above resetting the Identity of the MTS component to IWAM_ requires you to assign the account a new password in User Manager and then assign the new password in the Identity list.

Notes

Now, the first time you access an ISAPI in the pool, Dev Studio will start with the appropriate dllhost.exe (it's not mtx.exe anymore, now it's dllhost.exe) attached. If you want to debug any other isolated OOP ISAPI, you can just pick the appropriate application in the Component Manager above.

Preloading symbols to hook DLLMain or GetExtensionVersion

You can get the symbols loaded for your ISAPI any number of ways. The easiest is to just request the ISAPI once, which will load it and it's symbols. The downside to this is that you cannot set breakpoints on the first request, or in DllMain or GetExtensionVersion. There are two ways that you can get the symbols preloaded. First, is to specify your dll as an "additional dll" in Dev Studio. If you do this, then your symbols will be preloaded the next time you start the OOP pool process. Alternately, you can tell NT to break every time your dll is loaded. This will cause the attached debugger to break after the dll loads, but before any of your code runs. You can then open your sources and set breakpoints. To do this, create the following registry key, where <dll> is the name of your dll without the path (ie. myisapi.dll):

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\
CurrentVersion\Image File Execution Options\<dll>

In that key, create a new string value called "BreakOnDllLoad" and set it to "1".

Until you change the value of BreakOnDllLoad, your dll will break any time a debugger is attached to the process that loads it, and it will have no affect if the process is not running under a debugger.

Debugging filters or true In Process ISAPI extensions

If you want to debug a filter, or an "in process" ISAPI, you can use the same registry key and attach Dev Studio's debugger to inetinfo.exe before loading your ISAPI. In the case of a filter, you can do "net start iisadmin" to start inetinfo without starting the web service and then attach. Then, your filter will break on load when you do "net start w3svc". Alternately, you can use the above "Image File Execution Options" key to tell Windows 2000 to start inetinfo.exe (or any other process) under a debugger. To do this, create the following key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\
CurrentVersion\Image File Execution Options\inetinfo.exe

In that key, create a new string value called "Debugger" and set it to contain the full path to the debugger (in our case, msdev.exe).

When you do this, inetinfo.exe will immediately break whenever you start it. When this happens, you want to hit "run" in the debugger some time within the first 30 seconds, or else SCM gets confused about the state of the service. If you've configured your dll to break on load, then that won't be a problem because it will break again when your dll loads.

Ok, so that's it. These debugging steps should be pretty robust, and you'll be debugging in the actual environment of IIS running as a service, and it handles the case where you want to debug your ISAPI running either isolated or in the pool, which was quite a bit more difficult under IIS 4.