Getting Started with Replay Debugging

The article will get you started using the replay debugging features of VMware Workstation 6.5. What is replay debugging? Replay debugging allows C/C++ developers to use Visual Studio to debug recordings of programs running inside virtual machines. This is really useful for difficult-to-reproduce bugs. Check out this article for elaboration. You might also find the Workstation manual (see the side bar) a useful reference.

Note: If you find errors/ambiguities in this article, please leave comments here (use the link at the end of this article). If you encounter any problems with VMware Workstation, please post your concerns or comments on the VMware replay debugging forum. Thanks!

Requirements

Please ensure that your hardware and software are supported.

• Processor: P4, Core 2, Penryn (stepping B0+), Barcelona (stepping B3+)
• Host OS: 32/64-bit Windows XP SP2, Windows Server 2003, Vista (Recommended: 32-bit Windows XP SP2)
• Guest OS: 32-bit Windows XP, Windows Server 2003, Vista (Recommended: Windows XP SP2)
• Debugger: Visual Studio 2005 SP1 or Visual Studio 2008 (Recommended: Visual Studio 2005)
• VMware Workstation: Recent version of Workstation 6.5 (release candidate 2 or newer)

Installation

Please ensure that the necessary software is installed.

• Visual Studio: Ensure that you have installed Visual Studio before installing the latest version of VMware Workstation. This is necessary to ensure that the Workstation installation process installs the Visual Studio plugin.
• VMware Workstation 6.5: Ensure that you install Workstation after Visual Studio. Install Workstation 6.5 (release candidate 1 is available here for beta program members (free signup)). When it is released, Workstation 6.5 will be a free upgrade for all Workstation 6.0 license holders.
• Guest operating system: Ensure that a supported version of Windows (above) is installed in the virtual machine you intend to use for debugging. This virtual machine can only have one virtual processor. You find that performance is better if the virtual machine has a relatively small amount of memory (say 256MB) and if background snapshots are disabled (Edit > Preferences > Priority tab).
• Guest tools: Ensure that the latest version of guest tools are installed in the Windows virtual machine. This is essential. Replay debugging will not work without the latest tools.

Power On Your Virtual Machine

Ensure that the virtual machine is working, and don't forget to install the latest guest tools! This is a good time to create a snapshot (let's call it "InitialState"). We'll use this as the starting point for the recordings we create.

Create a Visual Studio Project

Start Visual Studio. You should see a VMware menu. If you do not, this indicates that VMware Workstation was not properly installed (did you ensure that Workstation was installed after Visual Studio was installed?).

Create a Visual Studio project as follows.

• File > New > Projects...

• Project type: Visual C++ (Win32)
• Visual Studio installed templates: Win32 Project
• Name: HiMom
• In Win32 Application Wizard, set application type to Console application

It is essential that the program we create be able to run in our virtual machine, so we must not use any DLLs that are not present in the VM. You'll probably need to change the project properties as follows.

• Project > HiMom Properties
• Configuration Properties > C/C++ > Code Generation: change runtime library to "Multi-threaded Debug (/MTd)"

Now edit your program in HiMom.cpp (make your mother proud!), save it, compile it (Build > Build Solution), and run it (Debug > Start Debugging). Okay, that's nice, but it doesn't look like replay debugging. It isn't. Be patient! We need to configure the VMware Options first.

Configure VMware Options in Visual Studio

You need to tell Visual Studio what virtual machine you would like to use for debugging.

• Visual Studio: VMware > Options...
• Configuration Properties > Replay Debugging in VM > General: Make sure you are changing the replay debugging properties (the remote debugging properties will have no effect for us). Set the "Virtual Machine" field to the full path of the .vmx file of the virtual machine you would like to use to debug (note that you can browse the filesystem to find this file). The "Recording to Replay" field specifies the recording you would like to use for debugging, but as you haven't created a recording yet, you can ignore this.
  

• Configuration Properties > Replay Debugging in VM > Advanced: Set the Base Snapshot for Recording field to the name of the snapshot you created, above ("InitialState"). Just before creating a recording, this snapshot will be restored. If no snapshot is specified, the virtual machine has to be powered on (and we don't want to wait for that!).

Create a Recording

Before we can debug a recording, we need to create a recording. Makes sense. We can do this via the VMware menu (VMware > Create Recording for Replay). This will start up Workstation, ask for your guest login credentials, restore the "InitialState" snapshot, share the program to debug (HiMom.exe) with the virtual machine, start a recording, run the program, and stop the recording. Note that this step automatically populates the "Recording to Replay" field in the General options with the name of the recording you just created (it'll have a boring name like "Recording 1").

Troubleshooting: If your program never starts, this may be because the program can't run in the VM (e.g., because it's missing a needed DLL). The best way to diagnose these problems is to copy the program to debug into the VM (drag and drop works well) and run it manually. This usually provides sufficient diagnostic information to figure out what the problem is.

Start Debugging

Now that you've created a recording in which the program is running, let's debug it. Set a breakpoint on the first line of main and select VMware > Start Replay Debugging. The recording will start replaying and you'll hit the breakpoint. You should be able to examine local variables, single step, set other breakpoints, and do (almost) all the usual debugging activities.

The important difference with regular debugging is that here you are debugging a recording of the program, so you cannot change the behavior of the program as you debug it. This means you can't change the value of variables/registers nor can you execute code that was not executed in the recording. In addition, you will notice that input to the program (network, keyboard, etc.) need not be provided during replay debugging, because the input provided while the recording was created is reused during replay.

Reverse Execution

Please try out the reverse execution feature (VMware > Reverse Continue). This feature simulates reverse execution by replaying from an earlier point in time. This is particularly useful for tracking down memory corruption problems. When you encountered a corrupted data value, place a data breakpoint on this data value and execute backward to find out where it was corrupted! In a subsequent article I'll discuss methods for improving the performance of reverse execution.

An Alternate Method for Creating a Recording

Recordings can be created via Visual Studio (see above), but they can also be created using the Workstation UI. You simply copy the program you would like to debug to the VM (or place it in a shared file system), start recording, start the program, and stop recording. That's it. Naturally, you'll have to configure Visual Studio (via the VMware > Options... menu) to use the recording you have created.