Getting Started with Replay Debugging

The article will get you started using the replay debugging features of VMware Workstation 6.5. What is replay debugging? Replay debugging allows C/C++ developers to use Visual Studio to debug recordings of programs running inside virtual machines. This is really useful for difficult-to-reproduce bugs. Check out this article for elaboration. You might also find the Workstation manual (see the side bar) a useful reference.

Note: If you find errors/ambiguities in this article, please leave comments here (use the link at the end of this article). If you encounter any problems with VMware Workstation, please post your concerns or comments on the VMware replay debugging forum. Thanks!

Requirements

Please ensure that your hardware and software are supported.

• Processor: P4, Core 2, Penryn (stepping B0+), Barcelona (stepping B3+)
• Host OS: 32/64-bit Windows XP SP2, Windows Server 2003, Vista (Recommended: 32-bit Windows XP SP2)
• Guest OS: 32-bit Windows XP, Windows Server 2003, Vista (Recommended: Windows XP SP2)
• Debugger: Visual Studio 2005 SP1 or Visual Studio 2008 (Recommended: Visual Studio 2005)
• VMware Workstation: Recent version of Workstation 6.5 (release candidate 2 or newer)

Installation

Please ensure that the necessary software is installed.

• Visual Studio: Ensure that you have installed Visual Studio before installing the latest version of VMware Workstation. This is necessary to ensure that the Workstation installation process installs the Visual Studio plugin.
• VMware Workstation 6.5: Ensure that you install Workstation after Visual Studio. Install Workstation 6.5 (release candidate 1 is available here for beta program members (free signup)). When it is released, Workstation 6.5 will be a free upgrade for all Workstation 6.0 license holders.
• Guest operating system: Ensure that a supported version of Windows (above) is installed in the virtual machine you intend to use for debugging. This virtual machine can only have one virtual processor. You find that performance is better if the virtual machine has a relatively small amount of memory (say 256MB) and if background snapshots are disabled (Edit > Preferences > Priority tab).
• Guest tools: Ensure that the latest version of guest tools are installed in the Windows virtual machine. This is essential. Replay debugging will not work without the latest tools.

Power On Your Virtual Machine

Ensure that the virtual machine is working, and don't forget to install the latest guest tools! This is a good time to create a snapshot (let's call it "InitialState"). We'll use this as the starting point for the recordings we create.

Create a Visual Studio Project

Start Visual Studio. You should see a VMware menu. If you do not, this indicates that VMware Workstation was not properly installed (did you ensure that Workstation was installed after Visual Studio was installed?).

Create a Visual Studio project as follows.

• File > New > Projects...

• Project type: Visual C++ (Win32)
• Visual Studio installed templates: Win32 Project
• Name: HiMom
• In Win32 Application Wizard, set application type to Console application

It is essential that the program we create be able to run in our virtual machine, so we must not use any DLLs that are not present in the VM. You'll probably need to change the project properties as follows.

• Project > HiMom Properties
• Configuration Properties > C/C++ > Code Generation: change runtime library to "Multi-threaded Debug (/MTd)"

Now edit your program in HiMom.cpp (make your mother proud!), save it, compile it (Build > Build Solution), and run it (Debug > Start Debugging). Okay, that's nice, but it doesn't look like replay debugging. It isn't. Be patient! We need to configure the VMware Options first.

Configure VMware Options in Visual Studio

You need to tell Visual Studio what virtual machine you would like to use for debugging.

• Visual Studio: VMware > Options...
• Configuration Properties > Replay Debugging in VM > General: Make sure you are changing the replay debugging properties (the remote debugging properties will have no effect for us). Set the "Virtual Machine" field to the full path of the .vmx file of the virtual machine you would like to use to debug (note that you can browse the filesystem to find this file). The "Recording to Replay" field specifies the recording you would like to use for debugging, but as you haven't created a recording yet, you can ignore this.
  

• Configuration Properties > Replay Debugging in VM > Advanced: Set the Base Snapshot for Recording field to the name of the snapshot you created, above ("InitialState"). Just before creating a recording, this snapshot will be restored. If no snapshot is specified, the virtual machine has to be powered on (and we don't want to wait for that!).

Create a Recording

Before we can debug a recording, we need to create a recording. Makes sense. We can do this via the VMware menu (VMware > Create Recording for Replay). This will start up Workstation, ask for your guest login credentials, restore the "InitialState" snapshot, share the program to debug (HiMom.exe) with the virtual machine, start a recording, run the program, and stop the recording. Note that this step automatically populates the "Recording to Replay" field in the General options with the name of the recording you just created (it'll have a boring name like "Recording 1").

Troubleshooting: If your program never starts, this may be because the program can't run in the VM (e.g., because it's missing a needed DLL). The best way to diagnose these problems is to copy the program to debug into the VM (drag and drop works well) and run it manually. This usually provides sufficient diagnostic information to figure out what the problem is.

Start Debugging

Now that you've created a recording in which the program is running, let's debug it. Set a breakpoint on the first line of main and select VMware > Start Replay Debugging. The recording will start replaying and you'll hit the breakpoint. You should be able to examine local variables, single step, set other breakpoints, and do (almost) all the usual debugging activities.

The important difference with regular debugging is that here you are debugging a recording of the program, so you cannot change the behavior of the program as you debug it. This means you can't change the value of variables/registers nor can you execute code that was not executed in the recording. In addition, you will notice that input to the program (network, keyboard, etc.) need not be provided during replay debugging, because the input provided while the recording was created is reused during replay.

Reverse Execution

Please try out the reverse execution feature (VMware > Reverse Continue). This feature simulates reverse execution by replaying from an earlier point in time. This is particularly useful for tracking down memory corruption problems. When you encountered a corrupted data value, place a data breakpoint on this data value and execute backward to find out where it was corrupted! In a subsequent article I'll discuss methods for improving the performance of reverse execution.

An Alternate Method for Creating a Recording

Recordings can be created via Visual Studio (see above), but they can also be created using the Workstation UI. You simply copy the program you would like to debug to the VM (or place it in a shared file system), start recording, start the program, and stop recording. That's it. Naturally, you'll have to configure Visual Studio (via the VMware > Options... menu) to use the recording you have created.

VMware Workstation 6.5: Reverse and Replay Debugging is Here!

I'm proud to announce that VMware Workstation 6.5 includes new experimental features that provide replay debugging for C/C++ developers using Microsoft Visual Studio. Replay debugging allows developers to debug recordings of programs running in virtual machines, and it is valuable for finding, diagnosing, and fixing bugs that are not easily reproduced, a particularly challenging class of bugs. Once the manifestation of a bug has been recorded, it can be replayed (and debugged) over and over again, and it is guaranteed to have instruction-by-instruction identical behavior each time. In addition, Workstation includes a feature that simulates reverse execution of the program, making it easier to pin point the origin of a bug.

Question: How can replay debugging help me?

Answer: Replay debugging helps find, diagnose, and fix bugs that are not easily reproduced. Consider the following examples:

Non-deterministic bugs. Some bugs only rear their ugly heads when a particular interleaving of concurrent entities (e.g., threads, the operating system, network activity, and user input) is present. Given that the programmer cannot normally control (or even reason about) this interleaving, these bugs can be very difficult to reproduce. But if such a bug is recorded, we can examine it in the debugger as often as we like. Once a bug is recorded, it is no longer non-deterministic.

Pseudo non-deterministic bugs. Some bugs are deterministic, but it's not always clear exactly what is necessary to reproduce the bug. Bug reports indicating that a program crashed "...after clicking on a bunch of different UI elements," are common but usually useless. If the bug is recorded, it is unnecessary to describe the steps for reproduction, because they are an implicit part of the recording itself.

Bugs that can only be reproduced with a complex environment. If a bug appears in an application that communicates with dozens of other local and remote processes that need to be setup and configured, reproducing the bug may be possible but prohibitively expensive. If the the bug is recorded, the developer need not setup the other components of the system (e.g., other processes or remote clients). The effect of the other components is recorded, allowing the program of interest to be debugged in isolation.

Heisenbugs. A Heisenbug (a play on the Heisenberg uncertainly principle) is a bug that does not manifest itself when it is examined in the debugger. Such bugs are common because debugging is an invasive process that dramatically changes the timing characteristics of the program. If a bug is recorded, it can be debugged without impacting the behavior of the program at all.

Memory corruption bugs. Memory corruption bugs are difficult to diagnose because almost any pointer dereference in the program can be responsible for the corruption. The replay debugging facility in VMware Workstation 6.5 includes a reverse execution feature that (among other things) greatly helps track down memory corruption bugs. When a corrupted data structure is identified, the developer simply sets a data breakpoint (i.e., a watchpoint) on the corrupted data and executes backward. This will take the developer to the last point in time that the corrupted data structure was modified.

Question: Do I have to learn a new debugging environment?

Answer: No, our replay debugging features are tightly integrated with the Visual Studio development environment. Aside from the replay-specific features, your debugging experience will be the same as a traditional debugging experience.

Question: How do I create recordings?

Answer: Our extension to Visual Studio provides a facility for creating recordings from within Visual Studio. Alternatively, recordings can be created from the VMware Workstation user interface. The later is useful when someone other than the developer (e.g., a tester) is creating recordings.

Question: How can I get it?

Answer: A release candidate for VMware Workstation 6.5 is currently available here. The release candidate is available to members of the beta program (free signup). The release version of the product will be available soon. Workstation 6.5 is a free upgrade for all Workstation 6.0 license holders.

Question: Where can I learn more?

Answer: We have created a number of resources to get you up and running with replay debugging.

Introductory video. This video introduces the replay debugging feature.

Tutorial screen cast. [Coming Soon!] This screencast shows the configuration and use of replay debugging.

Replay debugging forum. This form is a place for users and VMware engineers to get together to discuss replay debugging experiences, problems, and requests.

This blog. This blog will be updated with topics related to replay debugging. I expect to add articles on use cases, best practices, pitfalls, and so on.

VMware Workstation 6.5 Manual. The Workstation manual includes an appendix describing the Integrate Virtual Debugger for Visual Studio, which includes the replay debugging features. This is a useful reference but it does not include much more than the basic facts.