QEMU Virtual Machines

Pre-requisites

IMPORTANT: QEMU is installed during the REVEN server installation process.

The next sections introduce useful QEMU notions to work with REVEN:

  • KVM vs Emulated QEMU VMs.
  • Disk and live snapshots in QEMU.

KVM vs Emulation

There are two ways to run VMs with QEMU: either using KVM (virtualized mode) or emulated. The former is much faster, but the latter is required when recording. Saved VM states cannot be shared between the two modes, but disk snapshots can.

In the Project Manager's VM view you can select either modes, but when recording a scenario only the emulation mode is available.

KVM mode is convenient when installing software or OSes. See typical workflow below.

For a complete documentation on QEMU tools, please refer to QEMU's online documentation.

How do snapshots work in QEMU

IMPORTANT This section is crucial: REVEN uses the native disk / live snapshot mechanism from QEMU which might differ from what users expect.

There are two types of snapshots available in the qcow file format:

  • Disk snapshots represent the state of a disk, and may be organized in a tree structure to save disk space.
  • Live snapshots represent the full state of a VM, including memory, cpu registers and disk. They are stored inside a disk snapshot, and are what most users expect.

The two are used in conjunction to provide various functionalities:

  • Live snapshots allow the user to store the full state of a VM:
    • Loading a live snapshot will allow restoring a VM that is booted.
    • These are tightly coupled to the options the VM has been started with: not selecting the right options will prevent snapshots from loading. These options include "kvm", "network", or any custom option.
    • They are immutable by design
  • Disk snapshots contains the disk only:
    • They are mutable: they live with the VM
    • Loading a live snapshot will alter the disk snapshot, by restoring it to the saved state. Any modification is lost.
    • Starting a VM from a disk snapshot will, by design, require a full boot.
    • They can be linked to parent disk snapshots, to limit disk usage on the host. Note that altering a parent snapshot may render children unusable!

Disk and Live snapshots

Adding QEMU VMs for scenario recording

Adding an existing VM with the preparation Wizard

REVEN offers a VM preparation Wizard that will guide you in the process of registering and preparing a new QEMU VM in REVEN.

Using this Wizard, you will be able to register:

  • VMs that are in supported formats and live in the directory identified as QUASAR_QEMU_SCAN_PATH in your settings (by default, ~/VMs). The supported formats are listed by the QUASAR_QEMU_SCAN_FORMATS variable in your settings, see the documentation of this variable in settings.py for more information.
  • Other VMs from your local disk or from a remote URL.

After registering a VM, the Wizard will guide you through the initial steps of snapshot creation and lightening of the VM.

Example 1: adding a Windows 10 VM provided by Microsoft

Microsoft provides some free Windows 10 virtual machines that you can download and use locally, such as the MSEdge VM. Please review the license terms of this VM before using it.

To use this VM with REVEN, please follow the steps below:

  1. Download the HyperV version from your browser. The HyperV version is preferred because it is known to be compatible, and is lighter than the other versions.
  2. Unzip the MSEdge.Win10.HyperV.zip file you just downloaded.
  3. Use the VM preparation Wizard to upload the Virtual Hard Disks/MSEdge - Win10.vhdx of the archive from your disk to the server.
  4. Let the VM Wizard guide you through the following steps.

Example 2: adding a VM in OVA format

To use a VM in Open Virtual Appliance (OVA) format with REVEN, please follow the steps below:

  1. Unzip the OVA archive.
  2. Use the VM preparation Wizard to upload the .vmdk or .vdi (depending on the OVA) file contained in the archive from your disk to the server.
  3. Let the VM Wizard guide you through the following steps.

Creating a new QEMU VM from an ISO

You can create a QEMU .qcow2 VM by installing an operating system from an ISO.

The Project Manager does not support all steps required for this operation. As a consequence, in conjunction with using the VM import Wizard, you will also need to run commands via a terminal on the machine REVEN is runnning on. Below are two recommended methods to approach this.

For both methods, you need to know where the Project Manager stores the virtual machine's disks (by default, in ~/VMs) - in the steps below, we will call this path /path/to/VMs/. You can find this out either:

  • By starting the registration of a QEMU VM. In the 2nd step, where the wizard asks you to select which disk to import, you can find the VM disk path on the label "Select a VM file from the existing files".
  • Alternatively, in your settings file as QUASAR_QEMU_SCAN_PATH.

You must also copy your installation ISO file on the server by your own means (sFTP, scp, or other). In the next steps, we will call the path to the ISO file /path/to/ISO.

Without direct or SSH access

  1. In the Project Manager, open the "NOTEBOOKS" link in the page's footer to open Jupyter.
  2. Once in jupyter, open a new terminal session as follow:
    Jupyter new terminal
  3. From the terminal, run qemu-img create -f qcow2 /path/to/VMs/myimage.qcow2 80G, where /path/to/VMs/ is the VM directory found above. Note you should adapt the name and size of the disk to your requirements.
  4. Once this disk is created, you can start importing it using the Project Manager VM import wizard as described in previous sections.
  5. When asked to create a child snapshot, do so (even if this disk is still empty). Write the name down, we will refer to it as root.
  6. The first time the wizard requests you start the VM:
    • Check "Override custom options",
    • Enter the custom option -cdrom "/path/to/ISO",
    • Click on "Start" to start the VM with your ISO file.
  7. If the VM does not show up, use the link "Show in browser". Your ISO file will take precedence in the boot as expected, allowing you to install the VM.
  8. Once your VM is installed, shut it down properly.
  9. Leave the Wizard open at this step and go back to the Jupyter terminal.
  10. At this point, your base disk is empty, and your OS as been installed to the first disk snapshot. Instead we want the base disk to contain this vanilla installation, and later use the first snapshot to lighten the VM further. Hence, "commit" all changes to the base image using the command qemu-img commit /path/to/VMs/myimage.qcow2.snapshots/root.qcow2.
  11. You can now close the Jupyter terminal. Head back to the wizard. You might get a warning on this page, in which case simply click on the "Refresh" button.
  12. Uncheck "Override custom options" then click on "Save Setings".
  13. You are now back into the normal workflow: you can stay at this step to perform the necessary operations to make your VM lighter on root by disabling unnecessary services, then continue on to create the first live snapshot and finish the wizard.

With direct or SSH access

If you have direct access to the server and can log into it, or if you can connect to it via SSH, you can run steps similar as above but using your access instead of the Jupyter terminal:

  1. Get a shell on the server using your SSH or direct access. Make sure you are connected as the user that is running REVEN.
  2. Continue on with the procedure above, using this shell instead of Jupyter's.

Moreover, if you can run a GUI application on the server (X-forwarding or direct graphical access), you can chose to pre-install the OS to the new disk image prior to importing it in REVEN. Below is a general list of steps you can take in this situation:

  1. Start your terminal session allowing for GUI application: either by logging into the server's desktop, or by starting an SSH session with X-forwarding (-X or -Y).
  2. Execute the command source /path/to/REVEN/install/sourceme to make sure you have access to the necessary binaries.
  3. Create your VM disk with qemu-img create
  4. Manually start the VM, with the iso inserted by running panda-system-x86_64 -m "2048M" -hda "/path/to/VMs/myimage.qcow2" -usbdevice tablet -enable-kvm -cdrom "/path/to/ISO". Again, adapt parameters as needed.
  5. The VM should show up on your screen and start the installer from the ISO file. Continue on with the installation.
  6. Shutdown the VM properly.
  7. You can now run the Project Manager's Wizard to import your newly created VM disk.

Working with QEMU snapshots in REVEN

IMPORTANT NOTE: QEMU replayers do NOT support VMs with strictly more than 3072MB of RAM. DO NOT try to record with more than 3072MB of RAM or the replay will fail. The Project Manager web interface will prevent you from doing that.

In the Project Manager:

  • You register a QEMU VM and corresponding disk snapshots are automatically linked. Besides, both disk and live snapshots can be created.
    • Disk snapshots (usually generated with QEMU qemu-img) can be taken through the Take snapshot button on the VM list page. NOTE: Disks snapshots imply booting the VM, which can be quite long with QEMU without KVM (several minutes for a Windows 10 VM).
    • Live snapshots (usually generated with QEMU savevm) can be taken through the Manage button on the VM list page, then in the Running the VM section. You can also access them on the record page of a scenario.

QEMU snapshots options

By default, when you start a snapshot, it is launched with the VM options (RAM size, network, custom QEMU options) that were provided during the Register VM step. You can override these options for this specific snapshot in the Running the VM section. Overridden options for a snapshot will be applied when starting the VM on this snapshot. You can restore an option to its VM value by unchecking the checkbox associated to this option.

Typical QEMU workflow example

To illustrate the previous explanations, here is how users can typically work with QEMU to configure and prepare a VM before recording a trace:

In the Project Manager VM Manager

  1. Register an existing QEMU VM.
  2. Create a disk snapshot for a new project from a clean parent.
  3. Boot this snapshot with KVM enabled.
  4. Install required software (using the CD-Rom mounting feature to upload files to the VM).
  5. Properly shutdown the VM (on Windows, using Shift+Click on the Shutdown option is required, otherwise the VM is only hibernated!).
  6. You can now Prepare the snapshot: all required binaries are present on the disk.
  7. Boot the VM again in emulated mode, i.e. with KVM disabled, with the required options for recording.
  8. Run operations required before the scenario recording but that need not be recorded.
  9. Take a live snapshot.

WARNING: live snapshots taken with KVM enabled can not be used for recording in REVEN Project Manager with QEMU.

Then, in the Scenario Manager

  1. Create a new scenario, selecting the previously created disk snapshot.
  2. Load the previously created live snapshot.
  3. Record your trace.
  4. Force shutdown the VM.

NOTE: At this point, the disk snapshot contains an OS that didn't properly shutdown: it is usually not an issue because restoring the live snapshot will overwrite this state, but booting the VM from the disk snapshot itself will likely trigger any disk verification process the guest OS may have.

NOTE: You can save live snapshot during scenario creation as well, if necessary.

NOTE: For simpler situations, you might have a few live snapshots in emulation mode for various use cases: one with network, one without, etc.

Help! My snapshot doesn't load!

There are a few situations that will prevent a snapshot from loading. In all cases, you can go to the list of Sessions in the Project Manager to get the log of what went wrong. Several checks can be done, depending on the type of snapshot concerned.

Live snapshots

  • Make sure the selected options match that of the live snapshot, including kvm mode and custom options. As a convenience, the snapshots's name contains a summary of common ones.

Disk snapshots

  • Has the VM been properly shutdown? (Shift + Click on Shutdown in windows)
  • Have the parent disk snapshot been modified? If so, children snapshots are unusable

Note that in some cases your disk snapshot may become corrupted leading to the error Image is corrupt; cannot be opened read/write when launching QEMU. It can sometimes occur when having heavy disk I/O or killing QEMU.

To assert the level of corruption of your snapshot you can use the command qemu-img check /path/to/your/snapshot.qcow2.
A possible fix is to ask qemu-img to fix the corruption qemu-img check -r all /path/to/your/snapshot.qcow2.

©2019-2020 Tetrane