Troubleshooting Xen Virtual Machine Network Misuse and Over-Use on the Hyper-Visor


Reliable Hardware – Trained Staff
You are here:  Service / Technical Blog
By    -   Head of Technical Projects

If you or your company provide virtual servers within a Xen Virtualisation Environment, then it’s probably safe to say that you’ve run into Network overuse or misuse in the past, on one or more of your Hyper-Visors. Troubleshooting this and finding the VM responsible can be a tricky one, as many control panels don’t report live virtual interface data. (And even if they did, you can’t connect to it during a large scale attack!).

We’ve compiled a few of the simplest, and most direct ways of pinpointing exactly which pesky VM is the cause. The only thing you need to have installed? Sysstat.

Network Misuse or Overuse (Inbound or Outbound Attacks)

If your network graphs alert you to network spikes, or suspicious activity, such as either bursting or sustained high PPS (packets per second) then you could have an attack on your hands. With budget VM’s being so cheap and attainable, and instant deployment pretty much the norm, it makes sense for malicious 3rd parties to use them as staging platforms to participate in traditional traffic based DDoS and other common reflection based attacks.

If the attack is large enough, you will struggle to connect to your Hyper-Visor over the network. So physical access may be required for this one.

The following command will give you a solid overview of the network use, per interface. This includes the virtual interfaces bound to your VM’s:

sar –n DEV 1 3

Explained:

This command uses sar. Sar is a handy tool that collates and displays various pieces of data from system activity counters, and can also be used to display in more useful ways, the contents of binary data files containing system performance history

  • -n – Reports the network statistics
  • DEV – Targets specifically the network devices
  • 1 – Interview between re-polling sar
  • 3 – Number of times to poll sar before averaging the results
Running the command should garner you something along the lines of this:

Troubleshooting Xen Virtual Machine Network

The above is largely normal, if you excuse the odd marginally high traffic level. The first 4 columns are what should be of interest to you, receive and transmit packets per second, and receive and transmit kB/s. If a VM is attacking, or being attacked, these values will usually all be in the 100,000’s. It will become hard to read the specific values, as the columns merge together.

The virtual interfaces are nicely named with the VM ID included. So this immediately tells you the unfortunate target or the unscrupulous attacker. However, you still don’t know the IP Address. And with the attacks ongoing, you still can’t log in to the friendly web GUI to suspend the VM.

The following command can help. There may also be times when you simply don’t want to shut the VM down, but you do want to stop the attacks at network level Lets assume you want the IP of vm1686.

find / -name vm1686.cfg -exec grep “vif” {} \;

Explained:

This uses a typical find command, but is combined with the –exec switch for added functionality.

  • / – Start search in the root
  • -name – Search by full file name
  • vmXXX.cfg – Substitute the VM ID into here
  • -exec grep “vif” {} /; – This executes a simple grep command on every result find, and places the filename of the found result after the grep parameter.
Tip: You could even go further with this and awk it to cut down on the un-needed information |awk ‘{ print $3 }’

The output of the above should give you something that looks like this:

Troubleshooting Xen Virtual Machine Network

From there, you can block/blackhole/nullroute the IP as you please, without having to shutdown the VM, and without ever needing to access your Hyper-Visors web GUI.


Authors

Categories

Recent Posts

Review Us

Are you a VooServers customer? We would love to hear what you say!

Review us
Back to top

© VooServers Ltd 2016 - All Rights Reserved
Company No. 05598165