Quick & Dirty: NAT routing performance in virtualised environment

tl;dr: VyOS NAT throughput is >3 times that of pfSense in virtualised environment.

All tests are performed inside a box spotting an ASRock N3150-ITX board with 16GB DDR3 1600MHz RAM running Ubuntu 16.10 (with all updates applied as of 2017-03-18; libvirt 2.1.0-1, qemu-kvm 2.6.1+dfsg-0). Tests are carried out between a pair of Ubuntu Server 16.04 with a NAT router in the middle.

Each VM (Ubuntu Servers, NAT routers) is allocated 1GB of RAM and 1 of the N3150 core (unless otherwise specified); virtio_net is used for all network interfaces.

All of these are “out of the box" performance (no special tuning applied); except for pfSense and FreeBSD to work, Checksum Offloading has been disabled for vtnet driver. (Seems that you may also disable checksum offloading from libvirt on the host)

# cat /boot/loader.conf
hw.vtnet.csum_disable=1
# There's a web UI option if you are using pfSense

iperf3 server and client are running Ubuntu 16.04 with multiqueue enabled.
I tried enabling multiqueue support also for pfSense, but that causes kernel panic during startup so I have to disable it. And I kept it disabled for VyOS for fair comparison.

    <interface type='network'>
      <mac address='52:54:00:12:db:0b'/>
      <source network='pfLAN'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

繼續閱讀

tuned-adm profile virtual-host

# tuned-adm verify
2017-03-14 15:02:15,555 INFO tuned.daemon.daemon: verifying profile(s): virtual-host
2017-03-14 15:02:15,558 INFO tuned.plugins.base: verify: passed: 'kernel.sched_migration_cost_ns' = '5000000'
2017-03-14 15:02:15,561 INFO tuned.plugins.base: verify: passed: 'vm.dirty_ratio' = '40'
2017-03-14 15:02:15,563 INFO tuned.plugins.base: verify: passed: 'vm.dirty_background_ratio' = '5'
2017-03-14 15:02:15,565 INFO tuned.plugins.base: verify: passed: 'kernel.sched_wakeup_granularity_ns' = '15000000'
2017-03-14 15:02:15,567 INFO tuned.plugins.base: verify: passed: 'vm.swappiness' = '10'
2017-03-14 15:02:15,569 INFO tuned.plugins.base: verify: passed: 'kernel.sched_min_granularity_ns' = '10000000'

For copy&paste:

# cat /etc/sysctl.d/99-tuned-virtual-host.conf
kernel.sched_migration_cost_ns = 5000000
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000
vm.dirty_background_ratio = 5
vm.dirty_ratio = 40
vm.swappiness = 10

Failed to parse /etc/cgconfig.conf or /etc/cgconfig.d

Some of the configuration options in cgconfig.conf or cgconfig.d/* might be invalid (eg: out of range)

Try manually creating your desire hierarchy (through cgroupfs) to see which option might be causing problem.

Then you may use cgsnapshot to save a copy of cgconfig for reference.

In my case I tried setting blkio.weight = 10; it was a valid value as documented but the actual  minimal value accepted by cgroupfs was 100.

pfSense not passing TCP traffic from NAT subnets in virtualised environment

Configuration:

  1. pfSense (version 2.3.3 as of 2017-03-12)
  2. NAT clients and pfSense running on qemu-kvm on the same host using virtio networking interface

Symptoms:

  1. Can resolve DNS (UDP/53)
  2. Can ping WAN hosts (ICMP)
  3. Cannot SSH / HTTP to WAN hosts (TCP)
  4. tcpdump on WAN hosts see TCP packets but cannot establish connections

Work around:

  • Disabled Checksum offloading on pfSense
    • Reason: Checksum is not calculated for virtio network devices until it leaves via a physical NIC

References: