Quick & Dirty: NAT routing performance in virtualised environment

tl;dr: VyOS NAT throughput is >3 times that of pfSense in virtualised environment.

All tests are performed inside a box spotting an ASRock N3150-ITX board with 16GB DDR3 1600MHz RAM running Ubuntu 16.10 (with all updates applied as of 2017-03-18; libvirt 2.1.0-1, qemu-kvm 2.6.1+dfsg-0). Tests are carried out between a pair of Ubuntu Server 16.04 with a NAT router in the middle.

Each VM (Ubuntu Servers, NAT routers) is allocated 1GB of RAM and 1 of the N3150 core (unless otherwise specified); virtio_net is used for all network interfaces.

All of these are “out of the box" performance (no special tuning applied); except for pfSense and FreeBSD to work, Checksum Offloading has been disabled for vtnet driver. (Seems that you may also disable checksum offloading from libvirt on the host)

# cat /boot/loader.conf
# There's a web UI option if you are using pfSense

iperf3 server and client are running Ubuntu 16.04 with multiqueue enabled.
I tried enabling multiqueue support also for pfSense, but that causes kernel panic during startup so I have to disable it. And I kept it disabled for VyOS for fair comparison.

    <interface type='network'>
      <mac address='52:54:00:12:db:0b'/>
      <source network='pfLAN'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>


tuned-adm profile virtual-host

# tuned-adm verify
2017-03-14 15:02:15,555 INFO tuned.daemon.daemon: verifying profile(s): virtual-host
2017-03-14 15:02:15,558 INFO tuned.plugins.base: verify: passed: 'kernel.sched_migration_cost_ns' = '5000000'
2017-03-14 15:02:15,561 INFO tuned.plugins.base: verify: passed: 'vm.dirty_ratio' = '40'
2017-03-14 15:02:15,563 INFO tuned.plugins.base: verify: passed: 'vm.dirty_background_ratio' = '5'
2017-03-14 15:02:15,565 INFO tuned.plugins.base: verify: passed: 'kernel.sched_wakeup_granularity_ns' = '15000000'
2017-03-14 15:02:15,567 INFO tuned.plugins.base: verify: passed: 'vm.swappiness' = '10'
2017-03-14 15:02:15,569 INFO tuned.plugins.base: verify: passed: 'kernel.sched_min_granularity_ns' = '10000000'

For copy&paste:

# cat /etc/sysctl.d/99-tuned-virtual-host.conf
kernel.sched_migration_cost_ns = 5000000
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000
vm.dirty_background_ratio = 5
vm.dirty_ratio = 40
vm.swappiness = 10

Failed to parse /etc/cgconfig.conf or /etc/cgconfig.d

Some of the configuration options in cgconfig.conf or cgconfig.d/* might be invalid (eg: out of range)

Try manually creating your desire hierarchy (through cgroupfs) to see which option might be causing problem.

Then you may use cgsnapshot to save a copy of cgconfig for reference.

In my case I tried setting blkio.weight = 10; it was a valid value as documented but the actual  minimal value accepted by cgroupfs was 100.

pfSense not passing TCP traffic from NAT subnets in virtualised environment


  1. pfSense (version 2.3.3 as of 2017-03-12)
  2. NAT clients and pfSense running on qemu-kvm on the same host using virtio networking interface


  1. Can resolve DNS (UDP/53)
  2. Can ping WAN hosts (ICMP)
  3. Cannot SSH / HTTP to WAN hosts (TCP)
  4. tcpdump on WAN hosts see TCP packets but cannot establish connections

Work around:

  • Disabled Checksum offloading on pfSense
    • Reason: Checksum is not calculated for virtio network devices until it leaves via a physical NIC