path: root/docs/report/vpp_performance_tests/test_environment.rst

.. _test_environment:

Test Environment
================

CSIT performance tests are executed on the three identical physical testbeds
hosted by :abbr:`LF (Linux Foundation)` for FD.io project. Each testbed
consists of two servers acting as Systems Under Test (SUT) and one server
acting as Traffic Generator (TG).

Server Specification and Configuration
--------------------------------------

Complete specification and configuration of compute servers used in CSIT
physical testbeds is maintained on wiki page `CSIT LF Testbeds
<https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.

SUT Configuration
-----------------

**Host configuration**

All hosts are Cisco UCS C240-M4 (2x Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz,
18c, 512GB RAM)

::

    $ lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                36
    On-line CPU(s) list:   0-35
    Thread(s) per core:    1
    Core(s) per socket:    18
    Socket(s):             2
    NUMA node(s):          2
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 63
    Model name:            Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
    Stepping:              2
    CPU MHz:               2294.249
    BogoMIPS:              4589.82
    Virtualization:        VT-x
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              46080K
    NUMA node0 CPU(s):     0-17
    NUMA node1 CPU(s):     18-35
    Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts

**BIOS settings**

::

    C240 /bios # show advanced detail
    Set-up parameters:
        Intel(R) VT-d ATS Support: Enabled
        Adjacent Cache Line Prefetcher: Enabled
        All Onboard LOM Ports: Enabled
        Altitude: 300 M
        Bits per second: 115200
        Power Technology: Performance
        Channel Interleaving: Auto
        Intel(R) VT-d Coherency Support: Disabled
        Console Redirection: COM 0
        Number of Enabled Cores: All
        Energy Performance: Performance
        CPU Performance: Enterprise
        DCU IP Prefetcher: Enabled
        DCU Streamer Prefetch: Enabled
        Demand Scrub: Enabled
        Direct Cache Access Support: Auto
        Enhanced Intel Speedstep(R) Tec: Disabled
        Execute Disable: Enabled
        Flow Control: None
        Hardware Prefetcher: Enabled
        Intel(R) Hyper-Threading Techno: Disabled
        Intel(R) Turbo Boost Technology: Disabled
        Intel(R) VT: Enabled
        Intel(R) VT-d: Enabled
        Intel(R) Interrupt Remapping: Enabled
        Legacy USB Support: Enabled
        Extended APIC: XAPIC
        LOM Port 1 OptionROM: Enabled
        LOM Port 2 OptionROM: Enabled
        MMIO above 4GB: Enabled
        NUMA: Enabled
        PCI ROM CLP: Disabled
        Package C State Limit: C6 Retention
        Intel(R) Pass Through DMA: Disabled
        Patrol Scrub: Enabled
        xHCI Mode: Disabled
        All PCIe Slots OptionROM: Enabled
        PCIe Slot:1 OptionROM: Disabled
        PCIe Slot:2 OptionROM: Disabled
        PCIe Slot:3 OptionROM: Disabled
        PCIe Slot:4 OptionROM: Disabled
        PCIe Slot:5 OptionROM: Disabled
        PCIe Slot:6 OptionROM: Disabled
        PCIe Slot:HBA Link Speed: GEN3
        PCIe Slot:HBA OptionROM: Enabled
        PCIe Slot:MLOM OptionROM: Enabled
        PCIe Slot:N1 OptionROM: Enabled
        PCIe Slot:N2 OptionROM: Enabled
        Processor Power state C1 Enhanc: Disabled
        Processor C3 Report: Disabled
        Processor C6 Report: Disabled
        P-STATE Coordination: HW ALL
        Putty KeyPad: ESCN
        Energy Performance Tuning: BIOS
        QPI Link Frequency Select: Auto
        QPI Snoop Mode: Home Snoop
        Rank Interleaving: Auto
        Redirection After BIOS POST: Always Enable
        PCH SATA Mode: AHCI
        Select Memory RAS: Maximum Performance
        SR-IOV Support: Enabled
        Terminal Type: VT100
        Port 60/64 Emulation: Enabled
        Workload Configuration: Balanced
        CDN Support for VIC: Disabled
        Out-of-Band Management: Disabled

**NIC models and placement**

In addition to CIMC and Management, each TG has 4x Intel X710 10GB NIC
(=8 ports) and 2x Intel XL710 40GB NIC (=4 ports), whereas each SUT has:

- 1x Intel X520 NIC (10GB, 2 ports),
- 1x Cisco VIC 1385 (40GB, 2 ports),
- 1x Intel XL710 NIC (40GB, 2 ports),
- 1x Intel X710 NIC (10GB, 2 ports),
- 1x Cisco VIC 1227 (10GB, 2 ports).

This allows for a total of five ring topologies, each using ports on
specific NIC model, enabling per NIC model benchmarking.

- 0a:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
  Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server
  Adapter X520-2
- 0a:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
  Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server
  Adapter X520-2
- 06:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
  Subsystem: Cisco Systems Inc VIC 1227 PCIe Ethernet NIC
- 07:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
  Subsystem: Cisco Systems Inc VIC 1227 PCIe Ethernet NIC
- 13:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
  Subsystem: Cisco Systems Inc VIC 1385 PCIe Ethernet NIC
- 15:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2)
  Subsystem: Cisco Systems Inc VIC 1385 PCIe Ethernet NIC
- 85:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710
  for 40GbE QSFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged
  Network Adapter XL710-Q2
- 85:00.1 Ethernet controller: Intel Corporation Ethernet Controller XL710
  for 40GbE QSFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged
  Network Adapter XL710-Q2
- 87:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for
  10GbE SFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged Network
  Adapter X710-2
- 87:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for
  10GbE SFP+ (rev 01) Subsystem: Intel Corporation Ethernet Converged Network
  Adapter X710-2

SUT Configuration - Host OS Linux
---------------------------------

Software details (OS, configuration) of physical testbeds are maintained
on wiki page
`CSIT LF Testbeds <https://wiki.fd.io/view/CSIT/CSIT_LF_testbed>`_.

System provisioning is done by combination of PXE boot unattented
install and
`Ansible <https://www.ansible.com>`_ described in `CSIT Testbed Setup`_.

Below a subset of the running configuration:

::

    $ lsb_release -a
    No LSB modules are available.
    Distributor ID:	Ubuntu
    Description:	Ubuntu 16.04.1 LTS
    Release:	16.04
    Codename:	xenial

::

    $ cat /sys/devices/system/node/node*/meminfo
    Node 0 MemTotal:       264048168 kB
    Node 0 MemFree:        257730716 kB
    Node 0 MemUsed:         6317452 kB
    Node 0 Active:          1079920 kB
    Node 0 Inactive:         470064 kB
    Node 0 Active(anon):     674772 kB
    Node 0 Inactive(anon):   248572 kB
    Node 0 Active(file):     405148 kB
    Node 0 Inactive(file):   221492 kB
    Node 0 Unevictable:           0 kB
    Node 0 Mlocked:               0 kB
    Node 0 Dirty:                12 kB
    Node 0 Writeback:             0 kB
    Node 0 FilePages:       1270432 kB
    Node 0 Mapped:            20116 kB
    Node 0 AnonPages:        279548 kB
    Node 0 Shmem:            643796 kB
    Node 0 KernelStack:        3376 kB
    Node 0 PageTables:         1316 kB
    Node 0 NFS_Unstable:          0 kB
    Node 0 Bounce:                0 kB
    Node 0 WritebackTmp:          0 kB
    Node 0 Slab:              80428 kB
    Node 0 SReclaimable:      38288 kB
    Node 0 SUnreclaim:        42140 kB
    Node 0 AnonHugePages:    270336 kB
    Node 0 HugePages_Total:  2048
    Node 0 HugePages_Free:   2048
    Node 0 HugePages_Surp:      0
    Node 1 MemTotal:       264237596 kB
    Node 1 MemFree:        256758976 kB
    Node 1 MemUsed:         7478620 kB
    Node 1 Active:          1746052 kB
    Node 1 Inactive:         981104 kB
    Node 1 Active(anon):    1272936 kB
    Node 1 Inactive(anon):   849968 kB
    Node 1 Active(file):     473116 kB
    Node 1 Inactive(file):   131136 kB
    Node 1 Unevictable:           0 kB
    Node 1 Mlocked:               0 kB
    Node 1 Dirty:                 0 kB
    Node 1 Writeback:             0 kB
    Node 1 FilePages:       2715284 kB
    Node 1 Mapped:            75928 kB
    Node 1 AnonPages:         11920 kB
    Node 1 Shmem:           2111036 kB
    Node 1 KernelStack:        2576 kB
    Node 1 PageTables:         1348 kB
    Node 1 NFS_Unstable:          0 kB
    Node 1 Bounce:                0 kB
    Node 1 WritebackTmp:          0 kB
    Node 1 Slab:              90604 kB
    Node 1 SReclaimable:      55384 kB
    Node 1 SUnreclaim:        35220 kB
    Node 1 AnonHugePages:      6144 kB
    Node 1 HugePages_Total:  2048
    Node 1 HugePages_Free:   2048
    Node 1 HugePages_Surp:      0

**Kernel boot parameters used in CSIT performance testbeds**

- **isolcpus=<cpu number>-<cpu number>** used for all cpu cores apart from
  first core of each socket used for running VPP worker threads and Qemu/LXC
  processes
  https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt
- **intel_pstate=disable** - [X86] Do not enable intel_pstate as the default
  scaling driver for the supported processors. Intel P-State driver decide what
  P-state (CPU core power state) to use based on requesting policy from the
  cpufreq core. [X86 - Either 32-bit or 64-bit x86]
  https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt
- **nohz_full=<cpu number>-<cpu number>** - [KNL,BOOT] In kernels built with
  CONFIG_NO_HZ_FULL=y, set the specified list of CPUs whose tick will be stopped
  whenever possible. The boot CPU will be forced outside the range to maintain
  the timekeeping. The CPUs in this range must also be included in the
  rcu_nocbs= set. Specifies the adaptive-ticks CPU cores, causing kernel to
  avoid sending scheduling-clock interrupts to listed cores as long as they have
  a single runnable task. [KNL - Is a kernel start-up parameter, SMP - The
  kernel is an SMP kernel].
  https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt
- **rcu_nocbs** - [KNL] In kernels built with CONFIG_RCU_NOCB_CPU=y, set the
  specified list of CPUs to be no-callback CPUs, that never queue RCU callbacks
  (read-copy update).
  https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt

**Applied command line boot parameters:**

::

    $ cat /proc/cmdline
    BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=UUID=35ea11e4-e44f-4f67-8cbe-12f09c49ed90 ro isolcpus=1-17,19-35 nohz_full=1-17,19-35 rcu_nocbs=1-17,19-35 intel_pstate=disable console=tty0 console=ttyS0,115200n8

**Mount listing**

::

    $ cat /proc/mounts
    sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
    proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
    udev /dev devtmpfs rw,nosuid,relatime,size=264125468k,nr_inodes=66031367,mode=755 0 0
    devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
    tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=52828580k,mode=755 0 0
    /dev/sda2 / ext4 rw,relatime,errors=remount-ro,data=ordered 0 0
    securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
    tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
    tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
    tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
    cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
    pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
    cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
    cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
    cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
    cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
    cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
    cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
    cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
    cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset,clone_children 0 0
    cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
    cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
    systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
    hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
    debugfs /sys/kernel/debug debugfs rw,relatime 0 0
    mqueue /dev/mqueue mqueue rw,relatime 0 0
    tracefs /sys/kernel/debug/tracing tracefs rw,relatime 0 0
    fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
    /dev/sda1 /boot ext4 rw,relatime,data=ordered 0 0
    none /mnt/huge hugetlbfs rw,relatime,pagesize=2048k 0 0
    lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

**Package listing**

::

    $ dpkg -l | grep '^ii' | awk '{print $2 ": " $3}'
    accountsservice: 0.6.40-2ubuntu11.1
    acl: 2.2.52-3
    adduser: 3.113+nmu3ubuntu4
    apparmor: 2.10.95-0ubuntu2.6
    apt: 1.2.12~ubuntu16.04.1
    apt-transport-https: 1.2.24
    apt-utils: 1.2.12~ubuntu16.04.1
    aufs-tools: 1:3.2+20130722-1.1ubuntu1
    autoconf: 2.69-9
    automake: 1:1.15-4ubuntu1
    autotools-dev: 20150820.1
    base-files: 9.4ubuntu4.2
    base-passwd: 3.5.39
    bash: 4.3-14ubuntu1.1
    binutils: 2.26.1-1ubuntu1~16.04.3
    bridge-utils: 1.5-9ubuntu1
    bsdutils: 1:2.27.1-6ubuntu3.1
    build-essential: 12.1ubuntu2
    busybox-initramfs: 1:1.22.0-15ubuntu1
    busybox-static: 1:1.22.0-15ubuntu1
    bzip2: 1.0.6-8
    busybox-static: 1:1.22.0-15ubuntu1
    bzip2: 1.0.6-8
    ca-certificates: 20160104ubuntu1
    ca-certificates-java: 20160321
    cgroup-bin: 0.41-7ubuntu1
    cgroup-lite: 1.11
    cgroup-tools: 0.41-7ubuntu1
    cloud-image-utils: 0.27-0ubuntu24
    console-setup: 1.108ubuntu15.2
    console-setup-linux: 1.108ubuntu15.2
    corekeeper: 1.6
    coreutils: 8.25-2ubuntu2
    cpio: 2.11+dfsg-5ubuntu1
    cpp: 4:5.3.1-1ubuntu1
    cpp-5: 5.4.0-6ubuntu1~16.04.2
    cpu-checker: 0.7-0ubuntu7
    cpufrequtils: 008-1
    crda: 3.13-1
    cron: 3.0pl1-128ubuntu2
    crudini: 0.7-1
    dash: 0.5.8-2.1ubuntu2
    dbus: 1.10.6-1ubuntu3
    debconf: 1.5.58ubuntu1
    debconf-i18n: 1.5.58ubuntu1
    debianutils: 4.7
    debootstrap: 1.0.78+nmu1ubuntu1.3
    dh-python: 2.20151103ubuntu1.1
    diffutils: 1:3.3-3
    distro-info: 0.14build1
    distro-info-data: 0.28ubuntu0.1
    dkms: 2.2.0.3-2ubuntu11.2
    dmidecode: 3.0-2ubuntu0.1
    dns-root-data: 2015052300+h+1
    dnsmasq-base: 2.75-1ubuntu0.16.04.2
    docker-ce: 17.09.0~ce-0~ubuntu
    dpkg: 1.18.4ubuntu1.1
    dpkg-dev: 1.18.4ubuntu1.1
    e2fslibs:amd64: 1.42.13-1ubuntu1
    e2fsprogs: 1.42.13-1ubuntu1
    ebtables: 2.0.10.4-3.4ubuntu2
    eject: 2.1.5+deb1+cvs20081104-13.1
    ethtool: 1:4.5-1
    expect: 5.45-7
    fakeroot: 1.20.2-1ubuntu1
    file: 1:5.25-2ubuntu1
    findutils: 4.6.0+git+20160126-2
    fontconfig: 2.11.94-0ubuntu1.1
    fontconfig-config: 2.11.94-0ubuntu1.1
    fonts-dejavu-core: 2.35-1
    fonts-dejavu-extra: 2.35-1
    g++: 4:5.3.1-1ubuntu1
    g++-5: 5.4.0-6ubuntu1~16.04.2
    gcc: 4:5.3.1-1ubuntu1
    gcc-5: 5.4.0-6ubuntu1~16.04.2
    gcc-5-base:amd64: 5.4.0-6ubuntu1~16.04.2
    gcc-6-base:amd64: 6.0.1-0ubuntu1
    gdb: 7.11.1-0ubuntu1~16.5
    gdbserver: 7.11.1-0ubuntu1~16.5
    genisoimage: 9:1.1.11-3ubuntu1
    gettext-base: 0.19.7-2ubuntu3
    gir1.2-glib-2.0:amd64: 1.46.0-3ubuntu1
    git: 1:2.7.4-0ubuntu1
    git-man: 1:2.7.4-0ubuntu1
    gnupg: 1.4.20-1ubuntu3.1
    gpgv: 1.4.20-1ubuntu3.1
    grep: 2.25-1~16.04.1
    grub-common: 2.02~beta2-36ubuntu3.1
    grub-gfxpayload-lists: 0.7
    grub-pc: 2.02~beta2-36ubuntu3.1
    grub-pc-bin: 2.02~beta2-36ubuntu3.1
    grub2-common: 2.02~beta2-36ubuntu3.1
    gzip: 1.6-4ubuntu1
    hicolor-icon-theme: 0.15-0ubuntu1
    hostname: 3.16ubuntu2
    ifupdown: 0.8.10ubuntu1
    init: 1.29ubuntu2
    init-system-helpers: 1.29ubuntu2
    initramfs-tools: 0.122ubuntu8.1
    initramfs-tools-bin: 0.122ubuntu8.1
    initramfs-tools-core: 0.122ubuntu8.1
    initscripts: 2.88dsf-59.3ubuntu2
    insserv: 1.14.0-5ubuntu3
    installation-report: 2.60ubuntu1
    iproute2: 4.3.0-1ubuntu3
    iptables: 1.6.0-2ubuntu3
    iputils-ping: 3:20121221-5ubuntu2
    ipxe-qemu: 1.0.0+git-20150424.a25a16d-1ubuntu1
    isc-dhcp-client: 4.3.3-5ubuntu12.1
    isc-dhcp-common: 4.3.3-5ubuntu12.1
    iso-codes: 3.65-1
    iw: 3.17-1
    java-common: 0.56ubuntu2
    kbd: 1.15.5-1ubuntu4
    keyboard-configuration: 1.108ubuntu15.2
    klibc-utils: 2.0.4-8ubuntu1.16.04.1
    kmod: 22-1ubuntu4
    krb5-locales: 1.13.2+dfsg-5
    kubeadm: 1.8.4-00
    kubectl: 1.8.4-00
    kubelet: 1.8.4-00
    kubernetes-cni: 0.5.1-00
    language-selector-common: 0.165.3
    laptop-detect: 0.13.7ubuntu2
    less: 481-2.1
    libaccountsservice0:amd64: 0.6.40-2ubuntu11.1
    libacl1:amd64: 2.2.52-3
    libaio1:amd64: 0.3.110-2
    libalgorithm-diff-perl: 1.19.03-1
    libalgorithm-diff-xs-perl: 0.04-4build1
    libalgorithm-merge-perl: 0.08-3
    libapparmor-perl: 2.10.95-0ubuntu2.6
    libapparmor1:amd64: 2.10.95-0ubuntu2
    libapr1:amd64: 1.5.2-3
    libapt-inst2.0:amd64: 1.2.12~ubuntu16.04.1
    libapt-pkg5.0:amd64: 1.2.12~ubuntu16.04.1
    libasan2:amd64: 5.4.0-6ubuntu1~16.04.2
    libasn1-8-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libasound2:amd64: 1.1.0-0ubuntu1
    libasound2-data: 1.1.0-0ubuntu1
    libasprintf0v5:amd64: 0.19.7-2ubuntu3
    libasyncns0:amd64: 0.8-5build1
    libatk1.0-0:amd64: 2.18.0-1
    libatk1.0-data: 2.18.0-1
    libatm1:amd64: 1:2.5.1-1.5
    libatomic1:amd64: 5.4.0-6ubuntu1~16.04.2
    libattr1:amd64: 1:2.4.47-2
    libaudit-common: 1:2.4.5-1ubuntu2
    libaudit1:amd64: 1:2.4.5-1ubuntu2
    libavahi-client3:amd64: 0.6.32~rc+dfsg-1ubuntu2
    libavahi-common-data:amd64: 0.6.32~rc+dfsg-1ubuntu2
    libavahi-common3:amd64: 0.6.32~rc+dfsg-1ubuntu2
    libbabeltrace-ctf1:amd64: 1.3.2-1
    libbabeltrace1:amd64: 1.3.2-1
    libblkid1:amd64: 2.27.1-6ubuntu3.1
    libbluetooth3:amd64: 5.37-0ubuntu5
    libboost-iostreams1.58.0:amd64: 1.58.0+dfsg-5ubuntu3.1
    libboost-random1.58.0:amd64: 1.58.0+dfsg-5ubuntu3.1
    libboost-system1.58.0:amd64: 1.58.0+dfsg-5ubuntu3.1
    libboost-thread1.58.0:amd64: 1.58.0+dfsg-5ubuntu3.1
    libbrlapi0.6:amd64: 5.3.1-2ubuntu2.1
    libbsd0:amd64: 0.8.2-1
    libbz2-1.0:amd64: 1.0.6-8
    libc-bin: 2.23-0ubuntu3
    libc-dev-bin: 2.23-0ubuntu9
    libc6-dbg:amd64: 2.23-0ubuntu9
    libc6-dev:amd64: 2.23-0ubuntu9
    libcaca0:amd64: 0.99.beta19-2build2~gcc5.2
    libcacard0:amd64: 1:2.5.0-2
    libcairo2:amd64: 1.14.6-1
    libcap-ng0:amd64: 0.7.7-1
    libcap2:amd64: 1:2.24-12
    libcap2-bin: 1:2.24-12
    libcc1-0:amd64: 5.4.0-6ubuntu1~16.04.2
    libcgroup1:amd64: 0.41-7ubuntu1
    libcilkrts5:amd64: 5.4.0-6ubuntu1~16.04.2
    libcomerr2:amd64: 1.42.13-1ubuntu1
    libcpufreq0: 008-1
    libcryptsetup4:amd64: 2:1.6.6-5ubuntu2
    libcups2:amd64: 2.1.3-4
    libcurl3-gnutls:amd64: 7.47.0-1ubuntu2.1
    libdatrie1:amd64: 0.2.10-2
    libdb5.3:amd64: 5.3.28-11
    libdbus-1-3:amd64: 1.10.6-1ubuntu3
    libdbus-glib-1-2:amd64: 0.106-1
    libdebconfclient0:amd64: 0.198ubuntu1
    libdevmapper1.02.1:amd64: 2:1.02.110-1ubuntu10
    libdns-export162: 1:9.10.3.dfsg.P4-8ubuntu1.1
    libdpkg-perl: 1.18.4ubuntu1.1
    libdrm-amdgpu1:amd64: 2.4.67-1ubuntu0.16.04.2
    libdrm-intel1:amd64: 2.4.67-1ubuntu0.16.04.2
    libdrm-nouveau2:amd64: 2.4.67-1ubuntu0.16.04.2
    libdrm-radeon1:amd64: 2.4.67-1ubuntu0.16.04.2
    libdrm2:amd64: 2.4.67-1ubuntu0.16.04.2
    libedit2:amd64: 3.1-20150325-1ubuntu2
    libelf1:amd64: 0.165-3ubuntu1
    liberror-perl: 0.17-1.2
    libestr0: 0.1.10-1
    libexpat1:amd64: 2.1.0-7ubuntu0.16.04.2
    libexpat1-dev:amd64: 2.1.0-7ubuntu0.16.04.2
    libfakeroot:amd64: 1.20.2-1ubuntu1
    libfdisk1:amd64: 2.27.1-6ubuntu3.1
    libfdt1:amd64: 1.4.0+dfsg-2
    libffi6:amd64: 3.2.1-4
    libfile-fcntllock-perl: 0.22-3
    libflac8:amd64: 1.3.1-4
    libfontconfig1:amd64: 2.11.94-0ubuntu1.1
    libfontenc1:amd64: 1:1.1.3-1
    libfreetype6:amd64: 2.6.1-0.1ubuntu2
    libfribidi0:amd64: 0.19.7-1
    libfuse2:amd64: 2.9.4-1ubuntu3
    libgcc-5-dev:amd64: 5.4.0-6ubuntu1~16.04.2
    libgcc1:amd64: 1:6.0.1-0ubuntu1
    libgcrypt20:amd64: 1.6.5-2ubuntu0.2
    libgdbm3:amd64: 1.8.3-13.1
    libgdk-pixbuf2.0-0:amd64: 2.32.2-1ubuntu1.2
    libgdk-pixbuf2.0-common: 2.32.2-1ubuntu1.2
    libgif7:amd64: 5.1.4-0.3~16.04
    libgirepository-1.0-1:amd64: 1.46.0-3ubuntu1
    libgl1-mesa-dri:amd64: 11.2.0-1ubuntu2.2
    libgl1-mesa-glx:amd64: 11.2.0-1ubuntu2.2
    libglapi-mesa:amd64: 11.2.0-1ubuntu2.2
    libglib2.0-0:amd64: 2.48.1-1~ubuntu16.04.1
    libglib2.0-bin: 2.48.1-1~ubuntu16.04.1
    libglib2.0-data: 2.48.1-1~ubuntu16.04.1
    libglib2.0-dev: 2.48.1-1~ubuntu16.04.1
    libgmp10:amd64: 2:6.1.0+dfsg-2
    libgnutls-openssl27:amd64: 3.4.10-4ubuntu1.1
    libgnutls30:amd64: 3.4.10-4ubuntu1.1
    libgomp1:amd64: 5.4.0-6ubuntu1~16.04.2
    libgpg-error0:amd64: 1.21-2ubuntu1
    libgraphite2-3:amd64: 1.3.6-1ubuntu1
    libgssapi-krb5-2:amd64: 1.13.2+dfsg-5
    libgssapi3-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libgtk2.0-0:amd64: 2.24.30-1ubuntu1.16.04.2
    libgtk2.0-bin: 2.24.30-1ubuntu1.16.04.2
    libgtk2.0-common: 2.24.30-1ubuntu1.16.04.2
    libharfbuzz0b:amd64: 1.0.1-1ubuntu0.1
    libhcrypto4-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libheimbase1-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libheimntlm0-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libhogweed4:amd64: 3.2-1
    libhx509-5-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libice-dev:amd64: 2:1.0.9-1
    libice6:amd64: 2:1.0.9-1
    libicu55:amd64: 55.1-7
    libidn11:amd64: 1.32-3ubuntu1.1
    libisc-export160: 1:9.10.3.dfsg.P4-8ubuntu1.1
    libiscsi2:amd64: 1.12.0-2
    libisl15:amd64: 0.16.1-1
    libitm1:amd64: 5.4.0-6ubuntu1~16.04.2
    libjbig0:amd64: 2.1-3.1
    libjpeg-turbo8:amd64: 1.4.2-0ubuntu3
    libjpeg8:amd64: 8c-2ubuntu8
    libjson-c2:amd64: 0.11-4ubuntu2
    libk5crypto3:amd64: 1.13.2+dfsg-5
    libkeyutils1:amd64: 1.5.9-8ubuntu1
    libklibc: 2.0.4-8ubuntu1.16.04.1
    libkmod2:amd64: 22-1ubuntu4
    libkrb5-26-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libkrb5-3:amd64: 1.13.2+dfsg-5
    libkrb5support0:amd64: 1.13.2+dfsg-5
    liblcms2-2:amd64: 2.6-3ubuntu2
    libldap-2.4-2:amd64: 2.4.42+dfsg-2ubuntu3.1
    libllvm3.8:amd64: 1:3.8-2ubuntu4
    liblocale-gettext-perl: 1.07-1build1
    liblsan0:amd64: 5.4.0-6ubuntu1~16.04.2
    libltdl-dev:amd64: 2.4.6-0.1
    libltdl7:amd64: 2.4.6-0.1
    liblxc1: 2.0.7-0ubuntu1~16.04.2
    liblz4-1:amd64: 0.0~r131-2ubuntu2
    liblzma5:amd64: 5.1.1alpha+20120614-2ubuntu2
    libmagic1:amd64: 1:5.25-2ubuntu1
    libmnl0:amd64: 1.0.3-5
    libmount1:amd64: 2.27.1-6ubuntu3.1
    libmpc3:amd64: 1.0.3-1
    libmpdec2:amd64: 2.4.2-1
    libmpfr4:amd64: 3.1.4-1
    libmpx0:amd64: 5.4.0-6ubuntu1~16.04.2
    libncurses5:amd64: 6.0+20160213-1ubuntu1
    libncursesw5:amd64: 6.0+20160213-1ubuntu1
    libnetfilter-conntrack3:amd64: 1.0.5-1
    libnettle6:amd64: 3.2-1
    libnewt0.52:amd64: 0.52.18-1ubuntu2
    libnfnetlink0:amd64: 1.0.1-3
    libnih-dbus1:amd64: 1.0.3-4.3ubuntu1
    libnih1:amd64: 1.0.3-4.3ubuntu1
    libnl-3-200:amd64: 3.2.27-1
    libnl-genl-3-200:amd64: 3.2.27-1
    libnspr4:amd64: 2:4.12-0ubuntu0.16.04.1
    libnss3:amd64: 2:3.23-0ubuntu0.16.04.1
    libnss3-nssdb: 2:3.23-0ubuntu0.16.04.1
    libnuma-dev:amd64: 2.0.11-1ubuntu1
    libnuma1:amd64: 2.0.11-1ubuntu1
    libogg0:amd64: 1.3.2-1
    libopus0:amd64: 1.1.2-1ubuntu1
    libp11-kit0:amd64: 0.23.2-3
    libpam-cgfs: 2.0.6-0ubuntu1~16.04.1
    libpam-modules:amd64: 1.1.8-3.2ubuntu2
    libpam-modules-bin: 1.1.8-3.2ubuntu2
    libpam-runtime: 1.1.8-3.2ubuntu2
    libpam0g:amd64: 1.1.8-3.2ubuntu2
    libpango-1.0-0:amd64: 1.38.1-1
    libpangocairo-1.0-0:amd64: 1.38.1-1
    libpangoft2-1.0-0:amd64: 1.38.1-1
    libpcap-dev: 1.7.4-2
    libpcap0.8:amd64: 1.7.4-2
    libpcap0.8-dev: 1.7.4-2
    libpci3:amd64: 1:3.3.1-1.1ubuntu1
    libpciaccess0:amd64: 0.13.4-1
    libpcre16-3:amd64: 2:8.38-3.1
    libpcre3:amd64: 2:8.38-3.1
    libpcre3-dev:amd64: 2:8.38-3.1
    libpcre32-3:amd64: 2:8.38-3.1
    libpcrecpp0v5:amd64: 2:8.38-3.1
    libpcsclite1:amd64: 1.8.14-1ubuntu1.16.04.1
    libperl5.22:amd64: 5.22.1-9
    libpixman-1-0:amd64: 0.33.6-1
    libplymouth4:amd64: 0.9.2-3ubuntu13.1
    libpng12-0:amd64: 1.2.54-1ubuntu1
    libpolkit-gobject-1-0:amd64: 0.105-14.1
    libpopt0:amd64: 1.16-10
    libprocps4:amd64: 2:3.3.10-4ubuntu2
    libpthread-stubs0-dev:amd64: 0.3-4
    libpulse0:amd64: 1:8.0-0ubuntu3
    libpython-all-dev:amd64: 2.7.11-1
    libpython-dev:amd64: 2.7.11-1
    libpython-stdlib:amd64: 2.7.11-1
    libpython2.7:amd64: 2.7.12-1~16.04
    libpython2.7-dev:amd64: 2.7.12-1~16.04
    libpython2.7-minimal:amd64: 2.7.12-1~16.04
    libpython2.7-stdlib:amd64: 2.7.12-1~16.04
    libpython3-stdlib:amd64: 3.5.1-3
    libpython3.5:amd64: 3.5.2-2ubuntu0~16.04.1
    libpython3.5-minimal:amd64: 3.5.2-2ubuntu0~16.04.1
    libpython3.5-stdlib:amd64: 3.5.2-2ubuntu0~16.04.1
    libquadmath0:amd64: 5.4.0-6ubuntu1~16.04.2
    librados2: 10.2.2-0ubuntu0.16.04.2
    librbd1: 10.2.2-0ubuntu0.16.04.2
    libreadline6:amd64: 6.3-8ubuntu2
    libroken18-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    librtmp1:amd64: 2.4+20151223.gitfa8646d-1build1
    libsasl2-2:amd64: 2.1.26.dfsg1-14build1
    libsasl2-modules:amd64: 2.1.26.dfsg1-14build1
    libsasl2-modules-db:amd64: 2.1.26.dfsg1-14build1
    libsdl1.2debian:amd64: 1.2.15+dfsg1-3
    libseccomp2:amd64: 2.2.3-3ubuntu3
    libselinux1:amd64: 2.4-3build2
    libsemanage-common: 2.3-1build3
    libsemanage1:amd64: 2.3-1build3
    libsepol1:amd64: 2.4-2
    libsigsegv2:amd64: 2.10-4
    libslang2:amd64: 2.3.0-2ubuntu1
    libsm-dev:amd64: 2:1.2.2-1
    libsm6:amd64: 2:1.2.2-1
    libsmartcols1:amd64: 2.27.1-6ubuntu3.1
    libsndfile1:amd64: 1.0.25-10
    libspice-server1:amd64: 0.12.6-4ubuntu0.1
    libsqlite3-0:amd64: 3.11.0-1ubuntu1
    libss2:amd64: 1.42.13-1ubuntu1
    libssl1.0.0:amd64: 1.0.2g-1ubuntu4.5
    libstdc++-5-dev:amd64: 5.4.0-6ubuntu1~16.04.2
    libstdc++6:amd64: 5.4.0-6ubuntu1~16.04.2
    libsystemd0:amd64: 229-4ubuntu10
    libtasn1-6:amd64: 4.7-3ubuntu0.16.04.1
    libtcl8.6:amd64: 8.6.5+dfsg-2
    libtext-charwidth-perl: 0.04-7build5
    libtext-iconv-perl: 1.7-5build4
    libtext-wrapi18n-perl: 0.06-7.1
    libthai-data: 0.1.24-2
    libthai0:amd64: 0.1.24-2
    libtiff5:amd64: 4.0.6-1ubuntu0.2
    libtinfo5:amd64: 6.0+20160213-1ubuntu1
    libtk8.6:amd64: 8.6.5-1
    libtool: 2.4.6-0.1
    libtsan0:amd64: 5.4.0-6ubuntu1~16.04.2
    libtxc-dxtn-s2tc0:amd64: 0~git20131104-1.1
    libubsan0:amd64: 5.4.0-6ubuntu1~16.04.2
    libudev1:amd64: 229-4ubuntu10
    libusb-0.1-4:amd64: 2:0.1.12-28
    libusb-1.0-0:amd64: 2:1.0.20-1
    libusbredirparser1:amd64: 0.7.1-1
    libustr-1.0-1:amd64: 1.0.4-5
    libutempter0:amd64: 1.1.6-3
    libuuid1:amd64: 2.27.1-6ubuntu3.1
    libvorbis0a:amd64: 1.3.5-3
    libvorbisenc2:amd64: 1.3.5-3
    libwind0-heimdal:amd64: 1.7~git20150920+dfsg-4ubuntu1
    libwrap0:amd64: 7.6.q-25
    libx11-6:amd64: 2:1.6.3-1ubuntu2
    libx11-data: 2:1.6.3-1ubuntu2
    libx11-dev:amd64: 2:1.6.3-1ubuntu2
    libx11-doc: 2:1.6.3-1ubuntu2
    libx11-xcb1:amd64: 2:1.6.3-1ubuntu2
    libxau-dev:amd64: 1:1.0.8-1
    libxau6:amd64: 1:1.0.8-1
    libxaw7:amd64: 2:1.0.13-1
    libxcb-dri2-0:amd64: 1.11.1-1ubuntu1
    libxcb-dri3-0:amd64: 1.11.1-1ubuntu1
    libxcb-glx0:amd64: 1.11.1-1ubuntu1
    libxcb-present0:amd64: 1.11.1-1ubuntu1
    libxcb-render0:amd64: 1.11.1-1ubuntu1
    libxcb-shape0:amd64: 1.11.1-1ubuntu1
    libxcb-shm0:amd64: 1.11.1-1ubuntu1
    libxcb-sync1:amd64: 1.11.1-1ubuntu1
    libxcb1:amd64: 1.11.1-1ubuntu1
    libxcb1-dev:amd64: 1.11.1-1ubuntu1
    libxcomposite1:amd64: 1:0.4.4-1
    libxcursor1:amd64: 1:1.1.14-1
    libxdamage1:amd64: 1:1.1.4-2
    libxdmcp-dev:amd64: 1:1.1.2-1.1
    libxdmcp6:amd64: 1:1.1.2-1.1
    libxen-4.6:amd64: 4.6.0-1ubuntu4.1
    libxenstore3.0:amd64: 4.6.0-1ubuntu4.1
    libxext6:amd64: 2:1.3.3-1
    libxfixes3:amd64: 1:5.0.1-2
    libxft2:amd64: 2.3.2-1
    libxi6:amd64: 2:1.7.6-1
    libxinerama1:amd64: 2:1.1.3-1
    libxml2:amd64: 2.9.3+dfsg1-1ubuntu0.1
    libxmu6:amd64: 2:1.1.2-2
    libxmuu1:amd64: 2:1.1.2-2
    libxpm4:amd64: 1:3.5.11-1
    libxrandr2:amd64: 2:1.5.0-1
    libxrender1:amd64: 1:0.9.9-0ubuntu1
    libxshmfence1:amd64: 1.2-1
    libxss1:amd64: 1:1.2.2-1
    libxt-dev:amd64: 1:1.1.5-0ubuntu1
    libxt6:amd64: 1:1.1.5-0ubuntu1
    libxtables11:amd64: 1.6.0-2ubuntu3
    libxtst6:amd64: 2:1.2.2-1
    libxv1:amd64: 2:1.0.10-1
    libxxf86dga1:amd64: 2:1.1.4-1
    libxxf86vm1:amd64: 1:1.1.4-1
    libyajl2:amd64: 2.1.0-2
    linux-base: 4.0ubuntu1
    linux-firmware: 1.157.2
    linux-generic: 4.4.0.72.78
    linux-headers-4.4.0-72: 4.4.0-72.93
    linux-headers-4.4.0-72-generic: 4.4.0-72.93
    linux-headers-generic: 4.4.0.72.78
    linux-image-4.4.0-72-generic: 4.4.0-72.93
    linux-image-extra-4.4.0-72-generic: 4.4.0-72.93
    linux-image-generic: 4.4.0.72.78
    linux-libc-dev:amd64: 4.4.0-72.93
    locales: 2.23-0ubuntu3
    login: 1:4.2-3.1ubuntu5
    logrotate: 3.8.7-2ubuntu2
    lsb-base: 9.20160110ubuntu0.2
    lsb-release: 9.20160110ubuntu0.2
    lxc: 2.0.7-0ubuntu1~16.04.2
    lxc-common: 2.0.7-0ubuntu1~16.04.2
    lxc-templates: 2.0.7-0ubuntu1~16.04.2
    lxc1: 2.0.7-0ubuntu1~16.04.2
    lxcfs: 2.0.6-0ubuntu1~16.04.1
    m4: 1.4.17-5
    make: 4.1-6
    makedev: 2.3.1-93ubuntu1
    manpages: 4.04-2
    manpages-dev: 4.04-2
    mawk: 1.3.3-17ubuntu2
    mime-support: 3.59ubuntu1
    mount: 2.27.1-6ubuntu3.1
    mountall: 2.54ubuntu1
    msr-tools: 1.3-2
    multiarch-support: 2.23-0ubuntu3
    ncurses-base: 6.0+20160213-1ubuntu1
    ncurses-bin: 6.0+20160213-1ubuntu1
    ncurses-term: 6.0+20160213-1ubuntu1
    net-tools: 1.60-26ubuntu1
    netbase: 5.3
    netcat-openbsd: 1.105-7ubuntu1
    openjdk-8-jdk:amd64: 8u131-b11-2ubuntu1.16.04.3
    openjdk-8-jdk-headless:amd64: 8u131-b11-2ubuntu1.16.04.3
    openjdk-8-jre:amd64: 8u131-b11-2ubuntu1.16.04.3
    openjdk-8-jre-headless:amd64: 8u131-b11-2ubuntu1.16.04.3
    openssh-client: 1:7.2p2-4ubuntu2.1
    openssh-server: 1:7.2p2-4ubuntu2.1
    openssh-sftp-server: 1:7.2p2-4ubuntu2.1
    openssl: 1.0.2g-1ubuntu4.5
    os-prober: 1.70ubuntu3
    passwd: 1:4.2-3.1ubuntu5
    patch: 2.7.5-1
    pciutils: 1:3.3.1-1.1ubuntu1
    perl: 5.22.1-9
    perl-base: 5.22.1-9
    perl-modules-5.22: 5.22.1-9
    pkg-config: 0.29.1-0ubuntu1
    plymouth: 0.9.2-3ubuntu13.1
    plymouth-theme-ubuntu-text: 0.9.2-3ubuntu13.1
    procps: 2:3.3.10-4ubuntu2
    python: 2.7.11-1
    python-all: 2.7.11-1
    python-all-dev: 2.7.11-1
    python-apt: 1.1.0~beta1build1
    python-apt-common: 1.1.0~beta1build1
    python-dev: 2.7.11-1
    python-iniparse: 0.4-2.2
    python-minimal: 2.7.11-1
    python-pip: 8.1.1-2ubuntu0.2
    python-pip-whl: 8.1.1-2ubuntu0.2
    python-pkg-resources: 20.7.0-1
    python-setuptools: 20.7.0-1
    python-six: 1.10.0-3
    python-virtualenv: 15.0.1+ds-3
    python-wheel: 0.29.0-1
    python2.7: 2.7.12-1~16.04
    python2.7-dev: 2.7.12-1~16.04
    python2.7-minimal: 2.7.12-1~16.04
    python3: 3.5.1-3
    python3-apt: 1.1.0~beta1build1
    python3-chardet: 2.3.0-2
    python3-dbus: 1.2.0-3
    python3-gi: 3.20.0-0ubuntu1
    python3-lxc: 2.0.7-0ubuntu1~16.04.2
    python3-minimal: 3.5.1-3
    python3-pkg-resources: 20.7.0-1
    python3-requests: 2.9.1-3
    python3-six: 1.10.0-3
    python3-urllib3: 1.13.1-2ubuntu0.16.04.1
    python3-virtualenv: 15.0.1+ds-3
    python3.5: 3.5.2-2ubuntu0~16.04.1
    python3.5-minimal: 3.5.2-2ubuntu0~16.04.1
    qemu-block-extra:amd64: 1:2.5+dfsg-5ubuntu10.5
    qemu-system-common: 1:2.5+dfsg-5ubuntu10.5
    qemu-system-x86: 1:2.5+dfsg-5ubuntu10.5
    qemu-utils: 1:2.5+dfsg-5ubuntu10.5
    readline-common: 6.3-8ubuntu2
    rename: 0.20-4
    resolvconf: 1.78ubuntu2
    rsync: 3.1.1-3ubuntu1
    rsyslog: 8.16.0-1ubuntu3
    screen: 4.3.1-2build1
    seabios: 1.8.2-1ubuntu1
    sed: 4.2.2-7
    sensible-utils: 0.0.9
    sgml-base: 1.26+nmu4ubuntu1
    shared-mime-info: 1.5-2ubuntu0.1
    sharutils: 1:4.15.2-1
    socat: 1.7.3.1-1
    ssh-import-id: 5.5-0ubuntu1
    sudo: 1.8.16-0ubuntu1.1
    systemd: 229-4ubuntu10
    systemd-sysv: 229-4ubuntu10
    sysv-rc: 2.88dsf-59.3ubuntu2
    sysvinit-utils: 2.88dsf-59.3ubuntu2
    tar: 1.28-2.1
    tasksel: 3.34ubuntu3
    tasksel-data: 3.34ubuntu3
    tcl-expect:amd64: 5.45-7
    tcl8.6: 8.6.5+dfsg-2
    tcpd: 7.6.q-25
    telnet: 0.17-40
    tk8.6: 8.6.5-1
    tzdata: 2016f-0ubuntu0.16.04
    ubuntu-keyring: 2012.05.19
    ubuntu-minimal: 1.361
    ucf: 3.0036
    udev: 229-4ubuntu10
    uidmap: 1:4.2-3.1ubuntu5.3
    ureadahead: 0.100.0-19
    usbutils: 1:007-4
    util-linux: 2.27.1-6ubuntu3.1
    uuid-runtime: 2.27.1-6ubuntu3.2
    vim-common: 2:7.4.1689-3ubuntu1.1
    vim-tiny: 2:7.4.1689-3ubuntu1.1
    virtualenv: 15.0.1+ds-3
    wamerican: 7.1-1
    wget: 1.17.1-1ubuntu1.1
    whiptail: 0.52.18-1ubuntu2
    wireless-regdb: 2015.07.20-1ubuntu1
    x11-common: 1:7.7+13ubuntu3
    x11-utils: 7.7+3
    x11proto-core-dev: 7.0.31-1~ubuntu16.04.1
    x11proto-input-dev: 2.3.1-1
    x11proto-kb-dev: 1.0.7-0ubuntu1
    xauth: 1:1.0.9-1ubuntu2
    xbitmaps: 1.1.1-2
    xdg-user-dirs: 0.15-2ubuntu6
    xkb-data: 2.16-1ubuntu1
    xml-core: 0.13+nmu2
    xorg-sgml-doctools: 1:1.11-1
    xterm: 322-1ubuntu1
    xtrans-dev: 1.3.5-1
    xz-utils: 5.1.1alpha+20120614-2ubuntu2
    zlib1g:amd64: 1:1.2.8.dfsg-2ubuntu4
    zlib1g-dev:amd64: 1:1.2.8.dfsg-2ubuntu4

**Kernel module listing**

::

    $ lsmod | sort
    8250_fintek            16384  0
    ablk_helper            16384  1 aesni_intel
    acpi_pad               24576  0
    acpi_power_meter       20480  0
    aesni_intel           167936  0
    aes_x86_64             20480  1 aesni_intel
    ahci                   36864  0
    authenc                16384  1 intel_qat
    autofs4                40960  2
    bridge                126976  0
    coretemp               16384  0
    crc32_pclmul           16384  0
    crct10dif_pclmul       16384  0
    cryptd                 20480  3 ghash_clmulni_intel,aesni_intel,ablk_helper
    dca                    16384  2 igb,ixgbe
    edac_core              53248  1 sb_edac
    enclosure              16384  1 ses
    enic                   81920  0
    fjes                   28672  0
    fnic                  106496  0
    gf128mul               16384  1 lrw
    ghash_clmulni_intel    16384  0
    glue_helper            16384  1 aesni_intel
    hid                   118784  2 hid_generic,usbhid
    hid_generic            16384  0
    i2c_algo_bit           16384  1 igb
    i40e                  286720  0
    igb                   196608  0
    igb_uio                16384  0
    input_leds             16384  0
    intel_powerclamp       16384  0
    intel_qat             110592  2 qat_dh895xccvf,qat_dh895xcc
    intel_rapl             20480  0
    ip6_udp_tunnel         16384  1 vxlan
    ipmi_msghandler        49152  2 ipmi_ssif,ipmi_si
    ipmi_si                57344  0
    ipmi_ssif              24576  0
    iptable_filter         16384  1
    iptable_mangle         16384  1
    iptable_nat            16384  1
    ip_tables              24576  3 iptable_filter,iptable_mangle,iptable_nat
    ipt_MASQUERADE         16384  1
    irqbypass              16384  1 kvm
    ixgbe                 290816  0
    joydev                 20480  0
    kvm                   544768  1 kvm_intel
    kvm_intel             172032  0
    libahci                32768  1 ahci
    libfc                 114688  2 fnic,libfcoe
    libfcoe                65536  1 fnic
    llc                    16384  2 stp,bridge
    lpc_ich                24576  0
    lrw                    16384  1 aesni_intel
    mac_hid                16384  0
    mdio                   16384  1 ixgbe
    megaraid_sas          135168  3
    mei                    98304  1 mei_me
    mei_me                 36864  0
    Module                  Size  Used by
    nf_conntrack          106496  4 nf_nat,nf_nat_ipv4,nf_nat_masquerade_ipv4,nf_conntrack_ipv4
    nf_conntrack_ipv4      16384  1
    nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
    nf_nat                 24576  2 nf_nat_ipv4,nf_nat_masquerade_ipv4
    nf_nat_ipv4            16384  1 iptable_nat
    nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
    pps_core               20480  1 ptp
    ptp                    20480  3 igb,i40e,ixgbe
    qat_dh895xcc           20480  0
    qat_dh895xccvf         20480  0
    sb_edac                32768  0
    scsi_transport_fc      61440  2 fnic,libfc
    ses                    20480  0
    shpchp                 36864  0
    stp                    16384  1 bridge
    udp_tunnel             16384  1 vxlan
    uio                    20480  2 uio_pci_generic,igb_uio
    uio_pci_generic        16384  0
    usbhid                 49152  0
    veth                   16384  0
    vxlan                  49152  2 i40e,ixgbe
    wmi                    20480  0
    x86_pkg_temp_thermal    16384  0
    x_tables               36864  6 xt_CHECKSUM,ip_tables,xt_tcpudp,ipt_MASQUERADE,iptable_filter,iptable_mangle
    xt_CHECKSUM            16384  1
    xt_tcpudp              16384  5

**Sysctl listing**

::

    $ sysctl -a
    abi.vsyscall32 = 1
    debug.exception-trace = 1
    debug.kprobes-optimization = 1
    dev.cdrom.autoclose = 1
    dev.cdrom.autoeject = 0
    dev.cdrom.check_media = 0
    dev.cdrom.debug = 0
    dev.cdrom.info = CD-ROM information, Id: cdrom.c 3.20 2003/12/17
    dev.cdrom.info =
    dev.cdrom.info = drive name:
    dev.cdrom.info = drive speed:
    dev.cdrom.info = drive # of slots:
    dev.cdrom.info = Can close tray:
    dev.cdrom.info = Can open tray:
    dev.cdrom.info = Can lock tray:
    dev.cdrom.info = Can change speed:
    dev.cdrom.info = Can select disk:
    dev.cdrom.info = Can read multisession:
    dev.cdrom.info = Can read MCN:
    dev.cdrom.info = Reports media changed:
    dev.cdrom.info = Can play audio:
    dev.cdrom.info = Can write CD-R:
    dev.cdrom.info = Can write CD-RW:
    dev.cdrom.info = Can read DVD:
    dev.cdrom.info = Can write DVD-R:
    dev.cdrom.info = Can write DVD-RAM:
    dev.cdrom.info = Can read MRW:
    dev.cdrom.info = Can write MRW:
    dev.cdrom.info = Can write RAM:
    dev.cdrom.info =
    dev.cdrom.info =
    dev.cdrom.lock = 0
    dev.hpet.max-user-freq = 64
    dev.mac_hid.mouse_button2_keycode = 97
    dev.mac_hid.mouse_button3_keycode = 100
    dev.mac_hid.mouse_button_emulation = 0
    dev.raid.speed_limit_max = 200000
    dev.raid.speed_limit_min = 1000
    dev.scsi.logging_level = 0
    fs.aio-max-nr = 65536
    fs.aio-nr = 0
    fs.binfmt_misc.status = enabled
    fs.dentry-state = 69970	58326	45	0	0	0
    fs.dir-notify-enable = 1
    fs.epoll.max_user_watches = 108185784
    fs.file-max = 52706330
    fs.file-nr = 1224	0	52706330
    fs.inode-nr = 42965	369
    fs.inode-state = 42965	369	0	0	0	0	0
    fs.inotify.max_queued_events = 16384
    fs.inotify.max_user_instances = 128
    fs.inotify.max_user_watches = 8192
    fs.lease-break-time = 45
    fs.leases-enable = 1
    fs.mount-max = 100000
    fs.mqueue.msg_default = 10
    fs.mqueue.msg_max = 10
    fs.mqueue.msgsize_default = 8192
    fs.mqueue.msgsize_max = 8192
    fs.mqueue.queues_max = 256
    fs.nr_open = 1048576
    fs.overflowgid = 65534
    fs.overflowuid = 65534
    fs.pipe-max-size = 1048576
    fs.pipe-user-pages-hard = 0
    fs.pipe-user-pages-soft = 16384
    fs.protected_hardlinks = 1
    fs.protected_symlinks = 1
    fs.quota.allocated_dquots = 0
    fs.quota.cache_hits = 0
    fs.quota.drops = 0
    fs.quota.free_dquots = 0
    fs.quota.lookups = 0
    fs.quota.reads = 0
    fs.quota.syncs = 0
    fs.quota.writes = 0
    fs.suid_dumpable = 0
    kernel.acct = 4	2	30
    kernel.acpi_video_flags = 0
    kernel.auto_msgmni = 0
    kernel.bootloader_type = 114
    kernel.bootloader_version = 2
    kernel.cad_pid = 1
    kernel.cap_last_cap = 37
    kernel.compat-log = 1
    kernel.core_pattern = core
    kernel.core_pipe_limit = 0
    kernel.core_uses_pid = 0
    kernel.ctrl-alt-del = 0
    kernel.dmesg_restrict = 0
    kernel.domainname = (none)
    kernel.ftrace_dump_on_oops = 0
    kernel.ftrace_enabled = 1
    kernel.hardlockup_all_cpu_backtrace = 0
    kernel.hardlockup_panic = 0
    kernel.hostname = t2-sut1
    kernel.hotplug =
    kernel.hung_task_check_count = 4194304
    kernel.hung_task_panic = 0
    kernel.hung_task_timeout_secs = 120
    kernel.hung_task_warnings = 10
    kernel.io_delay_type = 1
    kernel.kexec_load_disabled = 0
    kernel.keys.gc_delay = 300
    kernel.keys.maxbytes = 20000
    kernel.keys.maxkeys = 200
    kernel.keys.persistent_keyring_expiry = 259200
    kernel.keys.root_maxbytes = 25000000
    kernel.keys.root_maxkeys = 1000000
    kernel.kptr_restrict = 1
    kernel.kstack_depth_to_print = 12
    kernel.max_lock_depth = 1024
    kernel.modprobe = /sbin/modprobe
    kernel.modules_disabled = 0
    kernel.moksbstate_disabled = 0
    kernel.msg_next_id = -1
    kernel.msgmax = 8192
    kernel.msgmnb = 16384
    kernel.msgmni = 32000
    kernel.ngroups_max = 65536
    kernel.nmi_watchdog = 1
    kernel.ns_last_pid = 11764
    kernel.numa_balancing = 1
    kernel.numa_balancing_scan_delay_ms = 1000
    kernel.numa_balancing_scan_period_max_ms = 60000
    kernel.numa_balancing_scan_period_min_ms = 1000
    kernel.numa_balancing_scan_size_mb = 256
    kernel.osrelease = 4.4.0-72-generic
    kernel.ostype = Linux
    kernel.overflowgid = 65534
    kernel.overflowuid = 65534
    kernel.panic = 0
    kernel.panic_on_io_nmi = 0
    kernel.panic_on_oops = 0
    kernel.panic_on_unrecovered_nmi = 0
    kernel.panic_on_warn = 0
    kernel.perf_cpu_time_max_percent = 25
    kernel.perf_event_max_sample_rate = 12500
    kernel.perf_event_mlock_kb = 516
    kernel.perf_event_paranoid = 1
    kernel.pid_max = 36864
    kernel.poweroff_cmd = /sbin/poweroff
    kernel.print-fatal-signals = 0
    kernel.printk = 4	4	1	7
    kernel.printk_delay = 0
    kernel.printk_ratelimit = 5
    kernel.printk_ratelimit_burst = 10
    kernel.pty.max = 4096
    kernel.pty.nr = 1
    kernel.pty.reserve = 1024
    kernel.random.boot_id = f683c836-6fc6-492a-a23b-62ab21895040
    kernel.random.entropy_avail = 200
    kernel.random.poolsize = 4096
    kernel.random.read_wakeup_threshold = 64
    kernel.random.urandom_min_reseed_secs = 60
    kernel.random.uuid = 144ff2ba-1bc7-4836-8fb7-6aaa0ab7e65f
    kernel.random.write_wakeup_threshold = 896
    kernel.randomize_va_space = 0
    kernel.real-root-dev = 0
    kernel.sched_autogroup_enabled = 1
    kernel.sched_cfs_bandwidth_slice_us = 5000
    kernel.sched_child_runs_first = 0
    kernel.sched_domain.cpu0.domain0.busy_factor = 32
    kernel.sched_domain.cpu0.domain0.busy_idx = 3
    kernel.sched_domain.cpu0.domain0.cache_nice_tries = 2
    kernel.sched_domain.cpu0.domain0.flags = 25647
    kernel.sched_domain.cpu0.domain0.forkexec_idx = 0
    kernel.sched_domain.cpu0.domain0.idle_idx = 2
    kernel.sched_domain.cpu0.domain0.imbalance_pct = 125
    kernel.sched_domain.cpu0.domain0.max_interval = 72
    kernel.sched_domain.cpu0.domain0.max_newidle_lb_cost = 1309
    kernel.sched_domain.cpu0.domain0.min_interval = 36
    kernel.sched_domain.cpu0.domain0.name = NUMA
    kernel.sched_domain.cpu0.domain0.newidle_idx = 0
    kernel.sched_domain.cpu0.domain0.wake_idx = 0
    kernel.sched_domain.cpu18.domain0.busy_factor = 32
    kernel.sched_domain.cpu18.domain0.busy_idx = 3
    kernel.sched_domain.cpu18.domain0.cache_nice_tries = 2
    kernel.sched_domain.cpu18.domain0.flags = 25647
    kernel.sched_domain.cpu18.domain0.forkexec_idx = 0
    kernel.sched_domain.cpu18.domain0.idle_idx = 2
    kernel.sched_domain.cpu18.domain0.imbalance_pct = 125
    kernel.sched_domain.cpu18.domain0.max_interval = 72
    kernel.sched_domain.cpu18.domain0.max_newidle_lb_cost = 2026
    kernel.sched_domain.cpu18.domain0.min_interval = 36
    kernel.sched_domain.cpu18.domain0.name = NUMA
    kernel.sched_domain.cpu18.domain0.newidle_idx = 0
    kernel.sched_domain.cpu18.domain0.wake_idx = 0
    kernel.sched_latency_ns = 24000000
    kernel.sched_migration_cost_ns = 500000
    kernel.sched_min_granularity_ns = 3000000
    kernel.sched_nr_migrate = 32
    kernel.sched_rr_timeslice_ms = 25
    kernel.sched_rt_period_us = 1000000
    kernel.sched_rt_runtime_us = 950000
    kernel.sched_shares_window_ns = 10000000
    kernel.sched_time_avg_ms = 1000
    kernel.sched_tunable_scaling = 1
    kernel.sched_wakeup_granularity_ns = 4000000
    kernel.secure_boot = 0
    kernel.sem = 32000	1024000000	500	32000
    kernel.sem_next_id = -1
    kernel.sg-big-buff = 32768
    kernel.shm_next_id = -1
    kernel.shm_rmid_forced = 0
    kernel.shmall = 18446744073692774399
    kernel.shmmax = 8589934592
    kernel.shmmni = 4096
    kernel.soft_watchdog = 1
    kernel.softlockup_all_cpu_backtrace = 0
    kernel.softlockup_panic = 0
    kernel.stack_tracer_enabled = 0
    kernel.sysctl_writes_strict = 0
    kernel.sysrq = 176
    kernel.tainted = 12288
    kernel.threads-max = 4126960
    kernel.timer_migration = 1
    kernel.traceoff_on_warning = 0
    kernel.tracepoint_printk = 0
    kernel.unknown_nmi_panic = 0
    kernel.unprivileged_bpf_disabled = 0
    kernel.unprivileged_userns_apparmor_policy = 1
    kernel.unprivileged_userns_clone = 1
    kernel.usermodehelper.bset = 4294967295	63
    kernel.usermodehelper.inheritable = 4294967295	63
    kernel.version = #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017
    kernel.watchdog = 1
    kernel.watchdog_cpumask = 0,18
    kernel.watchdog_thresh = 10
    kernel.yama.ptrace_scope = 1
    net.core.bpf_jit_enable = 0
    net.core.busy_poll = 0
    net.core.busy_read = 0
    net.core.default_qdisc = pfifo_fast
    net.core.dev_weight = 64
    net.core.flow_limit_cpu_bitmap = 0,00000000
    net.core.flow_limit_table_len = 4096
    net.core.max_skb_frags = 17
    net.core.message_burst = 10
    net.core.message_cost = 5
    net.core.netdev_budget = 300
    net.core.netdev_max_backlog = 1000
    net.core.netdev_rss_key = 29:61:61:e6:4e:d5:d0:a2:dc:81:6a:c8:44:1b:e2:8d:c8:6f:6a:2b:64:62:98:08:bb:63:48:8e:96:d1:6a:15:32:ca:da:8d:3c:0a:ee:a6:f8:59:be:63:33:47:e9:cf:d7:01:e3:18
    net.core.netdev_tstamp_prequeue = 1
    net.core.optmem_max = 20480
    net.core.rmem_default = 212992
    net.core.rmem_max = 212992
    net.core.rps_sock_flow_entries = 0
    net.core.somaxconn = 128
    net.core.tstamp_allow_data = 1
    net.core.warnings = 0
    net.core.wmem_default = 212992
    net.core.wmem_max = 212992
    net.core.xfrm_acq_expires = 30
    net.core.xfrm_aevent_etime = 10
    net.core.xfrm_aevent_rseqth = 2
    net.core.xfrm_larval_drop = 1
    net.fan.vxlan = 4
    net.ipv4.cipso_cache_bucket_size = 10
    net.ipv4.cipso_cache_enable = 1
    net.ipv4.cipso_rbm_optfmt = 0
    net.ipv4.cipso_rbm_strictvalid = 1
    net.ipv4.conf.all.accept_local = 0
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.all.accept_source_route = 0
    net.ipv4.conf.all.arp_accept = 0
    net.ipv4.conf.all.arp_announce = 0
    net.ipv4.conf.all.arp_filter = 0
    net.ipv4.conf.all.arp_ignore = 0
    net.ipv4.conf.all.arp_notify = 0
    net.ipv4.conf.all.bootp_relay = 0
    net.ipv4.conf.all.disable_policy = 0
    net.ipv4.conf.all.disable_xfrm = 0
    net.ipv4.conf.all.force_igmp_version = 0
    net.ipv4.conf.all.forwarding = 1
    net.ipv4.conf.all.igmpv2_unsolicited_report_interval = 10000
    net.ipv4.conf.all.igmpv3_unsolicited_report_interval = 1000
    net.ipv4.conf.all.ignore_routes_with_linkdown = 0
    net.ipv4.conf.all.log_martians = 0
    net.ipv4.conf.all.mc_forwarding = 0
    net.ipv4.conf.all.medium_id = 0
    net.ipv4.conf.all.promote_secondaries = 0
    net.ipv4.conf.all.proxy_arp = 0
    net.ipv4.conf.all.proxy_arp_pvlan = 0
    net.ipv4.conf.all.route_localnet = 0
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.conf.all.secure_redirects = 1
    net.ipv4.conf.all.send_redirects = 1
    net.ipv4.conf.all.shared_media = 1
    net.ipv4.conf.all.src_valid_mark = 0
    net.ipv4.conf.all.tag = 0
    net.ipv4.conf.default.accept_local = 0
    net.ipv4.conf.default.accept_redirects = 1
    net.ipv4.conf.default.accept_source_route = 1
    net.ipv4.conf.default.arp_accept = 0
    net.ipv4.conf.default.arp_announce = 0
    net.ipv4.conf.default.arp_filter = 0
    net.ipv4.conf.default.arp_ignore = 0
    net.ipv4.conf.default.arp_notify = 0
    net.ipv4.conf.default.bootp_relay = 0
    net.ipv4.conf.default.disable_policy = 0
    net.ipv4.conf.default.disable_xfrm = 0
    net.ipv4.conf.default.force_igmp_version = 0
    net.ipv4.conf.default.forwarding = 1
    net.ipv4.conf.default.igmpv2_unsolicited_report_interval = 10000
    net.ipv4.conf.default.igmpv3_unsolicited_report_interval = 1000
    net.ipv4.conf.default.ignore_routes_with_linkdown = 0
    net.ipv4.conf.default.log_martians = 0
    net.ipv4.conf.default.mc_forwarding = 0
    net.ipv4.conf.default.medium_id = 0
    net.ipv4.conf.default.promote_secondaries = 0
    net.ipv4.conf.default.proxy_arp = 0
    net.ipv4.conf.default.proxy_arp_pvlan = 0
    net.ipv4.conf.default.route_localnet = 0
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.secure_redirects = 1
    net.ipv4.conf.default.send_redirects = 1
    net.ipv4.conf.default.shared_media = 1
    net.ipv4.conf.default.src_valid_mark = 0
    net.ipv4.conf.default.tag = 0
    net.ipv4.conf.enp25s0f0.accept_local = 0
    net.ipv4.conf.enp25s0f0.accept_redirects = 1
    net.ipv4.conf.enp25s0f0.accept_source_route = 1
    net.ipv4.conf.enp25s0f0.arp_accept = 0
    net.ipv4.conf.enp25s0f0.arp_announce = 0
    net.ipv4.conf.enp25s0f0.arp_filter = 0
    net.ipv4.conf.enp25s0f0.arp_ignore = 0
    net.ipv4.conf.enp25s0f0.arp_notify = 0
    net.ipv4.conf.enp25s0f0.bootp_relay = 0
    net.ipv4.conf.enp25s0f0.disable_policy = 0
    net.ipv4.conf.enp25s0f0.disable_xfrm = 0
    net.ipv4.conf.enp25s0f0.force_igmp_version = 0
    net.ipv4.conf.enp25s0f0.forwarding = 1
    net.ipv4.conf.enp25s0f0.igmpv2_unsolicited_report_interval = 10000
    net.ipv4.conf.enp25s0f0.igmpv3_unsolicited_report_interval = 1000
    net.ipv4.conf.enp25s0f0.ignore_routes_with_linkdown = 0
    net.ipv4.conf.enp25s0f0.log_martians = 0
    net.ipv4.conf.enp25s0f0.mc_forwarding = 0
    net.ipv4.conf.enp25s0f0.medium_id = 0
    net.ipv4.conf.enp25s0f0.promote_secondaries = 0
    net.ipv4.conf.enp25s0f0.proxy_arp = 0
    net.ipv4.conf.enp25s0f0.proxy_arp_pvlan = 0
    net.ipv4.conf.enp25s0f0.route_localnet = 0
    net.ipv4.conf.enp25s0f0.rp_filter = 1
    net.ipv4.conf.enp25s0f0.secure_redirects = 1
    net.ipv4.conf.enp25s0f0.send_redirects = 1
    net.ipv4.conf.enp25s0f0.shared_media = 1
    net.ipv4.conf.enp25s0f0.src_valid_mark = 0
    net.ipv4.conf.enp25s0f0.tag = 0
    net.ipv4.conf.lo.accept_local = 0
    net.ipv4.conf.lo.accept_redirects = 1
    net.ipv4.conf.lo.accept_source_route = 1
    net.ipv4.conf.lo.arp_accept = 0
    net.ipv4.conf.lo.arp_announce = 0
    net.ipv4.conf.lo.arp_filter = 0
    net.ipv4.conf.lo.arp_ignore = 0
    net.ipv4.conf.lo.arp_notify = 0
    net.ipv4.conf.lo.bootp_relay = 0
    net.ipv4.conf.lo.disable_policy = 1
    net.ipv4.conf.lo.disable_xfrm = 1
    net.ipv4.conf.lo.force_igmp_version = 0
    net.ipv4.conf.lo.forwarding = 1
    net.ipv4.conf.lo.igmpv2_unsolicited_report_interval = 10000
    net.ipv4.conf.lo.igmpv3_unsolicited_report_interval = 1000
    net.ipv4.conf.lo.ignore_routes_with_linkdown = 0
    net.ipv4.conf.lo.log_martians = 0
    net.ipv4.conf.lo.mc_forwarding = 0
    net.ipv4.conf.lo.medium_id = 0
    net.ipv4.conf.lo.promote_secondaries = 0
    net.ipv4.conf.lo.proxy_arp = 0
    net.ipv4.conf.lo.proxy_arp_pvlan = 0
    net.ipv4.conf.lo.route_localnet = 0
    net.ipv4.conf.lo.rp_filter = 0
    net.ipv4.conf.lo.secure_redirects = 1
    net.ipv4.conf.lo.send_redirects = 1
    net.ipv4.conf.lo.shared_media = 1
    net.ipv4.conf.lo.src_valid_mark = 0
    net.ipv4.conf.lo.tag = 0
    net.ipv4.conf.lxcbr0.accept_local = 0
    net.ipv4.conf.lxcbr0.accept_redirects = 1
    net.ipv4.conf.lxcbr0.accept_source_route = 1
    net.ipv4.conf.lxcbr0.arp_accept = 0
    net.ipv4.conf.lxcbr0.arp_announce = 0
    net.ipv4.conf.lxcbr0.arp_filter = 0
    net.ipv4.conf.lxcbr0.arp_ignore = 0
    net.ipv4.conf.lxcbr0.arp_notify = 0
    net.ipv4.conf.lxcbr0.bootp_relay = 0
    net.ipv4.conf.lxcbr0.disable_policy = 0
    net.ipv4.conf.lxcbr0.disable_xfrm = 0
    net.ipv4.conf.lxcbr0.force_igmp_version = 0
    net.ipv4.conf.lxcbr0.forwarding = 1
    net.ipv4.conf.lxcbr0.igmpv2_unsolicited_report_interval = 10000
    net.ipv4.conf.lxcbr0.igmpv3_unsolicited_report_interval = 1000
    net.ipv4.conf.lxcbr0.ignore_routes_with_linkdown = 0
    net.ipv4.conf.lxcbr0.log_martians = 0
    net.ipv4.conf.lxcbr0.mc_forwarding = 0
    net.ipv4.conf.lxcbr0.medium_id = 0
    net.ipv4.conf.lxcbr0.promote_secondaries = 0
    net.ipv4.conf.lxcbr0.proxy_arp = 0
    net.ipv4.conf.lxcbr0.proxy_arp_pvlan = 0
    net.ipv4.conf.lxcbr0.route_localnet = 0
    net.ipv4.conf.lxcbr0.rp_filter = 1
    net.ipv4.conf.lxcbr0.secure_redirects = 1
    net.ipv4.conf.lxcbr0.send_redirects = 1
    net.ipv4.conf.lxcbr0.shared_media = 1
    net.ipv4.conf.lxcbr0.src_valid_mark = 0
    net.ipv4.conf.lxcbr0.tag = 0
    net.ipv4.fwmark_reflect = 0
    net.ipv4.icmp_echo_ignore_all = 0
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    net.ipv4.icmp_errors_use_inbound_ifaddr = 0
    net.ipv4.icmp_ignore_bogus_error_responses = 1
    net.ipv4.icmp_msgs_burst = 50
    net.ipv4.icmp_msgs_per_sec = 1000
    net.ipv4.icmp_ratelimit = 1000
    net.ipv4.icmp_ratemask = 6168
    net.ipv4.igmp_link_local_mcast_reports = 1
    net.ipv4.igmp_max_memberships = 20
    net.ipv4.igmp_max_msf = 10
    net.ipv4.igmp_qrv = 2
    net.ipv4.inet_peer_maxttl = 600
    net.ipv4.inet_peer_minttl = 120
    net.ipv4.inet_peer_threshold = 65664
    net.ipv4.ip_default_ttl = 64
    net.ipv4.ip_dynaddr = 0
    net.ipv4.ip_early_demux = 1
    net.ipv4.ip_forward = 1
    net.ipv4.ip_forward_use_pmtu = 0
    net.ipv4.ip_local_port_range = 32768	60999
    net.ipv4.ip_local_reserved_ports =
    net.ipv4.ip_no_pmtu_disc = 0
    net.ipv4.ip_nonlocal_bind = 0
    net.ipv4.ipfrag_high_thresh = 4194304
    net.ipv4.ipfrag_low_thresh = 3145728
    net.ipv4.ipfrag_max_dist = 64
    net.ipv4.ipfrag_secret_interval = 0
    net.ipv4.ipfrag_time = 30
    net.ipv4.neigh.default.anycast_delay = 100
    net.ipv4.neigh.default.app_solicit = 0
    net.ipv4.neigh.default.base_reachable_time_ms = 30000
    net.ipv4.neigh.default.delay_first_probe_time = 5
    net.ipv4.neigh.default.gc_interval = 30
    net.ipv4.neigh.default.gc_stale_time = 60
    net.ipv4.neigh.default.gc_thresh1 = 128
    net.ipv4.neigh.default.gc_thresh2 = 512
    net.ipv4.neigh.default.gc_thresh3 = 1024
    net.ipv4.neigh.default.locktime = 100
    net.ipv4.neigh.default.mcast_resolicit = 0
    net.ipv4.neigh.default.mcast_solicit = 3
    net.ipv4.neigh.default.proxy_delay = 80
    net.ipv4.neigh.default.proxy_qlen = 64
    net.ipv4.neigh.default.retrans_time_ms = 1000
    net.ipv4.neigh.default.ucast_solicit = 3
    net.ipv4.neigh.default.unres_qlen = 31
    net.ipv4.neigh.default.unres_qlen_bytes = 65536
    net.ipv4.neigh.enp25s0f0.anycast_delay = 100
    net.ipv4.neigh.enp25s0f0.app_solicit = 0
    net.ipv4.neigh.enp25s0f0.base_reachable_time_ms = 30000
    net.ipv4.neigh.enp25s0f0.delay_first_probe_time = 5
    net.ipv4.neigh.enp25s0f0.gc_stale_time = 60
    net.ipv4.neigh.enp25s0f0.locktime = 100
    net.ipv4.neigh.enp25s0f0.mcast_resolicit = 0
    net.ipv4.neigh.enp25s0f0.mcast_solicit = 3
    net.ipv4.neigh.enp25s0f0.proxy_delay = 80
    net.ipv4.neigh.enp25s0f0.proxy_qlen = 64
    net.ipv4.neigh.enp25s0f0.retrans_time_ms = 1000
    net.ipv4.neigh.enp25s0f0.ucast_solicit = 3
    net.ipv4.neigh.enp25s0f0.unres_qlen = 31
    net.ipv4.neigh.enp25s0f0.unres_qlen_bytes = 65536
    net.ipv4.neigh.lo.anycast_delay = 100
    net.ipv4.neigh.lo.app_solicit = 0
    net.ipv4.neigh.lo.base_reachable_time_ms = 30000
    net.ipv4.neigh.lo.delay_first_probe_time = 5
    net.ipv4.neigh.lo.gc_stale_time = 60
    net.ipv4.neigh.lo.locktime = 100
    net.ipv4.neigh.lo.mcast_resolicit = 0
    net.ipv4.neigh.lo.mcast_solicit = 3
    net.ipv4.neigh.lo.proxy_delay = 80
    net.ipv4.neigh.lo.proxy_qlen = 64
    net.ipv4.neigh.lo.retrans_time_ms = 1000
    net.ipv4.neigh.lo.ucast_solicit = 3
    net.ipv4.neigh.lo.unres_qlen = 31
    net.ipv4.neigh.lo.unres_qlen_bytes = 65536
    net.ipv4.neigh.lxcbr0.anycast_delay = 100
    net.ipv4.neigh.lxcbr0.app_solicit = 0
    net.ipv4.neigh.lxcbr0.base_reachable_time_ms = 30000
    net.ipv4.neigh.lxcbr0.delay_first_probe_time = 5
    net.ipv4.neigh.lxcbr0.gc_stale_time = 60
    net.ipv4.neigh.lxcbr0.locktime = 100
    net.ipv4.neigh.lxcbr0.mcast_resolicit = 0
    net.ipv4.neigh.lxcbr0.mcast_solicit = 3
    net.ipv4.neigh.lxcbr0.proxy_delay = 80
    net.ipv4.neigh.lxcbr0.proxy_qlen = 64
    net.ipv4.neigh.lxcbr0.retrans_time_ms = 1000
    net.ipv4.neigh.lxcbr0.ucast_solicit = 3
    net.ipv4.neigh.lxcbr0.unres_qlen = 31
    net.ipv4.neigh.lxcbr0.unres_qlen_bytes = 65536
    net.ipv4.ping_group_range = 1	0
    net.ipv4.route.error_burst = 1250
    net.ipv4.route.error_cost = 250
    net.ipv4.route.gc_elasticity = 8
    net.ipv4.route.gc_interval = 60
    net.ipv4.route.gc_min_interval = 0
    net.ipv4.route.gc_min_interval_ms = 500
    net.ipv4.route.gc_thresh = -1
    net.ipv4.route.gc_timeout = 300
    net.ipv4.route.max_size = 2147483647
    net.ipv4.route.min_adv_mss = 256
    net.ipv4.route.min_pmtu = 552
    net.ipv4.route.mtu_expires = 600
    net.ipv4.route.redirect_load = 5
    net.ipv4.route.redirect_number = 9
    net.ipv4.route.redirect_silence = 5120
    net.ipv4.tcp_abort_on_overflow = 0
    net.ipv4.tcp_adv_win_scale = 1
    net.ipv4.tcp_allowed_congestion_control = cubic reno
    net.ipv4.tcp_app_win = 31
    net.ipv4.tcp_autocorking = 1
    net.ipv4.tcp_available_congestion_control = cubic reno
    net.ipv4.tcp_base_mss = 1024
    net.ipv4.tcp_challenge_ack_limit = 1000
    net.ipv4.tcp_congestion_control = cubic
    net.ipv4.tcp_dsack = 1
    net.ipv4.tcp_early_retrans = 3
    net.ipv4.tcp_ecn = 2
    net.ipv4.tcp_ecn_fallback = 1
    net.ipv4.tcp_fack = 1
    net.ipv4.tcp_fastopen = 1
    net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
    net.ipv4.tcp_fin_timeout = 60
    net.ipv4.tcp_frto = 2
    net.ipv4.tcp_fwmark_accept = 0
    net.ipv4.tcp_invalid_ratelimit = 500
    net.ipv4.tcp_keepalive_intvl = 75
    net.ipv4.tcp_keepalive_probes = 9
    net.ipv4.tcp_keepalive_time = 7200
    net.ipv4.tcp_limit_output_bytes = 262144
    net.ipv4.tcp_low_latency = 0
    net.ipv4.tcp_max_orphans = 262144
    net.ipv4.tcp_max_reordering = 300
    net.ipv4.tcp_max_syn_backlog = 2048
    net.ipv4.tcp_max_tw_buckets = 262144
    net.ipv4.tcp_mem = 6188856	8251809	12377712
    net.ipv4.tcp_min_rtt_wlen = 300
    net.ipv4.tcp_min_tso_segs = 2
    net.ipv4.tcp_moderate_rcvbuf = 1
    net.ipv4.tcp_mtu_probing = 0
    net.ipv4.tcp_no_metrics_save = 0
    net.ipv4.tcp_notsent_lowat = -1
    net.ipv4.tcp_orphan_retries = 0
    net.ipv4.tcp_pacing_ca_ratio = 120
    net.ipv4.tcp_pacing_ss_ratio = 200
    net.ipv4.tcp_probe_interval = 600
    net.ipv4.tcp_probe_threshold = 8
    net.ipv4.tcp_recovery = 1
    net.ipv4.tcp_reordering = 3
    net.ipv4.tcp_retrans_collapse = 1
    net.ipv4.tcp_retries1 = 3
    net.ipv4.tcp_retries2 = 15
    net.ipv4.tcp_rfc1337 = 0
    net.ipv4.tcp_rmem = 4096	87380	6291456
    net.ipv4.tcp_sack = 1
    net.ipv4.tcp_slow_start_after_idle = 1
    net.ipv4.tcp_stdurg = 0
    net.ipv4.tcp_syn_retries = 6
    net.ipv4.tcp_synack_retries = 5
    net.ipv4.tcp_syncookies = 1
    net.ipv4.tcp_thin_dupack = 0
    net.ipv4.tcp_thin_linear_timeouts = 0
    net.ipv4.tcp_timestamps = 1
    net.ipv4.tcp_tso_win_divisor = 3
    net.ipv4.tcp_tw_recycle = 0
    net.ipv4.tcp_tw_reuse = 0
    net.ipv4.tcp_window_scaling = 1
    net.ipv4.tcp_wmem = 4096	16384	4194304
    net.ipv4.tcp_workaround_signed_windows = 0
    net.ipv4.udp_mem = 12377712	16503618	24755424
    net.ipv4.udp_rmem_min = 4096
    net.ipv4.udp_wmem_min = 4096
    net.ipv4.xfrm4_gc_thresh = 2147483647
    net.ipv6.anycast_src_echo_reply = 0
    net.ipv6.auto_flowlabels = 1
    net.ipv6.bindv6only = 0
    net.ipv6.conf.all.accept_dad = 1
    net.ipv6.conf.all.accept_ra = 1
    net.ipv6.conf.all.accept_ra_defrtr = 1
    net.ipv6.conf.all.accept_ra_from_local = 0
    net.ipv6.conf.all.accept_ra_min_hop_limit = 1
    net.ipv6.conf.all.accept_ra_mtu = 1
    net.ipv6.conf.all.accept_ra_pinfo = 1
    net.ipv6.conf.all.accept_ra_rt_info_max_plen = 0
    net.ipv6.conf.all.accept_ra_rtr_pref = 1
    net.ipv6.conf.all.accept_redirects = 1
    net.ipv6.conf.all.accept_source_route = 0
    net.ipv6.conf.all.autoconf = 1
    net.ipv6.conf.all.dad_transmits = 1
    net.ipv6.conf.all.disable_ipv6 = 0
    net.ipv6.conf.all.force_mld_version = 0
    net.ipv6.conf.all.force_tllao = 0
    net.ipv6.conf.all.forwarding = 0
    net.ipv6.conf.all.hop_limit = 64
    net.ipv6.conf.all.ignore_routes_with_linkdown = 0
    net.ipv6.conf.all.max_addresses = 16
    net.ipv6.conf.all.max_desync_factor = 600
    net.ipv6.conf.all.mc_forwarding = 0
    net.ipv6.conf.all.mldv1_unsolicited_report_interval = 10000
    net.ipv6.conf.all.mldv2_unsolicited_report_interval = 1000
    net.ipv6.conf.all.mtu = 1280
    net.ipv6.conf.all.ndisc_notify = 0
    net.ipv6.conf.all.proxy_ndp = 0
    net.ipv6.conf.all.regen_max_retry = 3
    net.ipv6.conf.all.router_probe_interval = 60
    net.ipv6.conf.all.router_solicitation_delay = 1
    net.ipv6.conf.all.router_solicitation_interval = 4
    net.ipv6.conf.all.router_solicitations = 3
    sysctl: reading key "net.ipv6.conf.all.stable_secret"
    net.ipv6.conf.all.suppress_frag_ndisc = 1
    net.ipv6.conf.all.temp_prefered_lft = 86400
    net.ipv6.conf.all.temp_valid_lft = 604800
    net.ipv6.conf.all.use_oif_addrs_only = 0
    net.ipv6.conf.all.use_tempaddr = 2
    net.ipv6.conf.default.accept_dad = 1
    net.ipv6.conf.default.accept_ra = 1
    net.ipv6.conf.default.accept_ra_defrtr = 1
    net.ipv6.conf.default.accept_ra_from_local = 0
    net.ipv6.conf.default.accept_ra_min_hop_limit = 1
    net.ipv6.conf.default.accept_ra_mtu = 1
    net.ipv6.conf.default.accept_ra_pinfo = 1
    net.ipv6.conf.default.accept_ra_rt_info_max_plen = 0
    net.ipv6.conf.default.accept_ra_rtr_pref = 1
    net.ipv6.conf.default.accept_redirects = 1
    net.ipv6.conf.default.accept_source_route = 0
    net.ipv6.conf.default.autoconf = 1
    net.ipv6.conf.default.dad_transmits = 1
    net.ipv6.conf.default.disable_ipv6 = 0
    net.ipv6.conf.default.force_mld_version = 0
    net.ipv6.conf.default.force_tllao = 0
    net.ipv6.conf.default.forwarding = 0
    net.ipv6.conf.default.hop_limit = 64
    net.ipv6.conf.default.ignore_routes_with_linkdown = 0
    net.ipv6.conf.default.max_addresses = 16
    net.ipv6.conf.default.max_desync_factor = 600
    net.ipv6.conf.default.mc_forwarding = 0
    net.ipv6.conf.default.mldv1_unsolicited_report_interval = 10000
    net.ipv6.conf.default.mldv2_unsolicited_report_interval = 1000
    net.ipv6.conf.default.mtu = 1280
    net.ipv6.conf.default.ndisc_notify = 0
    net.ipv6.conf.default.proxy_ndp = 0
    net.ipv6.conf.default.regen_max_retry = 3
    net.ipv6.conf.default.router_probe_interval = 60
    net.ipv6.conf.default.router_solicitation_delay = 1
    net.ipv6.conf.default.router_solicitation_interval = 4
    net.ipv6.conf.default.router_solicitations = 3
    sysctl: reading key "net.ipv6.conf.default.stable_secret"
    net.ipv6.conf.default.suppress_frag_ndisc = 1
    net.ipv6.conf.default.temp_prefered_lft = 86400
    net.ipv6.conf.default.temp_valid_lft = 604800
    net.ipv6.conf.default.use_oif_addrs_only = 0
    net.ipv6.conf.default.use_tempaddr = 2
    net.ipv6.conf.enp25s0f0.accept_dad = 1
    net.ipv6.conf.enp25s0f0.accept_ra = 1
    net.ipv6.conf.enp25s0f0.accept_ra_defrtr = 1
    net.ipv6.conf.enp25s0f0.accept_ra_from_local = 0
    net.ipv6.conf.enp25s0f0.accept_ra_min_hop_limit = 1
    net.ipv6.conf.enp25s0f0.accept_ra_mtu = 1
    net.ipv6.conf.enp25s0f0.accept_ra_pinfo = 1
    net.ipv6.conf.enp25s0f0.accept_ra_rt_info_max_plen = 0
    net.ipv6.conf.enp25s0f0.accept_ra_rtr_pref = 1
    net.ipv6.conf.enp25s0f0.accept_redirects = 1
    net.ipv6.conf.enp25s0f0.accept_source_route = 0
    net.ipv6.conf.enp25s0f0.autoconf = 1
    net.ipv6.conf.enp25s0f0.dad_transmits = 1
    net.ipv6.conf.enp25s0f0.disable_ipv6 = 0
    net.ipv6.conf.enp25s0f0.force_mld_version = 0
    net.ipv6.conf.enp25s0f0.force_tllao = 0
    net.ipv6.conf.enp25s0f0.forwarding = 0
    net.ipv6.conf.enp25s0f0.hop_limit = 64
    net.ipv6.conf.enp25s0f0.ignore_routes_with_linkdown = 0
    net.ipv6.conf.enp25s0f0.max_addresses = 16
    net.ipv6.conf.enp25s0f0.max_desync_factor = 600
    net.ipv6.conf.enp25s0f0.mc_forwarding = 0
    net.ipv6.conf.enp25s0f0.mldv1_unsolicited_report_interval = 10000
    net.ipv6.conf.enp25s0f0.mldv2_unsolicited_report_interval = 1000
    net.ipv6.conf.enp25s0f0.mtu = 1500
    net.ipv6.conf.enp25s0f0.ndisc_notify = 0
    net.ipv6.conf.enp25s0f0.proxy_ndp = 0
    net.ipv6.conf.enp25s0f0.regen_max_retry = 3
    net.ipv6.conf.enp25s0f0.router_probe_interval = 60
    net.ipv6.conf.enp25s0f0.router_solicitation_delay = 1
    net.ipv6.conf.enp25s0f0.router_solicitation_interval = 4
    net.ipv6.conf.enp25s0f0.router_solicitations = 3
    sysctl: reading key "net.ipv6.conf.enp25s0f0.stable_secret"
    net.ipv6.conf.enp25s0f0.suppress_frag_ndisc = 1
    net.ipv6.conf.enp25s0f0.temp_prefered_lft = 86400
    net.ipv6.conf.enp25s0f0.temp_valid_lft = 604800
    net.ipv6.conf.enp25s0f0.use_oif_addrs_only = 0
    net.ipv6.conf.enp25s0f0.use_tempaddr = 0
    net.ipv6.conf.lo.accept_dad = -1
    net.ipv6.conf.lo.accept_ra = 1
    net.ipv6.conf.lo.accept_ra_defrtr = 1
    net.ipv6.conf.lo.accept_ra_from_local = 0
    net.ipv6.conf.lo.accept_ra_min_hop_limit = 1
    net.ipv6.conf.lo.accept_ra_mtu = 1
    net.ipv6.conf.lo.accept_ra_pinfo = 1
    net.ipv6.conf.lo.accept_ra_rt_info_max_plen = 0
    net.ipv6.conf.lo.accept_ra_rtr_pref = 1
    net.ipv6.conf.lo.accept_redirects = 1
    net.ipv6.conf.lo.accept_source_route = 0
    net.ipv6.conf.lo.autoconf = 1
    net.ipv6.conf.lo.dad_transmits = 1
    net.ipv6.conf.lo.disable_ipv6 = 0
    net.ipv6.conf.lo.force_mld_version = 0
    net.ipv6.conf.lo.force_tllao = 0
    net.ipv6.conf.lo.forwarding = 0
    net.ipv6.conf.lo.hop_limit = 64
    net.ipv6.conf.lo.ignore_routes_with_linkdown = 0
    net.ipv6.conf.lo.max_addresses = 16
    net.ipv6.conf.lo.max_desync_factor = 600
    net.ipv6.conf.lo.mc_forwarding = 0
    net.ipv6.conf.lo.mldv1_unsolicited_report_interval = 10000
    net.ipv6.conf.lo.mldv2_unsolicited_report_interval = 1000
    net.ipv6.conf.lo.mtu = 65536
    net.ipv6.conf.lo.ndisc_notify = 0
    net.ipv6.conf.lo.proxy_ndp = 0
    net.ipv6.conf.lo.regen_max_retry = 3
    net.ipv6.conf.lo.router_probe_interval = 60
    net.ipv6.conf.lo.router_solicitation_delay = 1
    net.ipv6.conf.lo.router_solicitation_interval = 4
    net.ipv6.conf.lo.router_solicitations = 3
    sysctl: reading key "net.ipv6.conf.lo.stable_secret"
    net.ipv6.conf.lo.suppress_frag_ndisc = 1
    net.ipv6.conf.lo.temp_prefered_lft = 86400
    net.ipv6.conf.lo.temp_valid_lft = 604800
    net.ipv6.conf.lo.use_oif_addrs_only = 0
    net.ipv6.conf.lo.use_tempaddr = -1
    net.ipv6.conf.lxcbr0.accept_dad = 0
    net.ipv6.conf.lxcbr0.accept_ra = 1
    net.ipv6.conf.lxcbr0.accept_ra_defrtr = 1
    net.ipv6.conf.lxcbr0.accept_ra_from_local = 0
    net.ipv6.conf.lxcbr0.accept_ra_min_hop_limit = 1
    net.ipv6.conf.lxcbr0.accept_ra_mtu = 1
    net.ipv6.conf.lxcbr0.accept_ra_pinfo = 1
    net.ipv6.conf.lxcbr0.accept_ra_rt_info_max_plen = 0
    net.ipv6.conf.lxcbr0.accept_ra_rtr_pref = 1
    net.ipv6.conf.lxcbr0.accept_redirects = 1
    net.ipv6.conf.lxcbr0.accept_source_route = 0
    net.ipv6.conf.lxcbr0.autoconf = 1
    net.ipv6.conf.lxcbr0.dad_transmits = 1
    net.ipv6.conf.lxcbr0.disable_ipv6 = 0
    net.ipv6.conf.lxcbr0.force_mld_version = 0
    net.ipv6.conf.lxcbr0.force_tllao = 0
    net.ipv6.conf.lxcbr0.forwarding = 0
    net.ipv6.conf.lxcbr0.hop_limit = 64
    net.ipv6.conf.lxcbr0.ignore_routes_with_linkdown = 0
    net.ipv6.conf.lxcbr0.max_addresses = 16
    net.ipv6.conf.lxcbr0.max_desync_factor = 600
    net.ipv6.conf.lxcbr0.mc_forwarding = 0
    net.ipv6.conf.lxcbr0.mldv1_unsolicited_report_interval = 10000
    net.ipv6.conf.lxcbr0.mldv2_unsolicited_report_interval = 1000
    net.ipv6.conf.lxcbr0.mtu = 1500
    net.ipv6.conf.lxcbr0.ndisc_notify = 0
    net.ipv6.conf.lxcbr0.proxy_ndp = 0
    net.ipv6.conf.lxcbr0.regen_max_retry = 3
    net.ipv6.conf.lxcbr0.router_probe_interval = 60
    net.ipv6.conf.lxcbr0.router_solicitation_delay = 1
    net.ipv6.conf.lxcbr0.router_solicitation_interval = 4
    net.ipv6.conf.lxcbr0.router_solicitations = 3
    sysctl: reading key "net.ipv6.conf.lxcbr0.stable_secret"
    net.ipv6.conf.lxcbr0.suppress_frag_ndisc = 1
    net.ipv6.conf.lxcbr0.temp_prefered_lft = 86400
    net.ipv6.conf.lxcbr0.temp_valid_lft = 604800
    net.ipv6.conf.lxcbr0.use_oif_addrs_only = 0
    net.ipv6.conf.lxcbr0.use_tempaddr = 2
    net.ipv6.flowlabel_consistency = 1
    net.ipv6.flowlabel_state_ranges = 0
    net.ipv6.fwmark_reflect = 0
    net.ipv6.icmp.ratelimit = 1000
    net.ipv6.idgen_delay = 1
    net.ipv6.idgen_retries = 3
    net.ipv6.ip6frag_high_thresh = 4194304
    net.ipv6.ip6frag_low_thresh = 3145728
    net.ipv6.ip6frag_secret_interval = 0
    net.ipv6.ip6frag_time = 60
    net.ipv6.ip_nonlocal_bind = 0
    net.ipv6.mld_max_msf = 64
    net.ipv6.mld_qrv = 2
    net.ipv6.neigh.default.anycast_delay = 100
    net.ipv6.neigh.default.app_solicit = 0
    net.ipv6.neigh.default.base_reachable_time_ms = 30000
    net.ipv6.neigh.default.delay_first_probe_time = 5
    net.ipv6.neigh.default.gc_interval = 30
    net.ipv6.neigh.default.gc_stale_time = 60
    net.ipv6.neigh.default.gc_thresh1 = 128
    net.ipv6.neigh.default.gc_thresh2 = 512
    net.ipv6.neigh.default.gc_thresh3 = 1024
    net.ipv6.neigh.default.locktime = 0
    net.ipv6.neigh.default.mcast_resolicit = 0
    net.ipv6.neigh.default.mcast_solicit = 3
    net.ipv6.neigh.default.proxy_delay = 80
    net.ipv6.neigh.default.proxy_qlen = 64
    net.ipv6.neigh.default.retrans_time_ms = 1000
    net.ipv6.neigh.default.ucast_solicit = 3
    net.ipv6.neigh.default.unres_qlen = 31
    net.ipv6.neigh.default.unres_qlen_bytes = 65536
    net.ipv6.neigh.enp25s0f0.anycast_delay = 100
    net.ipv6.neigh.enp25s0f0.app_solicit = 0
    net.ipv6.neigh.enp25s0f0.base_reachable_time_ms = 30000
    net.ipv6.neigh.enp25s0f0.delay_first_probe_time = 5
    net.ipv6.neigh.enp25s0f0.gc_stale_time = 60
    net.ipv6.neigh.enp25s0f0.locktime = 0
    net.ipv6.neigh.enp25s0f0.mcast_resolicit = 0
    net.ipv6.neigh.enp25s0f0.mcast_solicit = 3
    net.ipv6.neigh.enp25s0f0.proxy_delay = 80
    net.ipv6.neigh.enp25s0f0.proxy_qlen = 64
    net.ipv6.neigh.enp25s0f0.retrans_time_ms = 1000
    net.ipv6.neigh.enp25s0f0.ucast_solicit = 3
    net.ipv6.neigh.enp25s0f0.unres_qlen = 31
    net.ipv6.neigh.enp25s0f0.unres_qlen_bytes = 65536
    net.ipv6.neigh.lo.anycast_delay = 100
    net.ipv6.neigh.lo.app_solicit = 0
    net.ipv6.neigh.lo.base_reachable_time_ms = 30000
    net.ipv6.neigh.lo.delay_first_probe_time = 5
    net.ipv6.neigh.lo.gc_stale_time = 60
    net.ipv6.neigh.lo.locktime = 0
    net.ipv6.neigh.lo.mcast_resolicit = 0
    net.ipv6.neigh.lo.mcast_solicit = 3
    net.ipv6.neigh.lo.proxy_delay = 80
    net.ipv6.neigh.lo.proxy_qlen = 64
    net.ipv6.neigh.lo.retrans_time_ms = 1000
    net.ipv6.neigh.lo.ucast_solicit = 3
    net.ipv6.neigh.lo.unres_qlen = 31
    net.ipv6.neigh.lo.unres_qlen_bytes = 65536
    net.ipv6.neigh.lxcbr0.anycast_delay = 100
    net.ipv6.neigh.lxcbr0.app_solicit = 0
    net.ipv6.neigh.lxcbr0.base_reachable_time_ms = 30000
    net.ipv6.neigh.lxcbr0.delay_first_probe_time = 5
    net.ipv6.neigh.lxcbr0.gc_stale_time = 60
    net.ipv6.neigh.lxcbr0.locktime = 0
    net.ipv6.neigh.lxcbr0.mcast_resolicit = 0
    net.ipv6.neigh.lxcbr0.mcast_solicit = 3
    net.ipv6.neigh.lxcbr0.proxy_delay = 80
    net.ipv6.neigh.lxcbr0.proxy_qlen = 64
    net.ipv6.neigh.lxcbr0.retrans_time_ms = 1000
    net.ipv6.neigh.lxcbr0.ucast_solicit = 3
    net.ipv6.neigh.lxcbr0.unres_qlen = 31
    net.ipv6.neigh.lxcbr0.unres_qlen_bytes = 65536
    net.ipv6.route.gc_elasticity = 9
    net.ipv6.route.gc_interval = 30
    net.ipv6.route.gc_min_interval = 0
    net.ipv6.route.gc_min_interval_ms = 500
    net.ipv6.route.gc_thresh = 1024
    net.ipv6.route.gc_timeout = 60
    net.ipv6.route.max_size = 4096
    net.ipv6.route.min_adv_mss = 1220
    net.ipv6.route.mtu_expires = 600
    net.ipv6.xfrm6_gc_thresh = 2147483647
    net.netfilter.nf_conntrack_acct = 0
    net.netfilter.nf_conntrack_buckets = 65536
    net.netfilter.nf_conntrack_checksum = 1
    net.netfilter.nf_conntrack_count = 2
    net.netfilter.nf_conntrack_events = 1
    net.netfilter.nf_conntrack_expect_max = 1024
    net.netfilter.nf_conntrack_generic_timeout = 600
    net.netfilter.nf_conntrack_helper = 1
    net.netfilter.nf_conntrack_icmp_timeout = 30
    net.netfilter.nf_conntrack_log_invalid = 0
    net.netfilter.nf_conntrack_max = 262144
    net.netfilter.nf_conntrack_tcp_be_liberal = 0
    net.netfilter.nf_conntrack_tcp_loose = 1
    net.netfilter.nf_conntrack_tcp_max_retrans = 3
    net.netfilter.nf_conntrack_tcp_timeout_close = 10
    net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
    net.netfilter.nf_conntrack_tcp_timeout_established = 432000
    net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
    net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
    net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
    net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
    net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
    net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
    net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
    net.netfilter.nf_conntrack_timestamp = 0
    net.netfilter.nf_conntrack_udp_timeout = 30
    net.netfilter.nf_conntrack_udp_timeout_stream = 180
    net.netfilter.nf_log.0 = NONE
    net.netfilter.nf_log.1 = NONE
    net.netfilter.nf_log.10 = NONE
    net.netfilter.nf_log.11 = NONE
    net.netfilter.nf_log.12 = NONE
    net.netfilter.nf_log.2 = NONE
    net.netfilter.nf_log.3 = NONE
    net.netfilter.nf_log.4 = NONE
    net.netfilter.nf_log.5 = NONE
    net.netfilter.nf_log.6 = NONE
    net.netfilter.nf_log.7 = NONE
    net.netfilter.nf_log.8 = NONE
    net.netfilter.nf_log.9 = NONE
    net.nf_conntrack_max = 262144
    net.unix.max_dgram_qlen = 512
    vm.admin_reserve_kbytes = 8192
    vm.block_dump = 0
    vm.compact_unevictable_allowed = 1
    vm.dirty_background_bytes = 0
    vm.dirty_background_ratio = 10
    vm.dirty_bytes = 0
    vm.dirty_expire_centisecs = 3000
    vm.dirty_ratio = 20
    vm.dirty_writeback_centisecs = 500
    vm.dirtytime_expire_seconds = 43200
    vm.drop_caches = 0
    vm.extfrag_threshold = 500
    vm.hugepages_treat_as_movable = 0
    vm.hugetlb_shm_group = 0
    vm.laptop_mode = 0
    vm.legacy_va_layout = 0
    vm.lowmem_reserve_ratio = 256	256	32	1
    vm.max_map_count = 200000
    vm.memory_failure_early_kill = 0
    vm.memory_failure_recovery = 1
    vm.min_free_kbytes = 90112
    vm.min_slab_ratio = 5
    vm.min_unmapped_ratio = 1
    vm.mmap_min_addr = 65536
    vm.nr_hugepages = 4096
    vm.nr_hugepages_mempolicy = 4096
    vm.nr_overcommit_hugepages = 0
    vm.nr_pdflush_threads = 0
    vm.numa_zonelist_order = default
    vm.oom_dump_tasks = 1
    vm.oom_kill_allocating_task = 0
    vm.overcommit_kbytes = 0
    vm.overcommit_memory = 0
    vm.overcommit_ratio = 50
    vm.page-cluster = 3
    vm.panic_on_oom = 0
    vm.percpu_pagelist_fraction = 0
    vm.stat_interval = 1
    vm.swappiness = 0
    vm.user_reserve_kbytes = 131072
    vm.vfs_cache_pressure = 100
    vm.zone_reclaim_mode = 0

**Services listing**

::

    $ service --status-all
     [ + ]  apparmor
     [ - ]  bootmisc.sh
     [ - ]  checkfs.sh
     [ - ]  checkroot-bootclean.sh
     [ - ]  checkroot.sh
     [ + ]  console-setup
     [ + ]  cpufrequtils
     [ + ]  cron
     [ + ]  dbus
     [ + ]  docker
     [ + ]  ebtables
     [ + ]  grub-common
     [ - ]  hostname.sh
     [ - ]  hwclock.sh
     [ + ]  keyboard-setup
     [ - ]  killprocs
     [ + ]  kmod
     [ + ]  loadcpufreq
     [ + ]  lxcfs
     [ - ]  mountall-bootclean.sh
     [ - ]  mountall.sh
     [ - ]  mountdevsubfs.sh
     [ - ]  mountkernfs.sh
     [ - ]  mountnfs-bootclean.sh
     [ - ]  mountnfs.sh
     [ + ]  networking
     [ - ]  ondemand
     [ - ]  plymouth
     [ - ]  plymouth-log
     [ + ]  procps
     [ + ]  qemu-kvm
     [ - ]  rc.local
     [ + ]  resolvconf
     [ - ]  rsync
     [ + ]  rsyslog
     [ - ]  screen-cleanup
     [ - ]  sendsigs
     [ + ]  ssh
     [ + ]  udev
     [ - ]  umountfs
     [ - ]  umountnfs.sh
     [ - ]  umountroot
     [ + ]  urandom
     [ - ]  uuidd
     [ - ]  x11-common

**Host CFS optimizations (QEMU+VPP)**

Applying CFS scheduler tuning on all Qemu vcpu worker threads (those are
handling testpmd - pmd threads) and VPP PMD worker threads. List of VPP PMD
threads can be obtained e.g. from:

::

    $ for psid in $(pgrep vpp)
    $ do
    $     for tid in $(ps -Lo tid --pid $psid | grep -v TID)
    $     do
    $         echo $tid
    $     done
    $ done

Or:

::

    $ cat /proc/`pidof vpp`/task/*/stat | awk '{print $1" "$2" "$39}'

Applying Round-robin scheduling with highest priority

::

    $ for psid in $(pgrep vpp)
    $ do
    $     for tid in $(ps -Lo tid --pid $psid | grep -v TID)
    $     do
    $         chrt -r -p 1 $tid
    $     done
    $ done

More information about Linux CFS can be found in: `Sched manual pages
<http://man7.org/linux/man-pages/man7/sched.7.html>`_.


**Host IRQ affinity**

Changing the default pinning of every IRQ to core 0. (Same does apply on both
guest VM and host OS)

::

    $ for l in `ls /proc/irq`; do echo 1 | sudo tee /proc/irq/$l/smp_affinity; done

**Host RCU affinity**

Changing the default pinning of RCU to core 0. (Same does apply on both guest VM
and host OS)

::

    $ for i in `pgrep rcu[^c]` ; do sudo taskset -pc 0 $i ; done

**Host Writeback affinity**

Changing the default pinning of writebacks to core 0. (Same does apply on both
guest VM and host OS)

::

    $ echo 1 | sudo tee /sys/bus/workqueue/devices/writeback/cpumask


DUT Configuration - VPP
-----------------------

**VPP Version**

|vpp-release|

**VPP Compile Parameters**

`FD.io VPP compile job`_

**VPP Install Parameters**

::

    $ dpkg -i --force-all vpp*

**VPP Startup Configuration**

VPP startup configuration changes per test case with different settings for CPU
cores, rx-queues and no-multi-seg parameter. Startup config is aligned with
applied test case tag:

Tagged by **1T1C**

::

    unix
    {
      cli-listen localhost:5002
      log /tmp/vpe.log
      nodaemon
    }
    cpu
    {
      corelist-workers 2
      main-core 1
    }
    ip4
    {
      heap-size "4G"
    }
    ip6
    {
      heap-size "4G"
      hash-buckets "2000000"
    }
    heapsize "4G"
    dpdk
    {
      dev 0000:88:00.1
      dev 0000:88:00.0
      no-multi-seg
      dev default
      {
        num-rx-desc 2048
        num-rx-queues 1
        num-tx-desc 2048
      }
      socket-mem "1024,1024
    }

Tagged by **2T1C**

::

    unix
    {
      cli-listen localhost:5002
      log /tmp/vpe.log
      nodaemon
    }
    cpu
    {
      corelist-workers 2,3
      main-core 1
    }
    ip4
    {
      heap-size "4G"
    }
    ip6
    {
      heap-size "4G"
      hash-buckets "2000000"
    }
    heapsize "4G"
    dpdk
    {
      dev 0000:88:00.1
      dev 0000:88:00.0
      no-multi-seg
      dev default
      {
        num-rx-desc 2048
        num-rx-queues 1
        num-tx-desc 2048
      }
      socket-mem "1024,1024
    }

Tagged by **4T4C**

::

    unix
    {
      cli-listen localhost:5002
      log /tmp/vpe.log
      nodaemon
    }
    cpu
    {
      corelist-workers 2,3,4,5
      main-core 1
    }
    ip4
    {
      heap-size "4G"
    }
    ip6
    {
      heap-size "4G"
      hash-buckets "2000000"
    }
    heapsize "4G"
    dpdk
    {
      dev 0000:88:00.1
      dev 0000:88:00.0
      no-multi-seg
      dev default
      {
        num-rx-desc 2048
        num-rx-queues 2
        num-tx-desc 2048
      }
      socket-mem "1024,1024
    }


TG Configuration - TRex
-----------------------

**TG Version**

|trex-release|

**DPDK version**

DPDK v17.11

**TG Build Script used**

`TRex intallation`_

**TG Startup Configuration**

::

    $ cat /etc/trex_cfg.yaml
    - port_limit      : 2
      version         : 2
      interfaces      : ["0000:0d:00.0","0000:0d:00.1"]
      port_info       :
        - dest_mac        :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf5]
          src_mac         :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf4]
        - dest_mac        :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf4]
          src_mac         :   [0x3c,0xfd,0xfe,0x9c,0xee,0xf5]

**TG Startup Command**

::

    $ sh -c 'cd <t-rex-install-dir>/scripts/ && sudo nohup ./t-rex-64 -i -c 7 --iom 0 > /tmp/trex.log 2>&1 &'> /dev/null

**TG common API - pointer to driver**

`TRex driver`_