Table of Contents

1. Abstract

Napatech SmartNIC family support 1/10/25/40/50/100 Gb/s Ethernet speeds on various cards. The NT200A01 card can potentially support all speeds within the one card. The Napatech SmartNICs operate using a bifurcated driver, the main driver resides in userspace and DPDK run on top of that driver. The Napatech SmartNIC driver suite provide it’s own kernel driver and does therefore not need igb_uio. In order to run TRex on Napatech SmartNICs the Napatech driver suite must be install and running.

2. Installation

As of writing this is the following release: "Huntington Beach 4.1½" - ntanl_package_3gd-11.0.2-linux.tar.gz

2.1. Extract and install

 [bash]>tar xf ntanl_package_3gd-11.0.2-linux.tar.gz
 [bash]>[bash]>cd ntanl_package_3gd_linux_11.0.2
 [bash]>sh ./package_install_3gd-11.0.2.sh --noprompt
[ Install documentation ]
[ Extract and install tools ]
Verifying archive integrity... All good.
Uncompressing Napatech A/S tools package v 3.8.2.16-4e54d....................................................................................................................................................................................................................................

######################################################################
######################################################################
######################################################################
######   _   _          _____     _______ ______ _____ _    _   ######
######  | \ | |   /\   |  __ \ /\|__   __|  ____/ ____| |  | |  ######
######  |  \| |  /  \  | |__) /  \  | |  | |__ | |    | |__| |  ######
######  |     | / /\ \ |  ___/ /\ \ | |  |  __|| |    |  __  |  ######
######  | |\  |/ ____ \| |  / ____ \| |  | |___| |____| |  | |  ######
######  |_| \_/_/    \_\_| /_/    \_\_|  |______\_____|_|  |_|  ######
######                                                          ######
######                      N T   T O O L S                     ######
######               ( T o o l s   P a c k a g e )              ######
######################################################################
######################################################################
######################################################################

Commencing installation procedure..
Creating installation directory at /opt/napatech3               [Done]
Installing tools                                                [Done]
Installing documentation                                        [Done]

######################################################################
#                 Installation completed successfully                #
#            See /tmp/nt_tools_3gd.log for a full install log        #
######################################################################
[ Extract and build driver ]
Verifying archive integrity... All good.
Uncompressing Napatech A/S driver package v 3.8.2.16-4e54d.............................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
.......................................................................................................................................................................................................................................................................................................................
..............................................................................................................................................................................................................................

######################################################################
######################################################################
######################################################################
######   _   _          _____     _______ ______ _____ _    _   ######
######  | \ | |   /\   |  __ \ /\|__   __|  ____/ ____| |  | |  ######
######  |  \| |  /  \  | |__) /  \  | |  | |__ | |    | |__| |  ######
######  |     | / /\ \ |  ___/ /\ \ | |  |  __|| |    |  __  |  ######
######  | |\  |/ ____ \| |  / ____ \| |  | |___| |____| |  | |  ######
######  |_| \_/_/    \_\_| /_/    \_\_|  |______\_____|_|  |_|  ######
######                                                          ######
######                    N T   D R I V E R S                   ######
######              ( D r i v e r   P a c k a g e )             ######
######################################################################
######################################################################
######################################################################

Commencing installation procedure..
Creating installation directory at /opt/napatech3               [Done]
Checking build and install disk space                           [Done]
Installing driver source                                        [Done]
Compiling the driver                                            [Done]
Installing driver                                               [Done]
Compiling the netdev driver                                     [Done]
Installing netdev driver                                        [Done]
Installing libraries                                            [Done]
Installing header files                                         [Done]
Installing binaries                                             [Done]
Installing config files                                         [Done]
Installing examples                                             [Done]
Installing documentation                                        [Done]
Checking dynamic linker bindings:
  Regenerate dynamic linker bindings                            [Done]
Installing symbolic links                                       [Done]

######################################################################
#                 Installation completed successfully                #
#            See /tmp/nt_driver_3gd.log for a full install log       #
######################################################################
[ Extract and install the Image control applications ]
[ Install Images ]
[ Package installation done... ]
[bash]>

2.2. Update the firmware

 [bash]>cd /opt/napatech3/images/
 [bash]>/opt/napatech3/bin/imgctrl -p --ini trex --ntimg NT200A01-2x100/200-8002-10-02-0000-180227-1015.ntimg
 imgctrl v3.6.2.1-b49d9 (Mar 12 2018 06:21:51) (Logfile: /tmp/imgctrl.log)
Scanning for Napatech adapters..
Adapter dual upgrade commencing..
Writing FPGA bank 0
  Erasing................... 100% done
  Writing................... 100% done
  Verifying................. 100% done
Done writing FPGA bank 0
Note: You can inspect the upload result through the Bank0 Contents
      section using the --query mode. To try the new image, remember
      to switch bank using the --switch mode. See --help for more information.
Writing FPGA bank 1
  Erasing................... 100% done
  Writing................... 100% done
  Verifying................. 100% done
Done writing FPGA bank 1
Note: You can inspect the upload result through the Bank1 Contents
      section using the --query mode. To try the new image, remember
      to switch bank using the --switch mode. See --help for more information.
FLASH UPGRADE completed
     BANK0           : 200-9516-10-07-00-180226-0843.ntimg
     BANK1           : 200-9515-10-07-00-180221-1628.ntimg
     PRIMARY BANK    : 0
Upgrade process complete

2.3. Create a driver configuration file

Starting the Napatech NTService daemon will create a default configuration file if no configuration file is present.

 [bash]>/opt/napatech3/bin/ntstart.sh
 Loading nt3gd driver                                        [Done]
 Creating driver device file                                 [Done]
 Loading nt3gd_netdev driver                                 [Done]
 Creating driver device file                                 [Done]
 Starting NTService (this may take a while)                  [Done]
 [bash]>/opt/napatech3/bin/ntstop.sh
 Stopping NTService (this may take a while)
 NTService stopped                                           [Done]
 [bash]>

2.4. Change the amount of Rx/Tx queues

The /opt/napatech3/config/ntservice.ini contains the following settings that control the number and size of hostbuffers. To enable TRex to simultaneous operate with may CPU cores/threads towards the SmartNIC, we need to increase the number of hostbuffers:

 ...
 HostBuffersRx = [4,16,-1]                # [x1, x2, x3], ...
 HostBuffersTx = [4,16,-1]                # [x1, x2, x3], ...
 ...
 ----
 They need to be changed to:
 ----
 ...
 HostBuffersRx = [32,16,-1]                # [x1, x2, x3], ...
 HostBuffersTx = [32,16,-1]                # [x1, x2, x3], ...
 ...
 [bash]>sed -i -- 's/x = \[4/x = \[32/g' /opt/napatech3/config/ntservice.ini
 [bash>

2.5. Start the Napatech driver

 [bash]>/opt/napatech3/bin/ntstart.sh
 Loading nt3gd driver                                        [Done]
 Creating driver device file                                 [Done]
 Loading nt3gd_netdev driver                                 [Done]
 Creating driver device file                                 [Done]
 Starting NTService (this may take a while)                  [Done]
 [bash]>

2.6. Build with Napatech support

Make sure to use a version of TRex downloaded from https://github.com/cisco-system-traffic-generator/trex-core (https://github.com/cisco-system-traffic-generator/trex-core/releases). TRex versions downloaded from here are in source form and contains a built-in DPDK

 [bash]>cd linux_dpdk
 [bash]>./b configure --with-ntacc
 Setting top to                           : /usr/builds/trex-core
 Setting out to                           : /usr/builds/trex-core/linux_dpdk/build_dpdk
 Checking for program 'g++, c++'          : /usr/bin/g++
 Checking for program 'ar'                : /usr/bin/ar
 Checking for program 'gcc, cc'           : /usr/bin/gcc
 Checking for program 'ar'                : /usr/bin/ar
 Checking for program 'ldd'               : /usr/bin/ldd
 Checking for library z                   : yes
 Checking for OFED                        : Found needed version 4.0
 Checking for library ibverbs             : yes
 Checking for NTAPI                       : Found needed NTAPI library
 'configure' finished successfully (14.827s)
 [bash]>./b
 Waf: Entering directory `/usr/builds/trex-core.github.rsync/linux_dpdk/build_dpdk'
 Info: Using external libverbs.
 update version files
 ...
 [1206/1210] Compiling version.c
 [1207/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug-o
 [1208/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug
 [1209/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-o
 [1210/1210] Linking build_dpdk/linux_dpdk/_t-rex-64
 Waf: Leaving directory `/usr/builds/trex-core.github.rsync/linux_dpdk/build_dpdk'
 *** generating softlinks ***
 'build' finished successfully (15.621s)

2.7. Get the adapter bus-id

  [bash]>/opt/napatech3/bin/adapterinfo
  adapterinfo (v. 3.8.2.16-4e54d)
  ==============================================================================
  System: 3.8.2.16
  ------------------------------------------------------------------------------
  Adapters:  1
  Ports:     2
  ==============================================================================
  Adapter: 2 - NT200A01-01-SCC-2x100-E3-FF-ANL Analysis Network Adapter
  ------------------------------------------------------------------------------
  Bus ID:             0000:82:00.0            #1
  PCI ID:             18F4:01A5
  FPGA ID:            200-9516-10-07-00
  Feature level:      N-ANL11
  Profile:            Capture
  Supported profiles: Capture

  AVR version:  3.5b

  Ports:            2
  Port Range:       0 - 1
  Descriptor:       Standard
  Timestamp:        Native Unix
  TimestampMethod:  EOF (End of Frame)

  RX host buffers:   32
  TX host buffers:   32

  Port #0 MAC : 00.0D.E9.11.22.33
  Port #1 MAC : 00.0D.E9.11.22.34

  SDRAM size  : 12GB

  Bus type                : PCI Express 3
  Link width supported    : x16
  Link width negotiated   : x16
  Link speed              : 8.0 GT/s
  Max payload supported   : 512
  Max payload negotiated  : 256
  Max expected throughput : 107209 Mbps
  ==============================================================================
   [bash]>
1 Napatech bus-id. Omit the domain 0000: in the config file

2.8. Config file example

To address the individual ports on the Napatech SmartNIC, you need to use a syntax of <pci-device>/<port-no>, see below for examples:

 ### Config file generated by dpdk_setup_ports.py ###

  - port_limit: 2
    version: 2
    interfaces: ['82:00.0/0', '82:00.0/1']
    port_info:
       - ip: 1.1.1.1
         default_gw: 2.2.2.2
       - ip: 2.2.2.2
         default_gw: 1.1.1.1

    platform:
       master_thread_id: 0
       latency_thread_id: 1
       dual_if:
         - socket: 1
           threads: [8,9,10,11,12,13,14]

In lieu of the ip and default_gw settings, you can alternatively specify a MAC-based configuration. In such case the port_info settings could look as follows:

      port_info:
      - dest_mac: 00:0d:e9:11:22:34 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  00:0d:e9:11:22:33
      - dest_mac: 00:0d:e9:11:22:33 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  00:0d:e9:11:22:34

You can obtain the MAC addresses from the output of the adapterinfo tool

It is possible to use dpdk_setup_ports.py script to produce the example config file:

[bash]> cd trex-core
[bash]> cd scripts/
[bash]> sudo ./dpdk_setup_ports.py -c 82:00.0/0 82:00.0/1 --force-macs
### Dumping config to screen, use -o flag to save to file
### Config file generated by dpdk_setup_ports.py ###

- port_limit: 2
  version: 2
  interfaces: ['82:00.0/0', '82:00.0/1']
  port_info:
      - dest_mac: 00:0d:e9:11:22:34 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  00:0d:e9:11:22:33
      - dest_mac: 00:0d:e9:11:22:33 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE
        src_mac:  00:0d:e9:11:22:34

  platform:
      master_thread_id: 0
      latency_thread_id: 2
      dual_if:
        - socket: 1
          threads: [1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63]
[source,bash]

Note that when using a NT100E3 SmartNIC (with a single 100G port) - for optimal performance, typically you would specify a dummy interface as the second port in a pair (see below as well)

3. Performance benchmark

3.1. Setup details

Server:

UCSC-C240-M4SX

CPU:

2 x Intel® Xeon® CPU E5-2650 v4 @ 2.20GHz

RAM:

64GB @ 2400 MHz

NICs:

2 x NT100E3-1-PTP Network Adapter 1x100Gb

QSFP:

2 x CFP4

OS:

CentOS 7.3

TRex:

pre-v2.38 using 10 cores per dual interface.

3.2. Topology

Loopback with optic cable.
We got two NICs, each one has single interface and belong to different NUMA.
In order to get optimal performance we have used 2 dummy interfaces as pair to real ones.
trex_cfg.yaml:

...
interfaces: ['07:00.0', 'dummy', 'dummy', '8a:00.0']
...

3.3. Stateless results

Ran bidirectional traffic between the two ports with following trex-console command:

start -p 0 3 -m 99.9% -f stl/imix.py
start -p 0 3 -m 99.9% -f stl/bench.py -t size=1514
start -p 0 3 -m 99.9% -f stl/bench.py -t size=590
start -p 0 3 -m 99.9% -f stl/bench.py -t size=128
start -p 0 3 -m 99.9% -f stl/bench.py -t size=64

Results summary:

images/napa_table.png