1. Abstract
Napatech SmartNIC family support 1/10/25/40/50/100 Gb/s Ethernet speeds on various cards. The NT200A01 card can potentially support all speeds within the one card. The Napatech SmartNICs operate using a bifurcated driver, the main driver resides in userspace and DPDK run on top of that driver. The Napatech SmartNIC driver suite provide it’s own kernel driver and does therefore not need igb_uio. In order to run TRex on Napatech SmartNICs the Napatech driver suite must be install and running.
2. Installation
-
Download latest driver suite from https://www.napatech.com/downloads/
As of writing this is the following release: "Huntington Beach 4.1½" - ntanl_package_3gd-11.0.2-linux.tar.gz
2.1. Extract and install
[bash]>tar xf ntanl_package_3gd-11.0.2-linux.tar.gz [bash]>cd ntanl_package_3gd_linux_11.0.2 [bash]>sh ./package_install_3gd-11.0.2.sh --noprompt [ Install documentation ] [ Extract and install tools ] Verifying archive integrity... All good. Uncompressing Napatech A/S tools package v 3.8.2.16-4e54d.................................................................................................................................................................................................................................... ###################################################################### ###################################################################### ###################################################################### ###### _ _ _____ _______ ______ _____ _ _ ###### ###### | \ | | /\ | __ \ /\|__ __| ____/ ____| | | | ###### ###### | \| | / \ | |__) / \ | | | |__ | | | |__| | ###### ###### | | / /\ \ | ___/ /\ \ | | | __|| | | __ | ###### ###### | |\ |/ ____ \| | / ____ \| | | |___| |____| | | | ###### ###### |_| \_/_/ \_\_| /_/ \_\_| |______\_____|_| |_| ###### ###### ###### ###### N T T O O L S ###### ###### ( T o o l s P a c k a g e ) ###### ###################################################################### ###################################################################### ###################################################################### Commencing installation procedure.. Creating installation directory at /opt/napatech3 [Done] Installing tools [Done] Installing documentation [Done] ###################################################################### # Installation completed successfully # # See /tmp/nt_tools_3gd.log for a full install log # ###################################################################### [ Extract and build driver ] Verifying archive integrity... All good. Uncompressing Napatech A/S driver package v 3.8.2.16-4e54d............................................................................................................................................................................................................................................................. ....................................................................................................................................................................................................................................................................................................................... ....................................................................................................................................................................................................................................................................................................................... ....................................................................................................................................................................................................................................................................................................................... ....................................................................................................................................................................................................................................................................................................................... ....................................................................................................................................................................................................................................................................................................................... ....................................................................................................................................................................................................................................................................................................................... .............................................................................................................................................................................................................................. ###################################################################### ###################################################################### ###################################################################### ###### _ _ _____ _______ ______ _____ _ _ ###### ###### | \ | | /\ | __ \ /\|__ __| ____/ ____| | | | ###### ###### | \| | / \ | |__) / \ | | | |__ | | | |__| | ###### ###### | | / /\ \ | ___/ /\ \ | | | __|| | | __ | ###### ###### | |\ |/ ____ \| | / ____ \| | | |___| |____| | | | ###### ###### |_| \_/_/ \_\_| /_/ \_\_| |______\_____|_| |_| ###### ###### ###### ###### N T D R I V E R S ###### ###### ( D r i v e r P a c k a g e ) ###### ###################################################################### ###################################################################### ###################################################################### Commencing installation procedure.. Creating installation directory at /opt/napatech3 [Done] Checking build and install disk space [Done] Installing driver source [Done] Compiling the driver [Done] Installing driver [Done] Compiling the netdev driver [Done] Installing netdev driver [Done] Installing libraries [Done] Installing header files [Done] Installing binaries [Done] Installing config files [Done] Installing examples [Done] Installing documentation [Done] Checking dynamic linker bindings: Regenerate dynamic linker bindings [Done] Installing symbolic links [Done] ###################################################################### # Installation completed successfully # # See /tmp/nt_driver_3gd.log for a full install log # ###################################################################### [ Extract and install the Image control applications ] [ Install Images ] [ Package installation done... ] [bash]>
2.2. Update the firmware
Please also see https://docs.napatech.com/reader/DJq6hS577FbL1RHaPnWnaQ/h19IOLPuF4gKSxOYrtod2w for detailed instructions
[bash]>cd /opt/napatech3/images/ [bash]>/opt/napatech3/bin/imgctrl -p --ini trex --ntimg NT200A01-2x100/200-8002-10-02-0000-180227-1015.ntimg imgctrl v3.6.2.1-b49d9 (Mar 12 2018 06:21:51) (Logfile: /tmp/imgctrl.log) Scanning for Napatech adapters.. Adapter dual upgrade commencing.. Writing FPGA bank 0 Erasing................... 100% done Writing................... 100% done Verifying................. 100% done Done writing FPGA bank 0 Note: You can inspect the upload result through the Bank0 Contents section using the --query mode. To try the new image, remember to switch bank using the --switch mode. See --help for more information. Writing FPGA bank 1 Erasing................... 100% done Writing................... 100% done Verifying................. 100% done Done writing FPGA bank 1 Note: You can inspect the upload result through the Bank1 Contents section using the --query mode. To try the new image, remember to switch bank using the --switch mode. See --help for more information. FLASH UPGRADE completed BANK0 : 200-9516-10-07-00-180226-0843.ntimg BANK1 : 200-9515-10-07-00-180221-1628.ntimg PRIMARY BANK : 0 Upgrade process complete
2.3. Create a driver configuration file
Starting the Napatech NTService daemon will create a default configuration file if no configuration file is present.
[bash]>/opt/napatech3/bin/ntstart.sh Loading nt3gd driver [Done] Creating driver device file [Done] Loading nt3gd_netdev driver [Done] Creating driver device file [Done] Starting NTService (this may take a while) [Done] [bash]>/opt/napatech3/bin/ntstop.sh Stopping NTService (this may take a while) NTService stopped [Done] [bash]>
2.4. Change the amount of Rx/Tx queues
The /opt/napatech3/config/ntservice.ini contains the following settings that control the number and size of hostbuffers. To enable TRex to simultaneous operate with may CPU cores/threads towards the SmartNIC, we need to increase the number of hostbuffers:
... HostBuffersRx = [4,16,-1] # [x1, x2, x3], ... HostBuffersTx = [4,16,-1] # [x1, x2, x3], ... ... ---- They need to be changed to: ---- ... HostBuffersRx = [32,16,-1] # [x1, x2, x3], ... HostBuffersTx = [32,16,-1] # [x1, x2, x3], ... ...
[bash]>sed -i -- 's/x = \[4/x = \[32/g' /opt/napatech3/config/ntservice.ini [bash>
2.5. Napatech NIC driver tweaks
The following changes must be made to /opt/napatech3/config/ntservice.ini:
[System] ... StatInterval=1 ...
[Adapter?] ... HostBufferRefreshIntervalTx = 50 CancelTxOnCloseMask = 1 # This must match the number of ports on the NIC (1,3,F) ...
2.6. Start the Napatech driver
[bash]>/opt/napatech3/bin/ntstart.sh Loading nt3gd driver [Done] Creating driver device file [Done] Loading nt3gd_netdev driver [Done] Creating driver device file [Done] Starting NTService (this may take a while) [Done] [bash]>
2.7. Build with Napatech support
Make sure to use a version of TRex downloaded from https://github.com/cisco-system-traffic-generator/trex-core (https://github.com/cisco-system-traffic-generator/trex-core/releases). TRex versions downloaded from here are in source form and contains a built-in DPDK
[bash]>cd linux_dpdk [bash]>./b configure --with-ntacc Setting top to : /usr/builds/trex-core Setting out to : /usr/builds/trex-core/linux_dpdk/build_dpdk Checking for program 'g++, c++' : /usr/bin/g++ Checking for program 'ar' : /usr/bin/ar Checking for program 'gcc, cc' : /usr/bin/gcc Checking for program 'ar' : /usr/bin/ar Checking for program 'ldd' : /usr/bin/ldd Checking for library z : yes Checking for OFED : Found needed version 4.0 Checking for library ibverbs : yes Checking for NTAPI : Found needed NTAPI library 'configure' finished successfully (14.827s) [bash]>./b Waf: Entering directory `/usr/builds/trex-core.github.rsync/linux_dpdk/build_dpdk' Info: Using external libverbs. update version files ... [1206/1210] Compiling version.c [1207/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug-o [1208/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-debug [1209/1210] Linking build_dpdk/linux_dpdk/_t-rex-64-o [1210/1210] Linking build_dpdk/linux_dpdk/_t-rex-64 Waf: Leaving directory `/usr/builds/trex-core.github.rsync/linux_dpdk/build_dpdk' *** generating softlinks *** 'build' finished successfully (15.621s)
2.8. Get the adapter bus-id
[bash]>/opt/napatech3/bin/adapterinfo adapterinfo (v. 3.8.2.16-4e54d) ============================================================================== System: 3.8.2.16 ------------------------------------------------------------------------------ Adapters: 1 Ports: 2 ============================================================================== Adapter: 2 - NT200A01-01-SCC-2x100-E3-FF-ANL Analysis Network Adapter ------------------------------------------------------------------------------ Bus ID: 0000:82:00.0 #PCI ID: 18F4:01A5 FPGA ID: 200-9516-10-07-00 Feature level: N-ANL11 Profile: Capture Supported profiles: Capture AVR version: 3.5b Ports: 2 Port Range: 0 - 1 Descriptor: Standard Timestamp: Native Unix TimestampMethod: EOF (End of Frame) RX host buffers: 32 TX host buffers: 32 Port #0 MAC : 00.0D.E9.11.22.33 Port #1 MAC : 00.0D.E9.11.22.34 SDRAM size : 12GB Bus type : PCI Express 3 Link width supported : x16 Link width negotiated : x16 Link speed : 8.0 GT/s Max payload supported : 512 Max payload negotiated : 256 Max expected throughput : 107209 Mbps ============================================================================== [bash]>
| Napatech bus-id. Omit the domain 0000: in the config file |
2.9. Config file example
To address the individual ports on the Napatech SmartNIC, you need to use a syntax of <pci-device>/<port-no>, see below for examples:
### Config file generated by dpdk_setup_ports.py ### - port_limit: 2 version: 2 interfaces: ['82:00.0/0', '82:00.0/1'] port_info: - ip: 1.1.1.1 default_gw: 2.2.2.2 - ip: 2.2.2.2 default_gw: 1.1.1.1 platform: master_thread_id: 0 latency_thread_id: 1 dual_if: - socket: 1 threads: [8,9,10,11,12,13,14]
In lieu of the ip and default_gw settings, you can alternatively specify a MAC-based configuration. In such case the port_info settings could look as follows:
port_info: - dest_mac: 00:0d:e9:11:22:34 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE src_mac: 00:0d:e9:11:22:33 - dest_mac: 00:0d:e9:11:22:33 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE src_mac: 00:0d:e9:11:22:34
You can obtain the MAC addresses from the output of the adapterinfo tool
It is possible to use dpdk_setup_ports.py script to produce the example config file:
[bash]> cd trex-core [bash]> cd scripts/ [bash]> sudo ./dpdk_setup_ports.py -c 82:00.0/0 82:00.0/1 --force-macs ### Dumping config to screen, use -o flag to save to file ### Config file generated by dpdk_setup_ports.py ### - port_limit: 2 version: 2 interfaces: ['82:00.0/0', '82:00.0/1'] port_info: - dest_mac: 00:0d:e9:11:22:34 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE src_mac: 00:0d:e9:11:22:33 - dest_mac: 00:0d:e9:11:22:33 # MAC OF LOOPBACK TO IT'S DUAL INTERFACE src_mac: 00:0d:e9:11:22:34 platform: master_thread_id: 0 latency_thread_id: 2 dual_if: - socket: 1 threads: [1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63] [source,bash]
Note that when using a NT100E3 SmartNIC (with a single 100G port) - for optimal performance, typically you would specify a dummy interface as the second port in a pair (see below as well)
3. Performance benchmark
3.1. Setup details
Server: |
UCSC-C240-M4SX |
CPU: |
2 x Intel® Xeon® CPU E5-2650 v4 @ 2.20GHz |
RAM: |
64GB @ 2400 MHz |
NICs: |
2 x NT100E3-1-PTP Network Adapter 1x100Gb |
QSFP: |
2 x CFP4 |
OS: |
CentOS 7.3 |
TRex: |
pre-v2.38 using 10 cores per dual interface. |
3.2. Topology
Loopback with optic cable.
We got two NICs, each one has single interface and belong to different NUMA.
In order to get optimal performance we have used 2 dummy interfaces as pair to real ones.
trex_cfg.yaml:
... interfaces: ['07:00.0', 'dummy', 'dummy', '8a:00.0'] ...
3.3. Stateless results
Ran bidirectional traffic between the two ports with following trex-console command:
start -p 0 3 -m 99.9% -f stl/imix.py start -p 0 3 -m 99.9% -f stl/bench.py -t size=1514 start -p 0 3 -m 99.9% -f stl/bench.py -t size=590 start -p 0 3 -m 99.9% -f stl/bench.py -t size=128 start -p 0 3 -m 99.9% -f stl/bench.py -t size=64
Results summary:
Table of Contents