Table of Contents

TRex performance trend

This document tracks the performance of TRex Stateless use-cases of various hardware and NICS. The intention is to track the TRex software performance and not the NIC driver performance, however from this document you can learn the difference between different driver implementation (cycles/packets) Performance is measured in a metric called MPPS/Core (Norm), which stands for Mega-Packet per second (Norm).
How the calculation is made?

We measure the MPPS of the DUT in its best working point.
The measurement is then scaled to fit 100% CPU utilization, and finally divided by the number of cores the device has, to get a normalized MPPS/Core which is denoted as MPPS/Core (Norm)

How to run these tests?

       trex-core/scripts> ./run_regression --stl -t <name of test> --cfg setups/<name of setup>

The Error (%) metric:

You will see a column titled Error (%) next to other metrics (avg,std etc)
Error(%) is calculated for each batch of test results

Taking the Max value (max) and the Min value (min), calculating the difference max-min
We take the difference and then normalize it to the Average value to get (max-min)/Avg
Then, we present it as percentage

Error (%) = ((max-min)/Avg)*100

1. Setup: XL710 (4x40GbE) trex-08

Table 1. Setup Details
Server Cisco UCS 240M3

CPU:

2 sockets x Intel® Xeon® CPU E5-2667 v3 @ 3.20GHz, 8 cores

NICs:

2 NICS x ( 2x 40GbE ) Intel XL710, Total ports 4

NUMA

NIC0-NUMA 0 , NIC1-NUMA1

Memory

2x16GB

PCIe

2x PCIe16 + 2xPCIe4

OS:

Fedora 18 - baremetal

Test Name MPPS\Core (Norm) Date

1

syn-attack-64-bytes-single-CPU

13.55

20-02-2019-23:18

2

syn-attack-64-bytes-multi-CPUs

12.77

20-02-2019-23:16

3

VM-64-bytes-multi-CPU-cache-size-1024

35.83

20-02-2019-23:20

4

VM-64-bytes-multi-CPUs

15.25

20-02-2019-23:19

5

VM-64-bytes-single-CPU-cache-size-1024

34.86

20-02-2019-23:22

6

VM-64-bytes-single-CPU

16.36

20-02-2019-23:21

1.1. Trend: Performance Over Time

Test Name Avg MPPS/Core (Norm) Min Max Std Error (%) Total Results

syn-attack-64-bytes-single-CPU

13.49

12.51

13.58

0.11

7.95

145

syn-attack-64-bytes-multi-CPUs

12.88

12.48

13.02

0.08

4.23

145

VM-64-bytes-multi-CPU-cache-size-1024

36.29

33.45

37.12

0.59

10.13

145

VM-64-bytes-multi-CPUs

15.68

14.8

16.05

0.22

7.98

145

VM-64-bytes-single-CPU-cache-size-1024

34.72

30.48

39.59

0.83

26.24

145

VM-64-bytes-single-CPU

16.24

13.01

16.45

0.32

21.16

145

2. Setup: X710 (8x10GbE) trex-09

Table 2. Setup Details
Server Cisco UCS 240M3

CPU:

2 sockets x Intel® Xeon® CPU E5-2667 v3 @ 3.20GHz, 8 cores

NICs:

2 NICS x ( 2x 40GbE ) Intel X710, Total 8 ports

NUMA

NIC0-NUMA0 , NIC1-NUMA1

Memory

2x16GB

PCIe

2x PCIe16 + 2xPCIe4

OS:

Fedora 18 - baremetal

Test Name MPPS\Core (Norm) Date

1

VM-64-bytes-multi-CPUs

15.89

20-02-2019-23:19

2

syn-attack-64-bytes-multi-CPUs

12.95

20-02-2019-23:17

3

VM-64-bytes-multi-CPU-cache-size-1024

35.14

20-02-2019-23:21

4

syn-attack-64-bytes-single-CPU

13.49

20-02-2019-23:18

5

VM-64-bytes-single-CPU-cache-size-1024

33.46

20-02-2019-23:23

6

VM-64-bytes-single-CPU

16.07

20-02-2019-23:22

2.1. Trend: Performance Over Time

Test Name Avg MPPS/Core (Norm) Min Max Std Error (%) Total Results

VM-64-bytes-multi-CPUs

15.82

15.28

16.02

0.15

4.7

141

syn-attack-64-bytes-multi-CPUs

12.82

12.02

13.0

0.12

7.59

141

VM-64-bytes-multi-CPU-cache-size-1024

35.0

33.16

36.26

0.43

8.84

141

syn-attack-64-bytes-single-CPU

13.44

13.06

13.61

0.08

4.1

141

VM-64-bytes-single-CPU-cache-size-1024

33.33

32.21

34.4

0.28

6.58

140

VM-64-bytes-single-CPU

16.0

15.6

16.19

0.08

3.65

140

3. Setup: 82599EB (4x10GbE) kiwi02

Table 3. Setup Details
Server Cisco UCS 220M3

CPU:

2 x Intel® Xeon® CPU E5-2650 0 @ 2.00GHz (low end)

NICs:

2(NICS)x (2x10GbE) (4 ports) Intel 82599EB Total 4 ports

NUMA

NIC0-NUMA0 , NIC1-NUMA1

Memory

2x16GB

PCIe

1x PCIe16 + 1xPCIe8

OS:

Fedora 18 - baremetal

Test Name MPPS\Core (Norm) Date

1

VM-64-bytes-multi-CPUs

10.7

20-02-2019-23:17

2

syn-attack-64-bytes-multi-CPUs

8.65

20-02-2019-23:15

3

VM-64-bytes-multi-CPU-cache-size-1024

24.4

20-02-2019-23:18

4

VM-64-bytes-single-CPU-cache-size-1024

26.16

20-02-2019-23:21

5

VM-64-bytes-single-CPU

12.43

20-02-2019-23:19

6

syn-attack-64-bytes-single-CPU

9.7

20-02-2019-23:16

3.1. Trend: Trend: Performance Over Time

Test Name Avg MPPS/Core (Norm) Min Max Std Error (%) Total Results

VM-64-bytes-multi-CPUs

10.83

10.55

11.1

0.1

5.1

69

syn-attack-64-bytes-multi-CPUs

8.49

7.67

8.73

0.15

12.5

69

VM-64-bytes-multi-CPU-cache-size-1024

24.86

23.76

25.83

0.33

8.32

69

VM-64-bytes-single-CPU-cache-size-1024

25.17

22.12

29.52

0.97

29.41

69

VM-64-bytes-single-CPU

11.67

10.81

12.56

0.4

15.01

69

syn-attack-64-bytes-single-CPU

9.24

8.63

9.95

0.26

14.27

69

4. Setup: Cisco 1300 VIC (2x40GbE) trex-11

Important
  • this setup is not baremetal, performance are low due to that

Table 4. Setup Details
Server Cisco UCS 240M3

CPU:

2 sockets x Intel® Xeon® CPU E5-2667 v3 @ 3.20GHz, 8 cores

NICs:

1 NIC x ( 2x 40GbE ) Cisco 1300 VIC

NUMA

Unknown due to ESXi

Memory

2x16GB

PCIe

2x PCIe16 + 2xPCIe4

OS:

Ubuntu 14 - ESXi only 4 cores are utilized

Test Name MPPS\Core (Norm) Date

1

syn-attack-64-bytes-single-CPU

13.76

20-02-2019-23:17

2

syn-attack-64-bytes-multi-CPUs

14.03

20-02-2019-23:16

3

VM-64-bytes-multi-CPU-cache-size-1024

39.46

20-02-2019-23:19

4

VM-64-bytes-single-CPU-cache-size-1024

39.68

20-02-2019-23:22

5

VM-64-bytes-multi-CPUs

17.34

20-02-2019-23:18

6

VM-64-bytes-single-CPU

16.93

20-02-2019-23:20

4.1. Trend: Performance Over Time

Test Name Avg MPPS/Core (Norm) Min Max Std Error (%) Total Results

syn-attack-64-bytes-single-CPU

13.96

11.88

15.2

0.27

23.74

154

syn-attack-64-bytes-multi-CPUs

13.96

11.9

15.36

0.28

24.76

154

VM-64-bytes-multi-CPU-cache-size-1024

39.59

33.07

42.02

0.66

22.6

154

VM-64-bytes-single-CPU-cache-size-1024

39.62

32.03

43.82

0.8

29.74

154

VM-64-bytes-multi-CPUs

17.44

15.1

19.73

0.37

26.55

154

VM-64-bytes-single-CPU

17.43

14.64

19.31

0.38

26.78

154

5. Setup: Mellanox ConnectX-4 (2x100GbE) trex-07

Table 5. Setup Details
Server Cisco UCS 240

CPU:

2 sockets x Intel® Xeon® CPU E5-2667 v3 @ 3.20GHz, 8 cores

NICs:

1 NIC x 100G (2 ports) Mellanox ConnectX-4 NUMA#1

NUMA

One NIC on NUMA1

Memory

2x16GB

PCIe

2x PCIe16 + 2xPCIe4

OS:

CentOS 7.2 - baremetal

Test Name MPPS\Core (Norm) Date

1

syn-attack-64-bytes-single-CPU

12.32

20-02-2019-23:17

2

syn-attack-64-bytes-multi-CPUs

11.77

20-02-2019-23:16

3

VM-64-bytes-multi-CPU-cache-size-1024

24.63

20-02-2019-23:20

4

VM-64-bytes-single-CPU-cache-size-1024

23.42

20-02-2019-23:22

5

VM-64-bytes-multi-CPUs

13.99

20-02-2019-23:19

6

VM-64-bytes-single-CPU

15.03

20-02-2019-23:21

5.1. Trend: Performance Over Time

Test Name Avg MPPS/Core (Norm) Min Max Std Error (%) Total Results

syn-attack-64-bytes-single-CPU

12.28

12.06

12.34

0.05

2.25

114

syn-attack-64-bytes-multi-CPUs

11.71

11.51

11.77

0.04

2.24

114

VM-64-bytes-multi-CPU-cache-size-1024

24.61

24.21

25.01

0.18

3.25

114

VM-64-bytes-single-CPU-cache-size-1024

23.42

22.72

23.8

0.14

4.6

114

VM-64-bytes-multi-CPUs

13.96

13.69

14.0

0.04

2.28

114

VM-64-bytes-single-CPU

14.98

14.66

15.07

0.08

2.7

114

6. Setup: Mellanox ConnectX-5 (2x100GbE) trex-19

Table 6. Setup Details
Server Cisco UCS 240

CPU:

2 sockets x Intel® Xeon® CPU E5-2660 v4 @ 2.0GHz, 14 cores

NICs:

1 NIC x 100G (2 ports) Mellanox ConnectX-5 NUMA#0

NUMA

One NIC on NUMA1

Memory

2x32GB

PCIe

2x PCIe16 + 2xPCIe4

OS:

CentOS 7.4 - baremetal

Note

This is a new version of UCS with low-end CPU (2Ghz vs 3.2GHz) so cycle/packet should be normalized accordingly

Test Name MPPS\Core (Norm) Date

1

VM-64-bytes-multi-CPUs

13.16

20-02-2019-23:19

2

syn-attack-64-bytes-multi-CPUs

10.24

20-02-2019-23:17

3

VM-64-bytes-multi-CPU-cache-size-1024

34.74

20-02-2019-23:20

4

syn-attack-64-bytes-single-CPU

11.9

20-02-2019-23:18

5

VM-64-bytes-single-CPU-cache-size-1024

24.36

20-02-2019-23:23

6

VM-64-bytes-single-CPU

15.54

20-02-2019-23:21

6.1. Trend: Performance Over Time

Test Name Avg MPPS/Core (Norm) Min Max Std Error (%) Total Results

VM-64-bytes-multi-CPUs

13.39

11.07

15.24

0.54

31.18

154

syn-attack-64-bytes-multi-CPUs

10.38

8.98

11.84

0.37

27.48

154

VM-64-bytes-multi-CPU-cache-size-1024

35.22

27.03

39.81

1.68

36.3

154

syn-attack-64-bytes-single-CPU

12.18

11.34

12.67

0.21

10.92

154

VM-64-bytes-single-CPU-cache-size-1024

24.15

22.6

24.54

0.36

8.04

154

VM-64-bytes-single-CPU

15.11

14.22

16.02

0.3

11.93

154

7. Comparison between Mellanox ConnectX-5 (2x100GbE) (trex-19) and XL710 (4x40GbE) (trex-08)

Test Name Mellanox ConnectX-5 Intel XL710 Date

0

VM-64-bytes-multi-CPUs

13.16

15.25

20-02-2019-23:19

1

syn-attack-64-bytes-multi-CPUs

10.24

12.77

20-02-2019-23:16

2

VM-64-bytes-multi-CPU-cache-size-1024

34.74

35.83

20-02-2019-23:20

3

syn-attack-64-bytes-single-CPU

11.9

13.55

20-02-2019-23:18

4

VM-64-bytes-single-CPU-cache-size-1024

24.36

34.86

20-02-2019-23:22

5

VM-64-bytes-single-CPU

15.54

16.36

20-02-2019-23:21

8. Raw information

raw information can be found here raw_info.csv