Overview
TRex Traffic Generator を Ubuntu 20.04 にインストールする。
Function Summary
- Stateless Mode – STL
- Stateful Mode / Advanced Stateful Mode – ASTF
- Emulation Server – EMU
- Bird Integration
Reference
Environment
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 94
Model name: Intel(R) Core(TM) i3-6100T CPU @ 3.20GHz
Stepping: 3
CPU MHz: 800.176
CPU max MHz: 3200.0000
CPU min MHz: 800.0000
BogoMIPS: 6399.96
Virtualization: VT-x
L1d cache: 64 KiB
L1i cache: 64 KiB
L2 cache: 512 KiB
L3 cache: 3 MiB
NUMA node0 CPU(s): 0-3
Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerab
le
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP c
onditional, RSB filling
Vulnerability Srbds: Mitigation; Microcode
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse3
6 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb r
dtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopolog
y nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx
est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe
popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnow
prefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_s
hadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2
smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveop
t xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act
_window hwp_epp md_clear flush_l1d
$
$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal $ $ uname -a Linux svr-trex 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux $
$ lspci | grep Ethernet 00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V 06:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) 06:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) 07:00.0 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) 07:00.1 Ethernet controller: Intel Corporation 82571EB/82571GB Gigabit Ethernet Controller (Copper) (rev 06) $
Installation
TRex
$ sudo su - # mkdir -p /opt/trex # cd /opt/trex # wget --no-cache --no-check-certificate https://trex-tgn.cisco.com/trex/release/v2.92.tar.gz # tar -xzvf v2.92.tar.gz # rm v2.92.tar.gz
Packages
# apt install linux-headers-`uname -r` # apt install build-essential
dpdk_setup_ports.py
dkdk_setup_ports.py は TRex で利用する Port を設定し /etc/trex_cfg.yaml を作成・更新するツールである。
root@svr-trex:/opt/trex/v2.92# ./dpdk_setup_ports.py -s
Network devices using DPDK-compatible driver
============================================
Network devices using kernel driver
===================================
0000:00:1f.6 'Ethernet Connection (2) I219-V' if=eno1 drv=e1000e unused=igb_uio,vfio-pci,uio_pci_generic *Active*
0000:06:00.0 '82571EB/82571GB Gigabit Ethernet Controller (Copper)' if=enp6s0f0 drv=e1000e unused=igb_uio,vfio-pci,uio_pci_generic
0000:06:00.1 '82571EB/82571GB Gigabit Ethernet Controller (Copper)' if=enp6s0f1 drv=e1000e unused=igb_uio,vfio-pci,uio_pci_generic
0000:07:00.0 '82571EB/82571GB Gigabit Ethernet Controller (Copper)' if=enp7s0f0 drv=e1000e unused=igb_uio,vfio-pci,uio_pci_generic
0000:07:00.1 '82571EB/82571GB Gigabit Ethernet Controller (Copper)' if=enp7s0f1 drv=e1000e unused=igb_uio,vfio-pci,uio_pci_generic
Other network devices
=====================
root@svr-trex:/opt/trex/v2.92#
root@svr-trex:/opt/trex/v2.92# ./dpdk_setup_ports.py -i
By default, IP based configuration file will be created. Do you want to use MAC based config? (y/N)N
+----+------+---------+-------------------+------------------------------------------------------+--------+----------+----------+
| ID | NUMA | PCI | MAC | Name | Driver | Linux IF | Active |
+====+======+=========+===================+======================================================+========+==========+==========+
| 0 | -1 | 00:1f.6 | ac:1f:6b:16:1f:79 | Ethernet Connection (2) I219-V | e1000e | eno1 | *Active* |
+----+------+---------+-------------------+------------------------------------------------------+--------+----------+----------+
| 1 | -1 | 06:00.0 | 00:1b:21:59:1d:11 | 82571EB/82571GB Gigabit Ethernet Controller (Copper) | e1000e | enp6s0f0 | |
+----+------+---------+-------------------+------------------------------------------------------+--------+----------+----------+
| 2 | -1 | 06:00.1 | 00:1b:21:59:1d:10 | 82571EB/82571GB Gigabit Ethernet Controller (Copper) | e1000e | enp6s0f1 | |
+----+------+---------+-------------------+------------------------------------------------------+--------+----------+----------+
| 3 | -1 | 07:00.0 | 00:1b:21:59:1d:13 | 82571EB/82571GB Gigabit Ethernet Controller (Copper) | e1000e | enp7s0f0 | |
+----+------+---------+-------------------+------------------------------------------------------+--------+----------+----------+
| 4 | -1 | 07:00.1 | 00:1b:21:59:1d:12 | 82571EB/82571GB Gigabit Ethernet Controller (Copper) | e1000e | enp7s0f1 | |
+----+------+---------+-------------------+------------------------------------------------------+--------+----------+----------+
Please choose an even number of interfaces from the list above, either by ID, PCI or Linux IF
Stateful will use order of interfaces: Client1 Server1 Client2 Server2 etc. for flows.
Stateless can be in any order.
Enter list of interfaces separated by space (for example: 1 3) : 1 2 3 4
For interface 1, assuming loopback to its dual interface 2.
Putting IP 1.1.1.1, default gw 2.2.2.2 Change it?(y/N).N
For interface 2, assuming loopback to its dual interface 1.
Putting IP 2.2.2.2, default gw 1.1.1.1 Change it?(y/N).N
For interface 3, assuming loopback to its dual interface 4.
Putting IP 3.3.3.3, default gw 4.4.4.4 Change it?(y/N).N
For interface 4, assuming loopback to its dual interface 3.
Putting IP 4.4.4.4, default gw 3.3.3.3 Change it?(y/N).N
Print preview of generated config? (Y/n)y
### Config file generated by dpdk_setup_ports.py ###
- version: 2
interfaces: ['06:00.0', '06:00.1', '07:00.0', '07:00.1']
port_info:
- ip: 1.1.1.1
default_gw: 2.2.2.2
- ip: 2.2.2.2
default_gw: 1.1.1.1
- ip: 3.3.3.3
default_gw: 4.4.4.4
- ip: 4.4.4.4
default_gw: 3.3.3.3
platform:
master_thread_id: 0
latency_thread_id: 1
dual_if:
- socket: 0
threads: [2]
- socket: 0
threads: [3]
Save the config to file? (Y/n)y
Default filename is /etc/trex_cfg.yaml
Press ENTER to confirm or enter new file:
Saved to /etc/trex_cfg.yaml.
root@svr-trex:/opt/trex/v2.92#
t-rex-64
help
# ./t-rex-64 --help
Starting TRex v2.92 please wait ...
Usage: t-rex-64 [mode]
mode is one of:
-f : YAML file with traffic template configuration (Will run TRex in 'stateful' mode)
-i : Run TRex in 'stateless' mode
Available options are:
--astf : Enable advanced stateful mode. profile should be in py format and not YAML format
--astf-server-only : Only server side ports (1,3..) are enabled with ASTF service. Traffic won't be transmitted on clients ports.
--astf-client-mask : Enable only specific client side ports with ASTF service.
For example, with 4 ports setup. 0x1 means that only port 0 will be enabled. ports 2 won't be enabled.
Can't be used with --astf-server-only.
--stl : Starts in stateless mode. must be provided along with '-i' for interactive mode
--active-flows : An experimental switch to scale up or down the number of active flows.
It is not accurate due to the quantization of flow scheduler and in some case does not work.
Example --active-flows 500000 wil set the ballpark of the active flow to be ~0.5M
--allow-coredump : Allow creation of core dump
--arp-refresh-period : Period in seconds between sending of gratuitous ARP for our addresses. Value of 0 means 'never send'
--limit-arp-request : The limit number of ARP requests for gw address resolution. Value of 0 means 'no limit.'
-c > : Number of hardware threads to allocate for each port pair. Overrides the 'c' argument from config file
--cfg : Use file as TRex config file instead of the default /etc/trex_cfg.yaml
--checksum-offload : Deprecated,enable by default. Enable IP, TCP and UDP tx checksum offloading, using DPDK. This requires all used interfaces to support this
--checksum-offload-disable : Disable IP, TCP and UDP tx checksum offloading, using DPDK. This requires all used interfaces to support this
--tso-disable : disable TSO (advanced TCP mode)
--lro-disable : disable LRO (advanced TCP mode)
--client-cfg : YAML file describing clients configuration
--close-at-end : Call rte_eth_dev_stop and close at exit. Calling these functions caused link down issues in older versions,
so we do not call them by default for now. Leaving this as option in case someone thinks it is helpful for him
This it temporary option. Will be removed in the future
-d : Duration of the test in sec (default is 3600). Look also at --nc
-e : Like -p but src/dst IP will be chosen according to the port (i.e. on client port send all packets with client src and server dest, and vice versa on server port
--flip : Each flow will be sent both from client to server and server to client. This can achieve better port utilization when flow traffic is asymmetric
--hdrh : Report latency using high dynamic range histograms (http://hdrhistogram.org)
--hops : If rx check is enabled, the hop number can be assigned. See manual for details
--iom : IO mode for server output [0- silent, 1- normal , 2- short]
--ipv6 : Work in ipv6 mode
-k : Run 'warm up' traffic for num seconds before starting the test.
Works only with the latency test (-l option)
-l : In parallel to the test, run latency check, sending packets at rate/sec from each interface
--l-pkt-mode <0-3> : Set mode for sending latency packets
0 (default) send SCTP packets
1 Send ICMP request packets
2 Send ICMP requests from client side, and response from server side (for working with firewall)
3 Send ICMP requests with sequence ID 0 from both sides
Rate of zero means no latency check
--learn (deprecated). Replaced by --learn-mode. To get older behaviour, use --learn-mode 2
--learn-mode [1-3] : Used for working in NAT environments. Dynamically learn the NAT translation done by the DUT
1 In case of TCP flow, use TCP ACK in first SYN to pass NAT translation information. Initial SYN packet must be first packet in the TCP flow
In case of UDP stream, NAT translation information will pass in IP ID field of first packet in flow. This means that this field is changed by TRex
2 Add special IP option to pass NAT translation information to first packet of each flow. Will not work on certain firewalls if they drop packets with IP options
3 Like 1, but without support for sequence number randomization in server->client direction. Performance (flow/second) better than 1
--learn-verify : Test the NAT translation mechanism. Should be used when there is no NAT in the setup
--limit-ports : Limit number of ports used. Must be even number (TRex always uses port pairs)
--lm : Hex mask of cores that should send traffic
For example: Value of 0x5 will cause only ports 0 and 2 to send traffic
--lo : Only run latency test
-m : Rate multiplier. Multiply basic rate of templates by this number
--mbuf-factor : Factor for packet memory
--nc : If set, will not wait for all flows to be closed, before terminating - see manual for more information
--no-flow-control-change : By default TRex disables flow-control. If this option is given, it does not touch it
--no-hw-flow-stat : Relevant only for Intel x710 stateless mode. Do not use HW counters for flow stats
: Enabling this will support lower traffic rate, but will also report RX byte count statistics. See manual for more details
--no-key : Daemon mode, don't get input from keyboard
--no-ofed-check : Disable the check of OFED version
--no-scapy-server : Disable Scapy server implicit start at stateless
--scapy-server : Enable Scapy server implicit start at ASTF
--no-termio : Do not use TERMIO. useful when using GDB and ctrl+c is needed.
--no-watchdog : Disable watchdog
--rt : Run TRex DP/RX cores in realtime priority
-p : Send all flow packets from the same interface (choosed randomly between client ad server ports) without changing their src/dst IP
-pm : Platform factor. If you have splitter in the setup, you can multiply the total results by this factor
e.g --pm 2.0 will multiply all the results bps in this factor
--prefix : For running multi TRex instances on the same machine. Each instance should have different name
--prom : Enable promiscuous for ASTF/STF mode
-pubd : Disable monitors publishers
--queue-drop : Do not retry to send packets on failure (queue full etc.)
--rpc-log : Save log of RPC conversation in logfile
--rx-check : Enable rx check. TRex will sample flows at 1/rate and check order, latency and more
-s : Single core. Run only one data path core. For debug
--send-debug-pkt : Do not run traffic generator. Just send debug packet and dump receive queues
Supported protocols are 1 for icmp, 2 for UDP, 3 for TCP, 4 for ARP, 5 for 9K UDP
--sleeps : Use sleeps instead of busy wait in scheduler (less accurate, more power saving)
--software : Do not configure any hardware rules. In this mode we use 1 core, and one RX queue and one TX queue per port
--unbind-unused-ports : Automatically unbind all unused bound ports in same NIC instead of exiting with error (i40e only)
--ignore-528-issue : Do not exit with error nor unbind them when other ports in the same NIC are kernel driven (i40e only),
: HW flow stats may be impaired and should not be used then [see https://trex-tgn.cisco.com/youtrack/issue/trex-528].
-v : The higher the value, print more debug information
--vlan : Relevant only for stateless mode with Intel 82599 10G NIC
When configuring flow stat and latency per stream rules, assume all streams uses VLAN
-w : Wait num seconds between init of interfaces and sending traffic, default is 1
--emu : Load emu server
--emu-zmq : For debug, just enable emu zmq channel.
--emu-zmq-tcp : Use TCP over ZMQ. Default is IPC.
--bird-server : Enable bird service
--gtpu : Enable GTPU mode
--disable-ieee-1588 : Enable Latency Measurement using HW timestamping and DPDK APIs. Currently works only for Stateless mode.
Need to Enable COMPILE time DPDK config RTE_LIBRTE_IEEE1588 inorder to use this feature
Uses PTP (IEEE 1588v2) Protocol to have the packets timestamped at NIC
Examples: basic trex run for 20 sec and multiplier of 10
t-rex-64 -f cap2/dns.yaml -m 10 -d 20
Copyright (c) 2015-2017 Cisco Systems, Inc.
Licensed under the Apache License, Version 2.0 (the 'License')
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Open Source Components / Libraries
DPDK (BSD)
YAML-CPP (BSD)
JSONCPP (MIT)
BPF (BSD)
HDR-HISTOGRAM-C (CC0)
Open Source Binaries
ZMQ (LGPL v3plus)
Version : v2.92
DPDK version : DPDK 21.02.0-rc1
User : hhaim
Date : Sep 15 2021 , 08:52:57
Uuid : d5a36f26-15e8-11ec-9af9-641225363740
Git SHA : e5521af766bbfad807eb8d185b6496eb25c65689
Compiled with GCC : 6.2.0
Compiled with glibc : 2.16 (host: 2.31)
Sanitized image : no
#
stateless mode
# ./t-rex-64 -i
WARNING: tried to configure 2048 hugepages for socket 0, but result is: 1425
Killing Scapy server... Scapy server is killed
Starting Scapy server........ Scapy server is started
Trying to bind to vfio-pci ...
Trying to compile and bind to igb_uio ...
ERROR: We don't have precompiled igb_uio.ko module for your kernel version.
Will try compiling automatically...
Success.
/usr/bin/python3 dpdk_nic_bind.py --bind=igb_uio 0000:00:08.0 0000:00:09.0
The ports are bound/configured.
Starting TRex v2.92 please wait ...
set driver name net_e1000_em
driver capability : TCP_UDP_OFFLOAD
set dpdk queues mode to ONE_QUE
Number of ports found: 2
zmq publisher at: tcp://*:4500
wait 1 sec .
port : 0
------------
link : link : Link Up - speed 1000 Mbps - full-duplex
promiscuous : 0
port : 1
------------
link : link : Link Up - speed 1000 Mbps - full-duplex
promiscuous : 0
number of ports : 2
max cores for 2 ports : 1
tx queues per port : 3
-------------------------------
RX core uses TX queue number 65535 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
------------------------------------------
1 0 0 1 0 0
-------------------------------
-Per port stats table
ports | 0 | 1
-----------------------------------------------------------------------------------------
opackets | 0 | 0
obytes | 0 | 0
ipackets | 0 | 0
ibytes | 0 | 0
ierrors | 0 | 0
oerrors | 0 | 0
Tx Bw | 0.00 bps | 0.00 bps
-Global stats enabled
Cpu Utilization : 0.0 %
Platform_factor : 1.0
Total-Tx : 0.00 bps
Total-Rx : 0.00 bps
Total-PPS : 0.00 pps
Total-CPS : 0.00 cps
Expected-PPS : 0.00 pps
Expected-CPS : 0.00 cps
Expected-BPS : 0.00 bps
Active-flows : 0 Clients : 0 Socket-util : 0.0000 %
Open-flows : 0 Servers : 0 Socket : 0 Socket/Clients : -nan
drop-rate : 0.00 bps
current time : 7.7 sec
test duration : 0.0 sec
*** TRex is shutting down - cause: 'CTRL + C detected'
All cores stopped !!
Killing Scapy server... Scapy server is killed
#