How to do VPP Packet Tracing in Kubernetes ========================================== This document describes the steps to do *manual* packet tracing (capture) using VPP in Kubernetes. Contiv/VPP also ships with a simple bash script `vpptrace.sh `__, which allows to *continuously* trace and *filter* packets incoming through a given set of interface types. Documentation for vpptrace.sh is available `here `__. More information about VPP packet tracing is in: - https://wiki.fd.io/view/VPP/Command-line_Interface_(CLI)_Guide#packet_tracer - https://wiki.fd.io/view/VPP/How_To_Use_The_Packet_Generator_and_Packet_Tracer - https://wiki.fd.io/view/VPP/Tutorial_Routing_and_Switching SSH into the Node ----------------- Perform the following commands to SSH into the node: :: cd vpp/vagrant/vagrant-scripts/ vagrant ssh k8s-worker1 Check the VPP Graph Nodes (Input and Output Queues) --------------------------------------------------- The following content shows what is running on VPP, via the ``show run`` command :: vagrant@k8s-worker1:~$ sudo vppctl _______ _ _ _____ ___ __/ __/ _ \ (_)__ | | / / _ \/ _ \ _/ _// // / / / _ \ | |/ / ___/ ___/ /_/ /____(_)_/\___/ |___/_/ /_/ vpp# show run Time 1026791.9, average vectors/node 1.12, last 128 main loops 0.00 per node 0.00 vector rates in 1.6459e-4, out 1.5485e-4, drop 1.3635e-5, punt 0.0000e0 Name State Calls Vectors Suspends Clocks Vectors/Call GigabitEthernet0/8/0-output active 56 69 0 1.34e3 1.23 GigabitEthernet0/8/0-tx active 54 67 0 8.09e5 1.24 acl-plugin-fa-cleaner-process event wait 0 0 1 2.84e4 0.00 admin-up-down-process event wait 0 0 1 4.59e3 0.00 api-rx-from-ring any wait 0 0 3316292 1.24e5 0.00 arp-input active 3 3 0 2.53e5 1.00 bfd-process event wait 0 0 1 5.94e3 0.00 cdp-process any wait 0 0 145916 1.36e4 0.00 dhcp-client-process any wait 0 0 10268 3.65e4 0.00 dns-resolver-process any wait 0 0 1027 5.86e4 0.00 dpdk-input polling 8211032318951 93 0 1.48e13 0.00 dpdk-ipsec-process done 1 0 0 2.10e5 0.00 dpdk-process any wait 0 0 342233 9.86e6 0.00 error-drop active 12 14 0 6.67e3 1.17 ethernet-input active 60 74 0 5.81e3 1.23 fib-walk any wait 0 0 513322 1.59e4 0.00 flow-report-process any wait 0 0 1 1.45e3 0.00 flowprobe-timer-process any wait 0 0 1 6.34e3 0.00 ikev2-manager-process any wait 0 0 1026484 1.18e4 0.00 interface-output active 2 2 0 3.23e3 1.00 ioam-export-process any wait 0 0 1 1.98e3 0.00 ip-route-resolver-process any wait 0 0 10268 3.02e4 0.00 ip4-arp active 1 1 0 1.49e4 1.00 ip4-input active 223 248 0 3.39e3 1.11 ip4-load-balance active 106 132 0 5.34e3 1.25 ip4-local active 86 92 0 2.46e3 1.07 ip4-local-end-of-arc active 86 92 0 1.00e3 1.07 ip4-lookup active 223 248 0 3.31e3 1.11 ip4-rewrite active 190 222 0 1.92e3 1.17 ip4-udp-lookup active 86 92 0 3.76e3 1.07 ip6-drop active 6 7 0 2.29e3 1.17 ip6-icmp-neighbor-discovery-ev any wait 0 0 1026484 1.13e4 0.00 ip6-input active 6 7 0 3.33e3 1.17 l2-flood active 2 2 0 4.42e3 1.00 l2-fwd active 138 157 0 2.13e3 1.14 l2-input active 140 159 0 2.41e3 1.14 l2-learn active 86 92 0 3.64e4 1.07 l2-output active 54 67 0 3.05e3 1.24 l2fib-mac-age-scanner-process event wait 0 0 85 5.01e4 0.00 lisp-retry-service any wait 0 0 513322 1.62e4 0.00 lldp-process event wait 0 0 1 5.02e4 0.00 loop0-output active 54 67 0 1.66e3 1.24 loop0-tx active 54 0 0 2.49e3 0.00 memif-process event wait 0 0 1 1.70e4 0.00 nat-det-expire-walk done 1 0 0 3.79e3 0.00 nat44-classify active 171 183 0 2.49e3 1.07 nat44-hairpinning active 86 92 0 1.80e3 1.07 nat44-in2out active 171 183 0 4.45e3 1.07 nat44-in2out-slowpath active 171 183 0 3.98e3 1.07 nat44-out2in active 52 65 0 1.28e4 1.25 nat64-expire-walk any wait 0 0 102677 5.95e4 0.00 nat64-expire-worker-walk interrupt wa 102676 0 0 7.39e3 0.00 send-garp-na-process event wait 0 0 1 1.28e3 0.00 startup-config-process done 1 0 1 4.19e3 0.00 tapcli-0-output active 1 1 0 6.97e3 1.00 tapcli-0-tx active 1 1 0 7.32e4 1.00 tapcli-1-output active 57 63 0 1.66e3 1.11 tapcli-1-tx active 57 63 0 1.35e5 1.11 tapcli-2-output active 28 28 0 3.26e3 1.00 tapcli-2-tx active 28 28 0 4.06e5 1.00 tapcli-rx interrupt wa 62 76 0 6.58e4 1.23 udp-ping-process any wait 0 0 1 1.79e4 0.00 unix-cli-127.0.0.1:43282 active 2 0 455 1.26e15 0.00 unix-epoll-input polling 8010763239 0 0 8.17e2 0.00 vhost-user-process any wait 0 0 1 1.96e3 0.00 vhost-user-send-interrupt-proc any wait 0 0 1 3.85e3 0.00 vpe-link-state-process event wait 0 0 8 9.79e4 0.00 vpe-oam-process any wait 0 0 503263 1.21e4 0.00 vxlan-gpe-ioam-export-process any wait 0 0 1 2.91e3 0.00 vxlan4-encap active 54 67 0 3.55e3 1.24 vxlan4-input active 86 92 0 3.79e3 1.07 wildcard-ip4-arp-publisher-pro event wait 0 0 1 6.44e3 0.00 ``tapcli-rx`` above is the node-level input queue for incoming packets into all the pods on the node. There is one ``tapcli-rx`` input queue for every node. The following are the input and output queues for each pod and the node: :: tapcli-0-output tapcli-0-tx tapcli-1-output tapcli-1-tx tapcli-2-output tapcli-2-tx Each pod and node has two queues, one for rx (``tapcli-X-output``), and one for tx (``tapcli-X-tx``). The above output is with two ``nginx`` pods in kubernetes. Clear Existing VPP Packet Trace ------------------------------- Enter the following command: :: vpp# clear trace How to Turn on VPP Packet Tracing --------------------------------- Enter the following commands: :: vpp# trace add vpp# trace add dpdk-input 1000 vpp# trace add tapcli-rx 1000 Send Traffic to the Pods ------------------------ Open another terminal, SSH into the master node, refer the documentation in ``vpp/vagrant/README.md`` and send traffic to the two ``nginx`` pods using ``wget``. :: cd vpp/vagrant/vagrant-scripts/ vagrant ssh k8s-master vagrant@k8s-master:~$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE nginx-8586cf59-768qw 1/1 Running 0 11d 10.1.2.3 k8s-worker1 nginx-8586cf59-d27h2 1/1 Running 0 11d 10.1.2.2 k8s-worker1 vagrant@k8s-master:~$ wget 10.1.2.2 --2018-02-08 16:46:01-- http://10.1.2.2/ Connecting to 10.1.2.2:80... connected. HTTP request sent, awaiting response... 200 OK Length: 612 [text/html] Saving to: ‘index.html’ index.html 100%[=========================================================>] 612 --.-KB/s in 0.004s 2018-02-08 16:46:01 (162 KB/s) - ‘index.html’ saved [612/612] vagrant@k8s-master:~$ wget 10.1.2.3 --2018-02-08 16:46:02-- http://10.1.2.3/ Connecting to 10.1.2.3:80... connected. HTTP request sent, awaiting response... 200 OK Length: 612 [text/html] Saving to: ‘index.html.1’ index.html.1 100%[=========================================================>] 612 --.-KB/s in 0.004s 2018-02-08 16:46:02 (143 KB/s) - ‘index.html.1’ saved [612/612] Check the Packets Captured by VPP --------------------------------- Back in the first terminal, check the packets captured by VPP. :: vpp# show trace ... ... Packet 33 21:34:51:476110: tapcli-rx tapcli-2 21:34:51:476115: ethernet-input IP4: 00:00:00:00:00:02 -> 02:fe:72:95:66:c7 21:34:51:476117: ip4-input TCP: 10.1.2.3 -> 172.30.1.2 tos 0x00, ttl 64, length 52, checksum 0x6fb4 fragment id 0x11ec, flags DONT_FRAGMENT TCP: 80 -> 58430 seq. 0x5db741c8 ack 0x709defa7 flags 0x11 FIN ACK, tcp header: 32 bytes window 235, checksum 0x55c3 21:34:51:476118: nat44-out2in NAT44_OUT2IN: sw_if_index 6, next index 1, session index -1 21:34:51:476120: ip4-lookup fib 0 dpo-idx 23 flow hash: 0x00000000 TCP: 10.1.2.3 -> 172.30.1.2 tos 0x00, ttl 64, length 52, checksum 0x6fb4 fragment id 0x11ec, flags DONT_FRAGMENT TCP: 80 -> 58430 seq. 0x5db741c8 ack 0x709defa7 flags 0x11 FIN ACK, tcp header: 32 bytes window 235, checksum 0x55c3 21:34:51:476121: ip4-load-balance fib 0 dpo-idx 23 flow hash: 0x00000000 TCP: 10.1.2.3 -> 172.30.1.2 tos 0x00, ttl 64, length 52, checksum 0x6fb4 fragment id 0x11ec, flags DONT_FRAGMENT TCP: 80 -> 58430 seq. 0x5db741c8 ack 0x709defa7 flags 0x11 FIN ACK, tcp header: 32 bytes window 235, checksum 0x55c3 21:34:51:476122: ip4-rewrite tx_sw_if_index 3 dpo-idx 5 : ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800 flow hash: 0x00000000 00000000: 1a2b3c4d5e011a2b3c4d5e0208004500003411ec40003f0670b40a010203ac1e 00000020: 01020050e43e5db741c8709defa7801100eb55c300000101080a0f4b 21:34:51:476123: loop0-output loop0 IP4: 1a:2b:3c:4d:5e:02 -> 1a:2b:3c:4d:5e:01 TCP: 10.1.2.3 -> 172.30.1.2 tos 0x00, ttl 63, length 52, checksum 0x70b4 fragment id 0x11ec, flags DONT_FRAGMENT TCP: 80 -> 58430 seq. 0x5db741c8 ack 0x709defa7 flags 0x11 FIN ACK, tcp header: 32 bytes window 235, checksum 0x55c3 21:34:51:476124: l2-input l2-input: sw_if_index 3 dst 1a:2b:3c:4d:5e:01 src 1a:2b:3c:4d:5e:02 21:34:51:476125: l2-fwd l2-fwd: sw_if_index 3 dst 1a:2b:3c:4d:5e:01 src 1a:2b:3c:4d:5e:02 bd_index 1 21:34:51:476125: l2-output l2-output: sw_if_index 4 dst 1a:2b:3c:4d:5e:01 src 1a:2b:3c:4d:5e:02 data 08 00 45 00 00 34 11 ec 40 00 3f 06 21:34:51:476126: vxlan4-encap VXLAN encap to vxlan_tunnel0 vni 10 21:34:51:476126: ip4-load-balance fib 4 dpo-idx 22 flow hash: 0x00000103 UDP: 192.168.16.2 -> 192.168.16.1 tos 0x00, ttl 254, length 102, checksum 0x1b33 fragment id 0x0000 UDP: 24320 -> 4789 length 82, checksum 0x0000 21:34:51:476127: ip4-rewrite tx_sw_if_index 1 dpo-idx 4 : ipv4 via 192.168.16.1 GigabitEthernet0/8/0: 080027b2610908002733fb6f0800 flow hash: 0x00000103 00000000: 080027b2610908002733fb6f08004500006600000000fd111c33c0a81002c0a8 00000020: 10015f0012b5005200000800000000000a001a2b3c4d5e011a2b3c4d 21:34:51:476127: GigabitEthernet0/8/0-output GigabitEthernet0/8/0 IP4: 08:00:27:33:fb:6f -> 08:00:27:b2:61:09 UDP: 192.168.16.2 -> 192.168.16.1 tos 0x00, ttl 253, length 102, checksum 0x1c33 fragment id 0x0000 UDP: 24320 -> 4789 length 82, checksum 0x0000 21:34:51:476128: GigabitEthernet0/8/0-tx GigabitEthernet0/8/0 tx queue 0 buffer 0xfa7f: current data -50, length 116, free-list 0, clone-count 0, totlen-nifb 0, trace 0x20 l2-hdr-offset 0 l3-hdr-offset 14 PKT MBUF: port 255, nb_segs 1, pkt_len 116 buf_len 2176, data_len 116, ol_flags 0x0, data_off 78, phys_addr 0x569ea040 packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0 IP4: 08:00:27:33:fb:6f -> 08:00:27:b2:61:09 UDP: 192.168.16.2 -> 192.168.16.1 tos 0x00, ttl 253, length 102, checksum 0x1c33 fragment id 0x0000 UDP: 24320 -> 4789 length 82, checksum 0x0000 In the above captured packet, we can see: - Input queue name ``tapcli-rx`` - Pod’s IP address ``10.1.2.3`` - IP address of the master node ``172.30.1.2``, which sent the ``wget`` traffic to the two pods - HTTP port ``80``, destination port and TCP protocol (``TCP: 80 -> 58430``) - NAT queue name ``nat44-out2in`` - VXLAN VNI ID ``VXLAN encap to vxlan_tunnel0 vni 10`` - VXLAN UDP port ``4789`` - IP address of ``GigabitEthernet0/8/0`` interface (``192.168.16.2``) - Packet on the outgoing queue ``GigabitEthernet0/8/0-tx`` Find IP Addresses of GigabitEthernet and the Tap Interfaces ----------------------------------------------------------- Enter the following commands to find the IP addresses and Tap interfaces: :: vpp# show int address GigabitEthernet0/8/0 (up): L3 192.168.16.2/24 local0 (dn): loop0 (up): L2 bridge bd-id 1 idx 1 shg 0 bvi L3 192.168.30.2/24 tapcli-0 (up): L3 172.30.2.1/24 tapcli-1 (up): L3 10.2.1.2/32 tapcli-2 (up): L3 10.2.1.3/32 vxlan_tunnel0 (up): L2 bridge bd-id 1 idx 1 shg 0 Other Useful VPP CLIs --------------------- Enter the following commands to see additional information about VPP: :: vpp# show int Name Idx State Counter Count GigabitEthernet0/8/0 1 up rx packets 138 rx bytes 18681 tx packets 100 tx bytes 29658 drops 1 ip4 137 tx-error 2 local0 0 down drops 1 loop0 3 up rx packets 137 rx bytes 9853 tx packets 200 tx bytes 49380 drops 1 ip4 136 tapcli-0 2 up rx packets 8 rx bytes 600 tx packets 1 tx bytes 42 drops 9 ip6 7 tapcli-1 5 up rx packets 56 rx bytes 13746 tx packets 78 tx bytes 6733 drops 1 ip4 56 tapcli-2 6 up rx packets 42 rx bytes 10860 tx packets 58 tx bytes 4996 drops 1 ip4 42 vxlan_tunnel0 4 up rx packets 137 rx bytes 11771 tx packets 100 tx bytes 28290 vpp# show hardware Name Idx Link Hardware GigabitEthernet0/8/0 1 up GigabitEthernet0/8/0 Ethernet address 08:00:27:33:fb:6f Intel 82540EM (e1000) carrier up full duplex speed 1000 mtu 9216 rx queues 1, rx desc 1024, tx queues 1, tx desc 1024 cpu socket 0 tx frames ok 100 tx bytes ok 29658 rx frames ok 138 rx bytes ok 19233 extended stats: rx good packets 138 tx good packets 100 rx good bytes 19233 tx good bytes 29658 local0 0 down local0 local loop0 3 up loop0 Ethernet address 1a:2b:3c:4d:5e:02 tapcli-0 2 up tapcli-0 Ethernet address 02:fe:95:07:df:9c tapcli-1 5 up tapcli-1 Ethernet address 02:fe:3f:5f:0f:9a tapcli-2 6 up tapcli-2 Ethernet address 02:fe:72:95:66:c7 vxlan_tunnel0 4 up vxlan_tunnel0 VXLAN vpp# show bridge-domain BD-ID Index BSN Age(min) Learning U-Forwrd UU-Flood Flooding ARP-Term BVI-Intf 1 1 1 off on on on on off loop0 vpp# show bridge-domain 1 detail BD-ID Index BSN Age(min) Learning U-Forwrd UU-Flood Flooding ARP-Term BVI-Intf 1 1 1 off on on on on off loop0 Interface If-idx ISN SHG BVI TxFlood VLAN-Tag-Rewrite loop0 3 3 0 * * none vxlan_tunnel0 4 1 0 - * none vpp# show l2fib verbose Mac-Address BD-Idx If-Idx BSN-ISN Age(min) static filter bvi Interface-Name 1a:2b:3c:4d:5e:02 1 3 0/0 - * - * loop0 1a:2b:3c:4d:5e:01 1 4 1/1 - - - - vxlan_tunnel0 L2FIB total/learned entries: 2/1 Last scan time: 0.0000e0sec Learn limit: 4194304 vpp# show ip fib ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] locks:[src:(nil):2, src:adjacency:3, src:default-route:1, ] 0.0.0.0/0 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:21 to:[0:0]] [0] [@5]: ipv4 via 172.30.2.2 tapcli-0: def35b93961902fe9507df9c0800 0.0.0.0/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]] [0] [@0]: dpo-drop ip4 10.1.1.0/24 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:24 buckets:1 uRPF:29 to:[0:0]] [0] [@10]: dpo-load-balance: [proto:ip4 index:23 buckets:1 uRPF:28 to:[0:0] via:[98:23234]] [0] [@5]: ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800 10.1.2.2/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:27 buckets:1 uRPF:12 to:[78:5641]] [0] [@5]: ipv4 via 10.1.2.2 tapcli-1: 00000000000202fe3f5f0f9a0800 10.1.2.3/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:29 buckets:1 uRPF:32 to:[58:4184]] [0] [@5]: ipv4 via 10.1.2.3 tapcli-2: 00000000000202fe729566c70800 10.2.1.2/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:26 buckets:1 uRPF:31 to:[0:0]] [0] [@2]: dpo-receive: 10.2.1.2 on tapcli-1 10.2.1.3/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:28 buckets:1 uRPF:33 to:[0:0]] [0] [@2]: dpo-receive: 10.2.1.3 on tapcli-2 172.30.1.0/24 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:25 buckets:1 uRPF:29 to:[98:23234]] [0] [@10]: dpo-load-balance: [proto:ip4 index:23 buckets:1 uRPF:28 to:[0:0] via:[98:23234]] [0] [@5]: ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800 172.30.2.0/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:15 to:[0:0]] [0] [@0]: dpo-drop ip4 172.30.2.0/24 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:14 to:[0:0]] [0] [@4]: ipv4-glean: tapcli-0 172.30.2.1/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:16 buckets:1 uRPF:19 to:[0:0]] [0] [@2]: dpo-receive: 172.30.2.1 on tapcli-0 172.30.2.2/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:17 buckets:1 uRPF:18 to:[0:0]] [0] [@5]: ipv4 via 172.30.2.2 tapcli-0: def35b93961902fe9507df9c0800 172.30.2.255/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:17 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.16.0/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.16.1/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:22 buckets:1 uRPF:34 to:[0:0] via:[100:28290]] [0] [@5]: ipv4 via 192.168.16.1 GigabitEthernet0/8/0: 080027b2610908002733fb6f0800 192.168.16.0/24 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:30 to:[0:0]] [0] [@4]: ipv4-glean: GigabitEthernet0/8/0 192.168.16.2/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[137:16703]] [0] [@2]: dpo-receive: 192.168.16.2 on GigabitEthernet0/8/0 192.168.16.255/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:11 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.30.0/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:19 buckets:1 uRPF:23 to:[0:0]] [0] [@0]: dpo-drop ip4 192.168.30.1/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:23 buckets:1 uRPF:28 to:[0:0] via:[98:23234]] [0] [@5]: ipv4 via 192.168.30.1 loop0: 1a2b3c4d5e011a2b3c4d5e020800 192.168.30.0/24 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:18 buckets:1 uRPF:22 to:[0:0]] [0] [@4]: ipv4-glean: loop0 192.168.30.2/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:21 buckets:1 uRPF:27 to:[0:0]] [0] [@2]: dpo-receive: 192.168.30.2 on loop0 192.168.30.255/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:20 buckets:1 uRPF:25 to:[0:0]] [0] [@0]: dpo-drop ip4 224.0.0.0/4 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]] [0] [@0]: dpo-drop ip4 240.0.0.0/4 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]] [0] [@0]: dpo-drop ip4 255.255.255.255/32 unicast-ip4-chain [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]] [0] [@0]: dpo-drop ip4