CSIT-234 IPFIX - ipv4 functional scale

 - add traffic script for classify session scale test
 - add keyword for classify session scale test
 - add packet scale and session scale test to IPFIX IPv4 suite

Change-Id: I3abd07dbf705fe664b723916ce1d82cd9649a2ae
Signed-off-by: selias <samelias@cisco.com>

3 files changed, 335 insertions, 0 deletions

diff --git a/resources/libraries/robot/ipfix.robot b/resources/libraries/robot/ipfix.robot
index a5adacfc0f..d8840a5261 100644
--- a/resources/libraries/robot/ipfix.robot
+++ b/resources/libraries/robot/ipfix.robot
@@ -65,3 +65,47 @@
 | | ... | ${args} --protocol ${protocol} --port ${port} --count ${count}
 | | Run Traffic Script On Node | ipfix_check.py | ${tg_node} | ${args}
 | | ... | ${timeout}
+
+| Send session sweep and verify IPFIX
+| | [Documentation] | Send simple TCP or UDP packets from source interface\
+| | ... | to destination interface using a range of source addresses. Listen\
+| | ... | for IPFIX flow report on source interface and verify received report\
+| | ... | against number of packets sent from each source address.
+| | ...
+| | ... | *Arguments:*
+| | ...
+| | ... | - tg_node - TG node. Type: dictionary
+| | ... | - dst_node - Destination node. Type: dictionary
+| | ... | - src_int - Source interface. Type: string
+| | ... | - dst_int - Destination interface. Type: string
+| | ... | - src_ip - Source IP address. Type: string
+| | ... | - dst_ip - Destination IP address. Type: string
+| | ... | - ip_range - Number of sequential source addresses. Type:integer
+| | ... | - protocol - TCP or UDP (Optional, defaults to TCP). Type: string
+| | ... | - port - Source and destination ports to use (Optional). Type: integer
+| | ... | - count - Number of packets to send (Optional). Type: integer
+| | ... | - timeout - Timeout value in seconds (optional). Type:integer
+| | ...
+| | ... | *Return:*
+| | ...
+| | ... | - No value returned
+| | ...
+| | ... | *Example:*
+| | ...
+| | ... | \| Send packets and verify IPFIX \| ${nodes['TG']} | ${nodes['DUT1']}\
+| | ... | \| eth1 \| GigabitEthernet0/8/0 \| 16.0.0.1 \| 192.168.0.2 \| 20 \|
+| | ... | UDP \| ${20} \| ${5} \| ${10} \|
+| | ... |
+| | [Arguments] | ${tg_node} | ${dst_node} | ${src_int} | ${dst_int} |
+| | ... | ${src_ip} | ${dst_ip} | ${ip_range} | ${protocol}=tcp | ${port}=20
+| | ... | ${count}=${1} | ${timeout}=${10}
+| | ${src_mac}= | Get Interface Mac | ${tg_node} | ${src_int}
+| | ${dst_mac}= | Set Variable | ${dut1_to_tg_mac}
+| | ${src_int_name}= | Get interface name | ${tg_node} | ${src_int}
+| | ${dst_int_name}= | Get interface name | ${dst_node} | ${dst_int}
+| | ${args}= | Traffic Script Gen Arg | ${dst_int_name} | ${src_int_name}
+| | ... | ${src_mac} | ${dst_mac} | ${src_ip} | ${dst_ip}
+| | ${args}= | Set Variable | ${args} --protocol ${protocol} --port ${port}
+| | ${args}= | Set Variable | ${args} --count ${count} --sessions ${ip_range}
+| | Run Traffic Script On Node | ipfix_sessions.py | ${tg_node} | ${args}
+| | ... | ${timeout}
\ No newline at end of file
diff --git a/resources/traffic_scripts/ipfix_sessions.py b/resources/traffic_scripts/ipfix_sessions.py
new file mode 100755
index 0000000000..385d21362b
--- /dev/null
+++ b/resources/traffic_scripts/ipfix_sessions.py
@@ -0,0 +1,225 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2016 Cisco and/or its affiliates.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Traffic script - IPFIX listener."""
+
+import sys
+from ipaddress import IPv4Address, IPv6Address, AddressValueError
+
+from scapy.layers.inet import IP, TCP, UDP
+from scapy.layers.inet6 import IPv6
+from scapy.layers.l2 import Ether
+
+from resources.libraries.python.IPFIXUtil import IPFIXHandler, IPFIXData
+from resources.libraries.python.PacketVerifier import RxQueue, TxQueue, auto_pad
+from resources.libraries.python.TrafficScriptArg import TrafficScriptArg
+
+
+def valid_ipv4(ip):
+    """Check if IP address has the correct IPv4 address format.
+
+    :param ip: IP address.
+    :type ip: str
+    :return: True in case of correct IPv4 address format,
+    otherwise return false.
+    :rtype: bool
+    """
+    try:
+        IPv4Address(unicode(ip))
+        return True
+    except (AttributeError, AddressValueError):
+        return False
+
+
+def valid_ipv6(ip):
+    """Check if IP address has the correct IPv6 address format.
+
+    :param ip: IP address.
+    :type ip: str
+    :return: True in case of correct IPv6 address format,
+    otherwise return false.
+    :rtype: bool
+    """
+    try:
+        IPv6Address(unicode(ip))
+        return True
+    except (AttributeError, AddressValueError):
+        return False
+
+
+def verify_data(data, count, src_ip, dst_ip, protocol):
+    """Compare data in IPFIX flow report against parameters used to send test
+     packets.
+
+     :param data: Dictionary of fields in IPFIX flow report.
+     :param count: Number of packets expected.
+     :param src_ip: Expected source IP address.
+     :param dst_ip: Expected destination IP address.
+     :param protocol: Expected protocol, TCP or UDP.
+     :type data: dict
+     :type count: int
+     :type src_ip: str
+     :type dst_ip: str
+     :type protocol: scapy.layers
+     """
+
+    # verify packet count
+    if data["packetTotalCount"] != count:
+        raise RuntimeError(
+            "IPFIX reported wrong packet count. Count was {0},"
+            " but should be {1}".format(data["packetTotalCount"], count))
+    # verify IP addresses
+    keys = data.keys()
+    e = "{0} mismatch. Packets used {1}, but were classified as {2}."
+    if valid_ipv4(src_ip) and valid_ipv4(dst_ip):
+        if "IPv4_src" in keys:
+            if data["IPv4_src"] != src_ip:
+                raise RuntimeError(
+                    e.format("Source IP", src_ip, data["IPv4_src"]))
+        if "IPv4_dst" in keys:
+            if data["IPv4_dst"] != dst_ip:
+                raise RuntimeError(
+                    e.format("Destination IP", dst_ip, data["IPv4_dst"]))
+    else:
+        if "IPv6_src" in keys:
+            if data["IPv6_src"] != src_ip:
+                raise RuntimeError(
+                    e.format("Source IP", src_ip, data["IPv6_src"]))
+        if "IPv6_dst" in keys:
+            if data["IPv6_dst"] != dst_ip:
+                raise RuntimeError(
+                    e.format("Source IP", src_ip, data["IPv6_dst"]))
+    # verify protocol ID
+    if "Protocol_ID" in keys:
+        if protocol == TCP and int(data["Protocol_ID"]) != 6:
+            raise RuntimeError(
+                "TCP Packets were classified as not TCP.")
+        if protocol == UDP and int(data["Protocol_ID"]) != 17:
+            raise RuntimeError(
+                "UDP Packets were classified as not UDP.")
+    # return port number
+    for item in ("src_port", "tcp_src_port", "udp_src_port",
+                 "dst_port", "tcp_dst_port", "udp_dst_port"):
+        if item in keys:
+            return int(data[item])
+        else:
+            raise RuntimeError("Data contains no port information.")
+
+
+def main():
+    """Send packets to VPP, then listen for IPFIX flow report. Verify that
+    the correct packet count was reported."""
+    args = TrafficScriptArg(
+        ['src_mac', 'dst_mac', 'src_ip', 'dst_ip', 'protocol', 'port', 'count',
+         'sessions']
+    )
+
+    dst_mac = args.get_arg('dst_mac')
+    src_mac = args.get_arg('src_mac')
+    src_ip = args.get_arg('src_ip')
+    dst_ip = args.get_arg('dst_ip')
+    tx_if = args.get_arg('tx_if')
+
+    protocol = args.get_arg('protocol')
+    source_port = int(args.get_arg('port'))
+    destination_port = int(args.get_arg('port'))
+    count = int(args.get_arg('count'))
+    sessions = int(args.get_arg('sessions'))
+
+    txq = TxQueue(tx_if)
+    rxq = RxQueue(tx_if)
+
+    # generate simple packet based on arguments
+    ip_version = None
+    if valid_ipv4(src_ip) and valid_ipv4(dst_ip):
+        ip_version = IP
+    elif valid_ipv6(src_ip) and valid_ipv6(dst_ip):
+        ip_version = IPv6
+    else:
+        ValueError("Invalid IP version!")
+
+    if protocol.upper() == 'TCP':
+        protocol = TCP
+    elif protocol.upper() == 'UDP':
+        protocol = UDP
+    else:
+        raise ValueError("Invalid type of protocol!")
+
+    packets = []
+    for x in range(sessions):
+        pkt = (Ether(src=src_mac, dst=dst_mac) /
+               ip_version(src=src_ip, dst=dst_ip) /
+               protocol(sport=x, dport=x))
+        pkt = auto_pad(pkt)
+        packets.append(pkt)
+
+    # do not print details for sent packets
+    verbose = False
+    print("Sending more than one packet. Details will be filtered for\
+    # all packets sent.")
+
+    ignore = []
+    for x in range(sessions):
+        for _ in range(count):
+            txq.send(packets[x], verbose=verbose)
+            ignore.append(packets[x])
+
+    # allow scapy to recognize IPFIX headers and templates
+    ipfix = IPFIXHandler()
+
+    # clear receive buffer
+    while True:
+        pkt = rxq.recv(1, ignore=packets, verbose=verbose)
+        if pkt is None:
+            break
+
+    data = None
+    ports = [x for x in range(sessions)]
+
+    # get IPFIX template and data
+    while True:
+        pkt = rxq.recv(5)
+        if pkt is None:
+            raise RuntimeError("RX timeout")
+        if pkt.haslayer("IPFIXHeader"):
+            if pkt.haslayer("IPFIXTemplate"):
+                # create or update template for IPFIX data packets
+                ipfix.update_template(pkt)
+            elif pkt.haslayer("IPFIXData"):
+                for x in range(sessions):
+                    try:
+                        data = pkt.getlayer(IPFIXData, x+1).fields
+                    except AttributeError:
+                        raise RuntimeError("Could not find data layer "
+                                           "#{0}".format(x+1))
+                    port = verify_data(data, count, src_ip, dst_ip, protocol)
+                    if port in ports:
+                        ports.remove(port)
+                    else:
+                        raise RuntimeError("Unexpected or duplicate port {0} "
+                                           "in flow report.".format(port))
+                print("All {0} sessions verified "
+                      "with packet count {1}.".format(sessions, count))
+                sys.exit(0)
+            else:
+                raise RuntimeError("Unable to parse IPFIX template "
+                                   "or data set.")
+        else:
+            raise RuntimeError("Received non-IPFIX packet or IPFIX header was"
+                               "not recognized.")
+
+if __name__ == "__main__":
+
+    main()
diff --git a/tests/func/ipfix/ipfix_ipv4.robot b/tests/func/ipfix/ipfix_ipv4.robot
index 3f2d753301..d0b01f6907 100644
--- a/tests/func/ipfix/ipfix_ipv4.robot
+++ b/tests/func/ipfix/ipfix_ipv4.robot
@@ -45,6 +45,7 @@
 | ${prefix_length}= | 24
 | ${ip_version}= | ip4
 | ${port}= | 80
+| ${sessions}= | 80
 
 *** Test Cases ***
 | TC01: DUT sends IPFIX template and data packets
@@ -202,4 +203,69 @@
 | | ... | ${tg_to_dut1} | ${dut1_to_tg} | ${tg_to_dut1_ip} | ${dut1_to_tg_ip}
 | | ... | port=${port}
 
+| TC06: DUT reports packet flow with a large number of packets
+| | [Documentation]
+| | ... | [Top] TG-DUT1-DUT2-TG. [Cfg] On DUT1 configure IPFIX with TG interface
+| | ... | address as collector and add classify session with TG source address.
+| | ... | [Ver] Make TG send packets to DUT1, then listen for IPFIX template
+| | ... | and data packets, verify that IPFIX reported the received packets.
+| | ... | [Ref] RFC 7011
+| | Given Path for 3-node testing is set
+| | ... | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['DUT2']} | ${nodes['TG']}
+| | And Interfaces in 3-node path are up
+| | And Set Interface Address | ${dut1_node}
+| | ... | ${dut1_to_tg} | ${dut1_to_tg_ip} | ${prefix_length}
+| | And Add ARP on DUT | ${dut1_node} | ${dut1_to_tg} | ${tg_to_dut1_ip}
+| | ... | ${tg_to_dut1_mac}
+| | ${table_index} | ${skip_n} | ${match_n}=
+| | ... | And VPP creates classify table L3 | ${dut1_node} | ${ip_version} | src
+| | And VPP configures classify session L3 | ${dut1_node} | permit
+| | ... | ${table_index} | ${skip_n} | ${match_n} | ${ip_version} | src
+| | ... | ${tg_to_dut1_ip}
+| | When Assign interface to flow table | ${dut1_node} | ${dut1_to_tg}
+| | ... | ${table_index} | ip_version=${ip_version}
+| | And setup IPFIX exporter | ${dut1_node} | ${tg_to_dut1_ip} | ${dut1_to_tg_ip}
+| | ... | interval=5
+| | And Set IPFIX stream | ${dut1_node} | ${1}
+| | And Assign classify table to exporter | ${dut1_node} | ${table_index}
+| | ... | ${ip_version}
+| | Then Send packets and verify IPFIX | ${tg_node} | ${dut1_node}
+| | ... | ${tg_to_dut1} | ${dut1_to_tg} | ${tg_to_dut1_ip} | ${dut1_to_tg_ip}
+| | ... | count=20000 | timeout=10
+
+| TC07: DUT reports packet flow when multiple sessions are configured
+| | [Documentation]
+| | ... | [Top] TG-DUT1-DUT2-TG. [Cfg] On DUT1 configure IPFIX with TG interface
+| | ... | address as collector and add several classify sessions with different
+| | ... | ports.
+| | ... | [Ver] Make TG send packets to DUT1 using a range of ports matching
+| | ... | configured sessions, then listen for IPFIX template and data packets,
+| | ... | verify that IPFIX reported the received packets for each session.
+| | ... | [Ref] RFC 7011
+| | Given Path for 3-node testing is set
+| | ... | ${nodes['TG']} | ${nodes['DUT1']} | ${nodes['DUT2']} | ${nodes['TG']}
+| | And Interfaces in 3-node path are up
+| | And Set Interface Address | ${dut1_node}
+| | ... | ${dut1_to_tg} | ${dut1_to_tg_ip} | ${prefix_length}
+| | And Add ARP on DUT | ${dut1_node} | ${dut1_to_tg} | ${tg_to_dut1_ip}
+| | ... | ${tg_to_dut1_mac}
+| | ${table_index} | ${skip_n} | ${match_n}=
+| | ... | And VPP creates classify table L3 | ${dut1_node} | ${ip_version}
+| | ... | src proto l4 src_port dst_port
+| | :FOR | ${index} | IN RANGE | ${sessions}
+| | | VPP configures classify session generic | ${dut1_node}
+| | | ... | acl-hit-next permit | ${table_index} | ${skip_n} | ${match_n}
+| | | ... | l3 ${ip_version} src ${tg_to_dut1_ip}
+| | | ... | proto 6 l4 src_port ${index} dst_port ${index}
+| | When Assign interface to flow table | ${dut1_node} | ${dut1_to_tg}
+| | ... | ${table_index} | ip_version=${ip_version}
+| | And setup IPFIX exporter | ${dut1_node} | ${tg_to_dut1_ip}
+| | ... | ${dut1_to_tg_ip}
+| | ... | mtu=1450 | interval=5
+| | And Set IPFIX stream | ${dut1_node} | ${1}
+| | And Assign classify table to exporter | ${dut1_node} | ${table_index}
+| | ... | ${ip_version}
+| | Then Send session sweep and verify IPFIX | ${tg_node} | ${dut1_node}
+| | ... | ${tg_to_dut1} | ${dut1_to_tg} | ${tg_to_dut1_ip} | ${dut1_to_tg_ip}
+| | ... | ${sessions} | timeout=10 | count=3
 # TODO: DUT reports packet flow when ACL is configured with wildcards