aboutsummaryrefslogtreecommitdiffstats
path: root/docs/cpta
AgeCommit message (Collapse)AuthorFilesLines
2022-01-31Trending: Add 2n-icxTibor Frank26-0/+880
Change-Id: I2d71c557f6b3c4d93466c096c9d5c0fabc36f486 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2022-01-28Doc: Add discussion about common anomaly patternsVratko Polak1-5/+136
Change-Id: Ic70c653cb660663abe48c2a0ae3e496a4e8c85f3 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-01-24Trending: Add awsTibor Frank4-0/+169
Change-Id: Ib571aef30424b84cf7c7abd7a8fee7b78f977b48 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2022-01-19Trending: Add NAT44Det tests for 2n-zn2Tibor Frank18-32/+80
Change-Id: I1068048590ff797b608f7578655fb9ce13f2ffe9 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2022-01-18Doc: Update anomaly methodology documentationVratko Polak1-20/+32
+ Define currently used compliance metrics (e.g. no short term change), + Clarify that (and why) column names in Dashboard and Alerting are somewhat different from the ones in the methodology document. Change-Id: Ie091b1981d7578245ca3e700820fab9ac93bbd17 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-01-18Trending: Unify anomaly related column namesVratko Polak1-6/+9
The most recent update merged was related to alerting, so this Change uses the new column names defined there also in Dashboard tables and legend. Change-Id: I548bc4b42af8237c4251433f3a46e98427898ea8 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2022-01-10Doc: Tweak trend analysis documentVratko Polak2-39/+75
Change-Id: Ia8de38cd20c9882727708686764f19c96d664eaf Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-12-03Trending: Fix 3n-tsh-x520-64b-memif-base-ixgbe graphsTibor Frank1-12/+5
Change-Id: Iec877a74701409c87f941b540494e1f313b32911 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-11-30Trending: Fix ethip4ipsecNspe tests for 2n-tx2Tibor Frank1-2/+2
Change-Id: I1cba1060d2d773d1209feb884ab46bbb7b1ada56 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-11-29Trending: Add ethip4ipsecNspe tests for 2n-tx2Tibor Frank1-8/+58
Change-Id: I59315e7ab10b6773cc17c72bae7e73774755efb5 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-10-21Trending: Add af_xdpTibor Frank32-25/+914
Change-Id: Ia9193dbe8a1975b2df42ddb2eeecd845050c9fb8 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-10-21Trending: Fix TRex graphs, remove VTSTibor Frank7-198/+1
Change-Id: I4c1ed83054d12b83f1a9a6a2d0081e7d037be2df Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-10-18Trending: Add TRex ndrpdr testsTibor Frank7-0/+232
Change-Id: I7b968a4747390224d89cdaf7407c10c9a4108deb Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-09-28Trending: Fix aws graphsTibor Frank1-0/+10
Change-Id: Icc078e45e0e7a243018bb9518dd4cfa83ea4c027 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-09-27Trending: Add aws test bedsTibor Frank6-0/+196
Change-Id: I0536d346277431e988155ca17e5891b9068cc074 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-09-03Trending: Fix linksTibor Frank3-5/+6
Change-Id: I8b18c3988435bced6dccd715ffdd790b2b385a61 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-08-05Trending: Fix broken graphsTibor Frank6-108/+0
Change-Id: I73b9446e9e76811f88d6b0fa2ee657e7ae0ceab3 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-08-04Trending: Add Latency trendingTibor Frank50-0/+1506
Change-Id: I28c1f1cf113d2537f3e564cdca0fd61ef57737a9 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-07-26Trending: Save latency data in csv filesTibor Frank2-15/+9
Change-Id: I397dc1711f52324c158c775522cfef3044724ad7 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-07-22Trending: Add graph with statisticsTibor Frank4-0/+218
- x-axis: time [MM-DD] - y-axis: nr of passed/failed tests - y-axis: build duration Change-Id: I269af5a728a32d02ffcdecc7e25138ad1a183c4b Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-06-03Trending: Partially remove 3n-hswTibor Frank4-61/+1
Change-Id: I2f08d455a51f3b891b38ad82bfac47dd7b9921c5 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-04-30Trending: Add 2n-tx2 cryptoTibor Frank2-0/+243
Change-Id: I1c0235a5a3799627ba759372fa1d3355aa2b7eea Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-04-13Trening: unhide cx556a testsTibor Frank10-31/+10
Change-Id: Ie5543ac0da678a10a2e62fc7be1b0b3f212c746b Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-02-25Trending: Fix file names fro nat44 tput graphsTibor Frank8-36/+36
Change-Id: I7e722b268595e8b1de370b3dde9d2b00c566f143 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-02-24Trending: Add nat44-tput testsTibor Frank15-1/+353
Change-Id: I0e670cfa4e0125856944656536f0d64d04bdbb6d Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-02-03Trending: Add IPv4 and IPv6 rnd testsTibor Frank10-7/+427
Change-Id: Ie90b88466575c33c2c45e8fb9c2b0c42dc795aa6 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-02-02Trending: Fix nat44 tests in NDRPDR trendingTibor Frank1-2/+6
Change-Id: Iab49c4e55d4cb3ad1e1c5399a406de6e9e8fbed1 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-01-26Trending: Add GENEVE testsTibor Frank4-0/+144
Change-Id: I5e20f6351dd3854eef075e1bf954dab5aee61698 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2021-01-21Trending: Add 2n-tx2Tibor Frank11-0/+379
Change-Id: Id2d0248128eee6a5b0969c7f27aae946ed47bbaa Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-11-30Trending: Fix links to NAT44 filesTibor Frank12-24/+24
Change-Id: Ia19cfa32430c7f12e0093d0b540d23876ca9da8a Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-11-30Trending: Fix filenamesTibor Frank23-0/+0
Change-Id: Idf754d624443ceef1db5ea5e1d4ab2a9f1c5649b Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-11-30Trending: Reorganize the NAT44 chaptersTibor Frank41-922/+1050
Change-Id: Ifec550ae69c6454d0adbc08d3e49435c3cca380f Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-11-20Trending: Add 2n-zn2Tibor Frank26-0/+1539
Change-Id: I49e2c50829af8f538155b6262ec2872a68560858 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-11-19Trending: Add NAT44 testsTibor Frank6-314/+774
Change-Id: I2601a786f58b1d89cc9a90cb16ff20e6331abbfe Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-15Trending: Add csit-dpdk-perf-mrr-weekly-master-3n-tshTibor Frank2-0/+51
Change-Id: Icce2fc809d3f72a6447cc07700df3c6f324af267 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-05Trending: Add NAT44Tibor Frank7-76/+476
Change-Id: I57d4b6f0ffa2ebfffbf95fc030bc8da70e46c8bc Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-24Trending: Replace snat by natTibor Frank2-14/+14
Change-Id: Ifcb1311d46b7134502b142933fd1574eb94cc230 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-23Trending: Fix graph file nameTibor Frank1-3/+3
2n-clx-xxv710-64b-8t4c-ip4-features-snat44det-avf.html Change-Id: I93b39f9edf3b0d85b08cec97ad647e398da83207 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-22Trending: Hide 2n-clx-cx556aTibor Frank10-10/+28
Change-Id: I3616ea486c96534534422057e2f80da486ded2b8 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-16Trending: Add snat44det tests to trending graphsTibor Frank2-0/+76
Change-Id: I7fe24c05a4f7f17355c3bf5c4f635f412656d52f Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-03Trending, Report: use test tag ethip4udp-snat44det-h1-p1-s1Tibor Frank1-0/+45
- use test tag ethip4udp-snat44det-h1-p1-s1 not ethip4udp-ip4base-nat44 Change-Id: Ie4897430c3b3074fdf3440d56cc1794a4c3046dd Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-08-17Trending: Fixes after reviewTibor Frank3-18/+18
Change-Id: Ia2fdd4d222c723c063f267a20e1a4f18b4e03d69 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-08-13Trending: NFV TestsTibor Frank13-99/+240
- 2n-skx - 2n-clx Change-Id: I6dd55ed167dc4267b2a4e4c52ec0874ad0e4a4b6 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-08-04Trending: Implement the latest changes in job specsTibor Frank76-2397/+1334
Graphs: - MRR - 2n-clx - 2n-skx - 3n-skx - 3n-hsw - 3n-tsh - NDRPDR - 2n-skx - 3n-skx - 3n-hsw - 2n-clx Change-Id: Ieab26ba1d2ee56e66c5c8598c4dba240a14e53e9 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-05-19Trending: CSIT-1717: Add links from dashboard to graphsTibor Frank6-0/+48
+ fix TB names in alerts Change-Id: Ia10e9b82b57e90f7ae4592bd6ecf8b71dea71cc8 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-05-15Trending: Add alerts for NDRPDR weekly testsTibor Frank4-17/+37
Change-Id: I35a806f8dd61e2c6d6dca2cdcfa4f7131e061cea Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-05-15Trending: Add NDRPDR trending graphsTibor Frank62-3/+4016
Change-Id: I57caf62b5084e36d8909330af68f766b50e3b748 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-05-05Trending: CSIT-1301: Display stdev in hover infoTibor Frank3-33/+72
+ make possible to use dashboard for ndrpdr tests + fixes in ndrpdr weekly comparison Change-Id: I846dc207ea620fad78424a445b32095160d4431d Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-04-28Trending: NDRPDR weekly comparisonTibor Frank2-0/+78
Change-Id: I6c5166e1fabfe7471af92a1423c0c89ca6b040a4 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-04-22Trending: Fixes in plots, dataTibor Frank2-2/+7
Change-Id: I91677a0f6e1c62633977c80fc8dec8019e580d0c Signed-off-by: Tibor Frank <tifrank@cisco.com>
"p">("Exception in scapy.rdpcap (%s): %s" % (self.out_path, format_exc())) return None before = len(output.res) if filter_out_fn: output.res = [p for p in output.res if not filter_out_fn(p)] removed = before - len(output.res) if removed: self.test.logger.debug( "Filtered out %s packets from capture (returning %s)" % (removed, len(output.res))) return output def get_capture(self, expected_count=None, remark=None, timeout=1, filter_out_fn=is_ipv6_misc): """ Get captured packets :param expected_count: expected number of packets to capture, if None, then self.test.packet_count_for_dst_pg_idx is used to lookup the expected count :param remark: remark printed into debug logs :param timeout: how long to wait for packets :param filter_out_fn: filter applied to each packet, packets for which the filter returns True are removed from capture :returns: iterable packets """ remaining_time = timeout capture = None name = self.name if remark is None else "%s (%s)" % (self.name, remark) based_on = "based on provided argument" if expected_count is None: expected_count = \ self.test.get_packet_count_for_if_idx(self.sw_if_index) based_on = "based on stored packet_infos" if expected_count == 0: raise Exception( "Internal error, expected packet count for %s is 0!" % name) self.test.logger.debug("Expecting to capture %s (%s) packets on %s" % ( expected_count, based_on, name)) while remaining_time > 0: before = time.time() capture = self._get_capture(remaining_time, filter_out_fn) elapsed_time = time.time() - before if capture: if len(capture.res) == expected_count: # bingo, got the packets we expected return capture elif len(capture.res) > expected_count: self.test.logger.error( ppc("Unexpected packets captured:", capture)) break else: self.test.logger.debug("Partial capture containing %s " "packets doesn't match expected " "count %s (yet?)" % (len(capture.res), expected_count)) elif expected_count == 0: # bingo, got None as we expected - return empty capture return PacketList() remaining_time -= elapsed_time if capture: self.generate_debug_aid("count-mismatch") raise Exception("Captured packets mismatch, captured %s packets, " "expected %s packets on %s" % (len(capture.res), expected_count, name)) else: raise Exception("No packets captured on %s" % name) def assert_nothing_captured(self, remark=None, filter_out_fn=is_ipv6_misc): """ Assert that nothing unfiltered was captured on interface :param remark: remark printed into debug logs :param filter_out_fn: filter applied to each packet, packets for which the filter returns True are removed from capture """ if os.path.isfile(self.out_path): try: capture = self.get_capture( 0, remark=remark, filter_out_fn=filter_out_fn) if not capture or len(capture.res) == 0: # junk filtered out, we're good return except: pass self.generate_debug_aid("empty-assert") if remark: raise AssertionError( "Non-empty capture file present for interface %s (%s)" % (self.name, remark)) else: raise AssertionError("Capture file present for interface %s" % self.name) def wait_for_capture_file(self, timeout=1): """ Wait until pcap capture file appears :param timeout: How long to wait for the packet (default 1s) :returns: True/False if the file is present or appears within timeout """ deadline = time.time() + timeout if not os.path.isfile(self.out_path): self.test.logger.debug("Waiting for capture file %s to appear, " "timeout is %ss" % (self.out_path, timeout)) else: self.test.logger.debug("Capture file %s already exists" % self.out_path) return True while time.time() < deadline: if os.path.isfile(self.out_path): break time.sleep(0) # yield if os.path.isfile(self.out_path): self.test.logger.debug("Capture file appeared after %fs" % (time.time() - (deadline - timeout))) else: self.test.logger.debug("Timeout - capture file still nowhere") return False return True def verify_enough_packet_data_in_pcap(self): """ Check if enough data is available in file handled by internal pcap reader so that a whole packet can be read. :returns: True if enough data present, else False """ orig_pos = self._pcap_reader.f.tell() # save file position enough_data = False # read packet header from pcap packet_header_size = 16 caplen = None end_pos = None hdr = self._pcap_reader.f.read(packet_header_size) if len(hdr) == packet_header_size: # parse the capture length - caplen sec, usec, caplen, wirelen = struct.unpack( self._pcap_reader.endian + "IIII", hdr) self._pcap_reader.f.seek(0, 2) # seek to end of file end_pos = self._pcap_reader.f.tell() # get position at end if end_pos >= orig_pos + len(hdr) + caplen: enough_data = True # yay, we have enough data self._pcap_reader.f.seek(orig_pos, 0) # restore original position return enough_data def wait_for_packet(self, timeout, filter_out_fn=is_ipv6_misc): """ Wait for next packet captured with a timeout :param timeout: How long to wait for the packet :returns: Captured packet if no packet arrived within timeout :raises Exception: if no packet arrives within timeout """ deadline = time.time() + timeout if self._pcap_reader is None: if not self.wait_for_capture_file(timeout): raise CaptureTimeoutError("Capture file %s did not appear " "within timeout" % self.out_path) while time.time() < deadline: try: self._pcap_reader = PcapReader(self.out_path) break except: self.test.logger.debug( "Exception in scapy.PcapReader(%s): %s" % (self.out_path, format_exc())) if not self._pcap_reader: raise CaptureTimeoutError("Capture file %s did not appear within " "timeout" % self.out_path) poll = False if timeout > 0: self.test.logger.debug("Waiting for packet") else: poll = True self.test.logger.debug("Polling for packet") while time.time() < deadline or poll: if not self.verify_enough_packet_data_in_pcap(): time.sleep(0) # yield poll = False continue p = self._pcap_reader.recv() if p is not None: if filter_out_fn is not None and filter_out_fn(p): self.test.logger.debug( "Packet received after %ss was filtered out" % (time.time() - (deadline - timeout))) else: self.test.logger.debug( "Packet received after %fs" % (time.time() - (deadline - timeout))) return p time.sleep(0) # yield poll = False self.test.logger.debug("Timeout - no packets received") raise CaptureTimeoutError("Packet didn't arrive within timeout") def create_arp_req(self): """Create ARP request applicable for this interface""" return (Ether(dst="ff:ff:ff:ff:ff:ff", src=self.remote_mac) / ARP(op=ARP.who_has, pdst=self.local_ip4, psrc=self.remote_ip4, hwsrc=self.remote_mac)) def create_ndp_req(self): """Create NDP - NS applicable for this interface""" nsma = in6_getnsma(inet_pton(socket.AF_INET6, self.local_ip6)) d = inet_ntop(socket.AF_INET6, nsma) return (Ether(dst=in6_getnsmac(nsma)) / IPv6(dst=d, src=self.remote_ip6) / ICMPv6ND_NS(tgt=self.local_ip6) / ICMPv6NDOptSrcLLAddr(lladdr=self.remote_mac)) def resolve_arp(self, pg_interface=None): """Resolve ARP using provided packet-generator interface :param pg_interface: interface used to resolve, if None then this interface is used """ if pg_interface is None: pg_interface = self self.test.logger.info("Sending ARP request for %s on port %s" % (self.local_ip4, pg_interface.name)) arp_req = self.create_arp_req() pg_interface.add_stream(arp_req) pg_interface.enable_capture() self.test.pg_start() self.test.logger.info(self.test.vapi.cli("show trace")) try: captured_packet = pg_interface.wait_for_packet(1) except: self.test.logger.info("No ARP received on port %s" % pg_interface.name) return arp_reply = captured_packet.copy() # keep original for exception # Make Dot1AD packet content recognizable to scapy if arp_reply.type == 0x88a8: arp_reply.type = 0x8100 arp_reply = Ether(str(arp_reply)) try: if arp_reply[ARP].op == ARP.is_at: self.test.logger.info("VPP %s MAC address is %s " % (self.name, arp_reply[ARP].hwsrc)) self._local_mac = arp_reply[ARP].hwsrc else: self.test.logger.info("No ARP received on port %s" % pg_interface.name) except: self.test.logger.error( ppp("Unexpected response to ARP request:", captured_packet)) raise def resolve_ndp(self, pg_interface=None, timeout=1): """Resolve NDP using provided packet-generator interface :param pg_interface: interface used to resolve, if None then this interface is used :param timeout: how long to wait for response before giving up """ if pg_interface is None: pg_interface = self self.test.logger.info("Sending NDP request for %s on port %s" % (self.local_ip6, pg_interface.name)) ndp_req = self.create_ndp_req() pg_interface.add_stream(ndp_req) pg_interface.enable_capture() self.test.pg_start() now = time.time() deadline = now + timeout # Enabling IPv6 on an interface can generate more than the # ND reply we are looking for (namely MLD). So loop through # the replies to look for want we want. while now < deadline: try: captured_packet = pg_interface.wait_for_packet( deadline - now, filter_out_fn=None) except: self.test.logger.error( "Timeout while waiting for NDP response") raise ndp_reply = captured_packet.copy() # keep original for exception # Make Dot1AD packet content recognizable to scapy if ndp_reply.type == 0x88a8: ndp_reply.type = 0x8100 ndp_reply = Ether(str(ndp_reply)) try: ndp_na = ndp_reply[ICMPv6ND_NA] opt = ndp_na[ICMPv6NDOptDstLLAddr] self.test.logger.info("VPP %s MAC address is %s " % (self.name, opt.lladdr)) self._local_mac = opt.lladdr self.test.logger.debug(self.test.vapi.cli("show trace")) # we now have the MAC we've been after return except: self.test.logger.info( ppp("Unexpected response to NDP request:", captured_packet)) now = time.time() self.test.logger.debug(self.test.vapi.cli("show trace")) raise Exception("Timeout while waiting for NDP response")