aboutsummaryrefslogtreecommitdiffstats
path: root/docs/report/introduction/introduction.rst
blob: c15df8d9bc3e10b2174c514584e38efff590c365 (plain) 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141

Report Structure
================

FD.io |csit-release| report contains system performance and functional
testing data of |vpp-release|. `PDF version of this report`_ is
available for download.

|csit-release| report is structured as follows:

#. INTRODUCTION: General introduction to FD.io |csit-release|.

   - **Introduction**: This section.
   - **Test Scenarios Overview**: A brief overview of test scenarios
     covered in this report.
   - **Physical Testbeds**: Description of physical testbeds.
   - **Test Methodology**: Performance benchmarking and functional test
     methodologies.

#. VPP PERFORMANCE: VPP performance tests executed in physical
   FD.io testbeds.

   - **Overview**: Tested logical topologies, test coverage and naming
     specifics.
   - **Release Notes**: Changes in |csit-release|, added tests,
     environment or methodology changes, known issues.
   - **Packet Throughput**: NDR, PDR throughput graphs based on results
     from repeated same test job executions to verify repeatibility of
     measurements.
   - **Packet Latency**: Latency graphs based on results from test job
     executions.
   - **Speedup Multi-Core**: NDR, PDR throughput multi-core speedup
     graphs based on results from test job executions.
   - **HTTP/TCP Performance**: HTTP/TCP VPP test server performance
     graphs.
   - **Comparisons**: Performance comparisons between VPP releases and
     between different testbed types.
   - **Throughput Trending**: References to continuous VPP performance
     trending.
   - **Test Environment**: Performance test environment configuration.
   - **Documentation**: Documentation of K8s Pod/Container orchestration
     in CSIT and pointers to CSIT source code documentation for VPP
     performance tests.

#. DPDK PERFORMANCE: DPDK performance tests executed in physical
   FD.io testbeds.

   - **Overview**: Tested logical topologies, test coverage.
   - **Release Notes**: Changes in |csit-release|, known issues.
   - **Packet Throughput**: NDR, PDR throughput graphs based on results
     from repeated same test job executions to verify repeatibility of
     measurements.
   - **Packet Latency**: Latency graphs based on results from test job
     executions.
   - **Comparisons**: Performance comparisons between DPDK releases and
     between different testbed types.
   - **Throughput Trending**: References to regular DPDK performance
     trending.
   - **Test Environment**: Performance test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for
     DPDK performance tests.

#. VPP FUNCTIONAL: VPP functional tests executed in virtual FD.io
   testbeds.

   - **Overview**: Tested virtual topologies, test coverage and naming
     specifics;
   - **Release Notes**: Changes in |csit-release|, added tests,
     environment or methodology changes, known issues.
   - **Test Environment**: Functional test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for
     VPP functional tests.

#. HONEYCOMBE FUNCTIONAL: Honeycomb functional tests executed in
   virtual FD.io testbeds.

   - **Overview**: Tested virtual topologies, test coverage and naming
     specifics;
   - **Release Notes**: Changes in |csit-release|, known issues.
   - **Test Environment**: Functional test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for
     Honeycomb functional tests.

#. NSH_SFC FUNCTIONAL: NSH_SFC functional tests executed in
   virtual FD.io testbeds.

   - **Overview**: Tested virtual topologies, test coverage and naming
     specifics;
   - **Release Notes**: Changes in |csit-release|, known issues.
   - **Test Environment**: Functional test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for
     NSH_SFC functional tests.

#. DMM FUNCTIONAL: DMM functional tests executed in
   virtual FD.io testbeds.

   - **Overview**: Tested virtual topologies, test coverage and naming
     specifics;
   - **Release Notes**: Changes in |csit-release|, known issues.
   - **Test Environment**: Functional test environment configuration.
   - **Documentation**: Pointers to CSIT source code documentation for
     DMM functional tests.

#. DETAILED RESULTS: Detailed result tables auto-generated from CSIT
   test job executions using RF (Robot Framework) output files as
   sources.

   - **VPP Performance NDR/PDR**: VPP NDR/PDR throughput and latency.
   - **VPP Performance MRR**: VPP MRR throughput.
   - **VPP K8s Container Memif**: VPP K8s Container/Pod topologies
     NDR/PDR throughput.
   - **DPDK Performance**: DPDK Testpmd and L3fwd NDR/PDR throughput
     and latency.
   - **VPP Functional**: Detailed VPP functional results.
   - **Honeycomb Functional**: Detailed HoneyComb functional results.
   - **NSH_SFC Functional**: Detailed nsh-plugin functional results.
   - **DMM Functional**: Detailed DMM functional results.

#. TEST CONFIGURATION: VPP DUT configuration data based on VPP API
   Test (VAT) Commands History auto-generated from CSIT test job
   executions using RF output files as sources.

   - **VPP Performance NDR/PDR**: Configuration data.
   - **VPP Performance MRR**: Configuration data.
   - **VPP K8s Container Memif**: Configuration data.
   - **VPP Functional**: Configuration data.

#. TEST OPERATIONAL DATA: VPP DUT operational data auto-generated
   from CSIT test job executions using RFoutput files as sources.

   - **VPP Performance NDR/PDR**: VPP `show run` outputs under test
     load.

#. CSIT FRAMEWORK DOCUMENTATION: Description of the overall FD.io
   CSIT framework.

   - **Design**: Framework modular design hierarchy.
   - **Test naming**: Test naming convention.
   - **Presentation and Analytics Layer**: Description of PAL CSIT
     analytics module.
   - **CSIT RF Tags Descriptions**: CSIT RF Tags used for test suite and
     test case grouping and selection.
ass="n">lisp_add_del_locator( locator_set_name=self._ls_name, sw_if_index=self._sw_if_index, priority=self._priority, weight=self._weight, ) self._test.registry.register(self, self.test.logger) def get_lisp_locator_dump_entry(self): locators = self.test.vapi.lisp_locator_dump( is_index_set=0, ls_name=self._ls_name ) for locator in locators: if locator.sw_if_index == self._sw_if_index: return locator return None def query_vpp_config(self): locator = self.get_lisp_locator_dump_entry() return locator is not None def remove_vpp_config(self): self.test.vapi.lisp_add_del_locator( locator_set_name=self._ls_name, sw_if_index=self._sw_if_index, priority=self._priority, weight=self._weight, is_add=0, ) self._test.registry.register(self, self.test.logger) def object_id(self): return "lisp-locator-%s-%d" % (self._ls_name, self._sw_if_index) class LispEIDType: PREFIX = 0 MAC = 1 NSH = 2 class LispKeyIdType: NONE = 0 SHA1 = 1 SHA256 = 2 class LispEID: """Lisp endpoint identifier""" def __init__(self, eid): self.eid = eid self._type = -1 # find out whether EID is ip prefix, or MAC try: self.prefix = ip_network(self.eid) self._type = LispEIDType.PREFIX return except ValueError: if self.eid.count(":") == 5: # MAC address self.mac = self.eid self._type = LispEIDType.MAC return raise Exception("Unsupported EID format {!s}!".format(eid)) @property def eid_type(self): return self._type @property def address(self): if self.eid_type == LispEIDType.PREFIX: return self.prefix elif self.eid_type == LispEIDType.MAC: return self.mac elif self.eid_type == LispEIDType.NSH: return Exception("Unimplemented") @property def packed(self): if self.eid_type == LispEIDType.PREFIX: return {"type": self._type, "address": {"prefix": self.prefix}} elif self.eid_type == LispEIDType.MAC: return {"type": self._type, "address": {"mac": self.mac}} elif self.eid_type == LispEIDType.NSH: return Exception("Unimplemented") class LispKey: """Lisp Key""" def __init__(self, key_type, key): self._key_type = key_type self._key = key @property def packed(self): return {"id": self._key_type, "key": self._key} class VppLispMapping(VppObject): """Represents common features for remote and local LISP mapping in VPP""" def __init__(self, test, eid, vni=0, priority=1, weight=1): self._eid = LispEID(eid) self._test = test self._priority = priority self._weight = weight self._vni = vni @property def test(self): return self._test @property def vni(self): return self._vni @property def eid(self): return self._eid @property def priority(self): return self._priority @property def weight(self): return self._weight def get_lisp_mapping_dump_entry(self): return self.test.vapi.lisp_eid_table_dump( eid_set=1, vni=self._vni, eid=self._eid.packed ) def query_vpp_config(self): mapping = self.get_lisp_mapping_dump_entry() return mapping def object_id(self): return "lisp-mapping-[%s]-%s-%s-%s" % ( self.vni, self.eid.address, self.priority, self.weight, ) class VppLocalMapping(VppLispMapping): """LISP Local mapping""" def __init__( self, test, eid, ls_name, vni=0, priority=1, weight=1, key_id=LispKeyIdType.NONE, key="", ): super(VppLocalMapping, self).__init__(test, eid, vni, priority, weight) self._ls_name = ls_name self._key = LispKey(key_id, key) @property def ls_name(self): return self._ls_name @property def key_id(self): return self._key_id @property def key(self): return self._key def add_vpp_config(self): self.test.vapi.lisp_add_del_local_eid( locator_set_name=self._ls_name, eid=self._eid.packed, vni=self._vni, key=self._key.packed, ) self._test.registry.register(self, self.test.logger) def remove_vpp_config(self): self.test.vapi.lisp_add_del_local_eid( locator_set_name=self._ls_name, eid=self._eid.packed, vni=self._vni, is_add=0, ) def object_id(self): return "lisp-eid-local-mapping-%s[%d]" % (self._eid.address, self._vni) class LispRemoteLocator: def __init__(self, addr, priority=1, weight=1): self.addr = addr self.priority = priority self.weight = weight @property def packed(self): return { "priority": self.priority, "weight": self.weight, "ip_address": self.addr, } class VppRemoteMapping(VppLispMapping): def __init__(self, test, eid, rlocs=None, vni=0, priority=1, weight=1): super(VppRemoteMapping, self).__init__(test, eid, vni, priority, weight) self._rlocs = rlocs @property def rlocs(self): rlocs = [] for rloc in self._rlocs: rlocs.append(rloc.packed) return rlocs def add_vpp_config(self): self.test.vapi.lisp_add_del_remote_mapping( rlocs=self.rlocs, deid=self._eid.packed, vni=self._vni, rloc_num=len(self._rlocs), ) self._test.registry.register(self, self.test.logger) def remove_vpp_config(self): self.test.vapi.lisp_add_del_remote_mapping( deid=self._eid.packed, vni=self._vni, is_add=0, rloc_num=0 ) def object_id(self): return "lisp-eid-remote-mapping-%s[%d]" % (self._eid.address, self._vni) class VppLispAdjacency(VppObject): """Represents LISP adjacency in VPP""" def __init__(self, test, leid, reid, vni=0): self._leid = LispEID(leid) self._reid = LispEID(reid) if self._leid.eid_type != self._reid.eid_type: raise Exception("remote and local EID are different types!") self._vni = vni self._test = test @property def test(self): return self._test @property def leid(self): return self._leid @property def reid(self): return self._reid @property def vni(self): return self._vni def add_vpp_config(self): self.test.vapi.lisp_add_del_adjacency( leid=self._leid.packed, reid=self._reid.packed, vni=self._vni ) self._test.registry.register(self, self.test.logger) @staticmethod def eid_equal(eid, eid_api): if eid.eid_type != eid_api.type: return False if eid_api.type == LispEIDType.PREFIX: if eid.address.prefixlen != eid_api.address.prefix.prefixlen: return False if eid.address != eid_api.address: return False return True def query_vpp_config(self): res = self.test.vapi.lisp_adjacencies_get(vni=self._vni) for adj in res.adjacencies: if self.eid_equal(self._leid, adj.leid) and self.eid_equal( self._reid, adj.reid ): return True return False def remove_vpp_config(self): self.test.vapi.lisp_add_del_adjacency( leid=self._leid.packed, reid=self._reid.packed, vni=self._vni, is_add=0 ) def object_id(self): return "lisp-adjacency-%s-%s[%d]" % (self._leid, self._reid, self._vni)