aboutsummaryrefslogtreecommitdiffstats
path: root/resources/libraries/robot/performance
AgeCommit message (Collapse)AuthorFilesLines
2019-09-24Add support for HDRhistogramVratko Polak1-1/+1
+ Enable hdrh in trex server. + Append hdrh coded output after min/avg/max/. + Read (not show nor decode) hdrh value in PAL. + Also, remove old ndrpdrdisc code. Change-Id: I99d99f10386a621772b5419ca1f36080fa15aca7 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-09-24Reconf tests: Fix async measurementsVratko Polak1-3/+8
TRex does not zero the server counters. It copies the values to use as reference, and subtracts them when asked for results. But the reference is stored in the client (not the server). And CSIT uses different scripts to start and stop async traffic, which means different clients. This patch introduces a workaround. Async start will return xstats objects to use as reference, and async stop will use the objects to compute the correct results. The xstats objects are stored in TrafficGenerator instance. Sync measurement does not export the counters, to shorten logs. Other improvements: + Make stop_traffic_on_tg return measurement results directly. + Rename --async to --async_start as "async" is reserved in Python 3.7 + Minor pylint, docstring and typo fixes. Change-Id: I5fc56a0763afb7d62cfa7c0651f96b6867de3e15 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-09-12Support unidirection in all searchesVratko Polak1-61/+84
+ Rename bool unidirection to int traffic_directions. + Rename "untagged" to "initial" for bandwidth calculation. + Fix latency measurement for unidirectional traffic. + Remove duplicate colon in soak test message. + Edit PAL to accept both forms. + Fix minor documentation issues. Change-Id: I6c76f2dc090ae493f2fbd7e9ccd45229d2306dea Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-09-10Refactor getting telemetryPeter Mikus1-3/+4
+ Ability to get stats from CNF via SocketPAPI - Remove obsolete functions Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I4d1b32a7279244592be96644e4f8a530c4f29a15
2019-09-06Read environment variables in Constants.pyVratko Polak1-2/+3
Instead of using EnsureGlobalVariable, which is clunky to use from Python. As a consequence, all caps variables from Constants.py are used directly and tests/__init__.robot and robot_enhancements.robot are deleted. + Rename the CRC global kill switch based on ci-man review. Change-Id: I10723792475bc83352bf8c3b7f7946ecf885a194 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-08-23Introduce reconfig suites, for dot1q+ip4+vxlanVratko Polak2-12/+136
Ticket: CSIT-1551 + Config copied from other suites. + Layer keywords prepared for additional chains. + Renamed "create" argument to "vlan_per_chain". + TrafficGenerator result extraction improvements. + Also unified TG type and subtype checking. + Throughput estimated as PDR for zero PLR. + Actual Qemu start is skipped. + Parse xstats instead of stats in trex stop scipt. - Because stats are unconditionally cleared on each script connect. + Remember start values to get better ReceiveMeasurementResult at stop. + Expose loss count and time as appropriate test message. + Autogen support. + Multiple chain amounts and core densities. - Only 1 additional chain. + Proper TAGs. Documented. + Clarified min framesize tags are for TG-DUT. + Added a TODO concerning overhead values. + Added 118B tag. + Robot performance keywords improvements. + Added missing return value documentation. + Frame size as required test variable, not an explicit argument. + MLRsearch keyword to store result (not into test message). - TODO: De-duplicate: main keyword to measure, minor to set test message. - But implementation uses the fact default arguments are different. + Keywords for starting and stopping background traffic. - No checks whether start-stop pair is complete. + Multiple minor indentation and logging improvements. - Missing / TODOs: - Support also vlan_per_chain=True? - Support dot1q without vxlan? - Support also deletion? - How to package config steps so -reconf can be generated from -ndrpdr? Change-Id: If0270697f9ffa837c34ef802917220a53f156703 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-08-23Improve PLRsearch yet againVratko Polak1-1/+1
Logic improvements for rls1908 and post-pylint fixes: + Reduce search time to 30 minutes. + Use average instead alternating loads. + Rework log/exp avg/stdev from two estimates. + Introduce and use pessimistic variance of dual trackers. + Introduce safe_exp to use when None does not skip code. + Use dot relative imports (instead of disabling pylint). + Complete docstrings for simpler functions. + Append docstrings to named tuples. + Somewhat unify docstrings related to the same arguments. + Slightly improve intentation. + State named tuples as rtype where used. + Add returns and rtype to __repr__ where missing. + Return what docstring say (or update them). + Explicit copying for vector and matrix values. Change-Id: I884c68b4839c5df5e8bef95e463666599603a0ff Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-08-12Clean: Various librariesPeter Mikus2-666/+3
- Remove translate TC as they have no perf opposite - Remove unused keywords - Merge similar keywords / rename - Move keywords from perf library to categories Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I6433006136e2497ad8fb17193aa20c578d836999
2019-08-09Clean: L2 bridge domainPeter Mikus1-815/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I3721900b91a9c8e9db8766bdcb9771b6bcf95bca
2019-08-09Clean: L2_xconnect keywordsPeter Mikus1-284/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ic15a15c426db04defc15840883a772a0eb8e4885
2019-08-09Clean: L2_patchPeter Mikus1-9/+0
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ib3620689acb3641b2ec9a49e2ce552bab3f00eef
2019-08-09Clean: Interface KeywordsPeter Mikus1-176/+16
Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: Ideddcea56120cf770aec78ca24813baf7d05ba11
2019-08-07Clean: IPv6 keywordsPeter Mikus1-214/+6
+ Move all base ipv6 related KW to library + Remove useless KW to save space in output.xml + Remove cross dependency in ip6.robot Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: I8c5f72e9852c1017356ab6034c4d271cf7b53265
2019-08-07Clean: IPv4 keywordsPeter Mikus1-547/+25
+ Move all base ipv4 related KW to library + Remove useless KW to save space in output.xml + Remove cross dependency in ip4.robot Signed-off-by: Peter Mikus <pmikus@cisco.com> Change-Id: If1e3a6370d36caf4ce7411fb956a0b7d0f939b6b
2019-08-06VPPD: Dot1QPeter Mikus1-18/+0
Change-Id: I0d3c925ea4a6896a0df98db6ddaf4238e6291bf1 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-26Change: Move few KWPeter Mikus1-35/+0
Change-Id: I8c8ae1d7eed8cf28fc63ce42b2dc36dd16a5c0da Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-24NF_density dot1qip4udpvxlanPeter Mikus1-26/+41
+ NF_density compatible + Baseline VM tests compatible + Scalability NxM - 3n variant is missing in VXLAN and DOT1Q due to requests to have it asymmetrical - TODO Change-Id: Ia817d7be00622b594b451bc07756240d1a379e04 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-19FIX: COP ipv6 missing FIBPeter Mikus1-2/+2
- From unknown reason VAT was more tolerant Change-Id: Ia740843c67604a18de40d14cc448138b9f079b41 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-17Remove obsolete DPDK bondingPeter Mikus1-99/+0
- Code not used Change-Id: I312635f8038c6d4d4db21d93e43df6b3636feb34 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-17Refactor VPP Device VM vhost testsjuraj.linkes1-85/+71
* replace the current VM image with kernel img * rework keyword usage to make it consistent with performance tests * remove resources/libraries/robot/shared/qemu.robot as it's not used anywhere anymore Change-Id: Ia5bc19e9e6ed9af031e4d9b5c0c89431fb49fd33 Signed-off-by: juraj.linkes <juraj.linkes@pantheon.tech>
2019-07-12VPPD: Cop, iACL, PolicerPeter Mikus1-58/+0
- Alignment with PERF Change-Id: Id7b2cb91c88e44cf9265c3376330f6aa579abf7a Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-07-12PapiExecutor always verifiesVratko Polak1-2/+2
Do not support returning unverified replies anymore. Basically, ".get_replies().verify_replies()" is now just ".get_replies()". This allows fairly large simplifications both at call sites and in PapiExecutor.py + Rename get_dumps to get_details. + Introduce get_reply and get_sw_if_index. + Rename variables holding get_*() value, + e.g. get_stats() value is stored to variable named "stats". + Rename "item" of subsequent loop to hint the type instead. + Rename "details" function argument to "verbose". + Process reply details in place, instead of building new list. - Except hybrid blocks which can return both list or single item. - Except human readable text building blocks. + Rename most similar names to sw_if_index. - Except "vpp_sw_index" and some function names. + Use single run_cli_cmd from PapiExecutor. + Do not chain methods over multiple lines. + Small space gain is not worth readability loss. + Include minor code and docstrings improvement. + Add some TODOs. Change-Id: Ib2110a3d2101a74d5837baab3a58dc46aafc6ce3 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-07-09CSIT-1539 Aligning FUNC/PERF keywords - Phase IIIPeter Mikus1-56/+96
Change-Id: Ibd3e99390bf004173d3d126946b95ffe9ba8f196 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-06-19FIX: VXLAN scale testsJan Gelety1-4/+3
Change-Id: I9287d5ed6d7876b98add3fb3d523e9764094843b Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-07-05Remove pkt_trace as global variablePeter Mikus1-2/+3
- Use explicit parameters Change-Id: I21cbc8cf13ec5e455b20c685c4427a53a655f71f Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-06-28VAT-to-PAPI: ClassifyTibor Frank1-4/+2
Change-Id: Ic06a0a65429680e6ecdc3f5288d091c2c2630921 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2019-06-16FIX: IPUtil after vpp api changesJan Gelety1-4/+4
+ do not log all ip_route_add_del api calls to PAPI history in case of scale tests + fix for SRv6 proxy tests + support of more complex structures in api calls/replies in PapiExecutor + update of vpp stable build to version 19.08-rc0~487-gb98dbb1 Change-Id: I9972e126b68ce1cb57f781e3d90ea9b3914de69e Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-06-25Align suite/test teardown/setupPeter Mikus2-672/+1
+ Phase II - aligning FUNC/PERF setup. + 2n/3n compatibility of suite setup. + Same structure of testcase for DEV/PERF. + DEVICE tests are now running VPP startup config in same way as PERF. + DEVICE is using COREKEEPER! - DEVICE tests are running topology compute twice for backward compatibility. TODO to remove. - Still phase III needed to align TAGS, LIBRARIES and remove OLD setup phase. Change-Id: Ib6e1b0ebf4abebf3cbe73b4fda5cac953c2a7270 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-06-21Align suite/test teardown/setupPeter Mikus1-250/+1
+ Phase I - aligning FUNC/PERF Change-Id: I46b1e43687ddb29f5ed3c6335fe1baf21ec02822 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-06-19Add continue/fail option for enable tracesTibor Frank1-1/+1
Change-Id: I675f9664391c9f18f312773812f765f345983cc1 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2019-06-18CSIT-1459: Migrate IP libraries from VAT to PAPIJan Gelety1-164/+192
Change-Id: Id84cf9c3afd53260b5bbf658a6a3870aa0c91333 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-06-19VAT-to-PAPI: VPPUtilsTibor Frank1-2/+0
Change-Id: If4eb592e2718c4564f1f7929ca6ff91c02f074a6 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2019-06-19Add scale ipsec tunnel testsPeter Mikus1-1/+0
- 5K - 10K Change-Id: Iaa2c77d07c3686e09fe327c663d7d7fc42024034 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-06-09CSIT-1521 Convert NestedVM testcase into KernelVMPeter Mikus1-418/+94
Change-Id: Ic493253dc29dad4fc7e6541e4b839ca5898a6a40 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-06-05Even more Crypto combinationsPeter Mikus1-6/+0
Change-Id: I10eeb4ee30a57712824e68176d92d1ecb5f0d1b0 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-05-31CSIT-1468: InterfaceUtil migration from VAT to PAPIJan Gelety1-6/+6
Change-Id: I0062710d58996be767a852dc00545fedd60a5c72 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-05-10PLRsearch: Use stat trackers to shorten IntegratorVratko Polak1-2/+2
+ Extract several tracker classes and use them in Integrator. + Apply next_rate workarounds to focus more on critical region. + Rewrite stretch function, as the previous implementation was wrong. + Rework logging: ++ Use injected trace() function in Integrator. ++ Inject function that skips trace logging on default initialization. ++ Use the same multiprocessing-safe passing, but separate queue. + Set duration to 120m in Robot, but keep at 30m in Python. + Apply minor changes to make computations more reproducible: ++ Attempt to log everything needed by reproducibility in one line. ++ Log samples used, and make it settable as upper limit. ++ Use repr output in TRex scripts, to avoid rounding when copypasting. +- Numpy seems to be sharing PRNG between processes. +-- An edit to disable one thread is needed for full reproducibility. +-- Such an edit is prepared, but will not be merged anytime soon. Change-Id: Icf56429b34ab6de2d18a12e6c676e2d242b3ba4c Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-05-10FIX: Missing show run data in NDRPDRPeter Mikus1-0/+27
Change-Id: If4b013ef4bf4bc8f857913f3847ab3b2e2aa5a5d Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-05-06CSIT-1493 VPP restart handling codePeter Mikus1-1/+2
Change-Id: Ibe52125089f39e0ff17ec607a3ed00c61d52ab8c Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-04-26NF density tests with dtc=0.5 and dtcr=2Peter Mikus2-64/+14
Change-Id: Icff556142280ad0b6261e0a2bfb71672ee6b3807 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-04-24FIX: Crypto testsPeter Mikus1-0/+9
- By adding crypto_ipsecmb.so into the list of loaded plugins Change-Id: I52de9b410441422ad8e337208f40360fbdea264d Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-04-17Clean up traffic_profile vs osi_layerVratko Polak2-51/+57
The two types were not well distinguished before. Error introduced in: https://gerrit.fd.io/r/#/c/17811/84/resources/libraries/robot/performance/performance_setup.robot@255 Error hotfixed in: https://gerrit.fd.io/r/#/c/18847/4/resources/libraries/robot/shared/default.robot@109 + Rename some arguments and improve method docstrings. Newly introduced argument name osi_layer should be dissimilar enough. Change-Id: Ie0f6f97dc010fc6477f09c54574970f1d15462e2 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-04-15Remove obsolete QEMU installationPeter Mikus1-141/+0
+Unify QEMU parameter to Contants. Change-Id: I95adfc97dd7008c19e60ba3fa8668818b503bee8 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-04-12Convert existing nf_density testpmd-mac into KernelVMPeter Mikus1-2/+2
Change-Id: I9ec1701d9012de59cc42cc5f9a4744f01695ff9a Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-04-08Upgrade autogen to NICs and search typesVratko Polak3-453/+309
+ All perf suites updated, as autogen change is backward incompatible. + Only x710 -ndrpdr suites remain in git repository. + Removed retry feature of run_tests. + Renamed topology_type and traffic_type to traffic_profile. + Renamed framesize to frame_size; nic_model to nic_name. + Reordered Variables table to start with nic_name and overhead. + Fixed wrong overhead value in some cbc-sha1 suites. + Fixed some suite tags. ++ Introduced tags to distinguish suites, such as IPSECINT. + Bound crypto hardware to NIC name. + Implemented NIC limit hiding. + Also search and teardown related arguments are now hidden. + Main measurement keywords updated to new arg handling. + Max rate related keywords moved to performance_limits.robot + Teardowns unified. + Generated tests are archived. ++ Generated directory is .gitignore-d. + Regenerator raises an exception on seeing non-compatible suite. ++ Relatively helpful message should be seen in exception. + Suite and template Documentation has generated parts. +- With short NIC names only. + Autogen checker also upgraded. + Tag expressions replaced with long files to ensure analogous tests. Change-Id: I60e9a999187e7da1f60d0eb4fb02afa14682aa46 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2019-04-06FIX: Operational issuesPeter Mikus1-48/+29
- FIX MSR package installation - FIX VTS - FIX sockets by removing from TCs Change-Id: I043c60f9e3e9c39cbeaee5e09b7a8351965efd35 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-03-27FIX: change socket location for vhost-user interfaceJan Gelety1-2/+2
Change-Id: Icce908d274f9fcd92e7eaa731da094b63fd3ee0e Signed-off-by: Jan Gelety <jgelety@cisco.com>
2019-03-28Add ipsec crypto test for Denverton platformYulong Pei1-10/+10
add ipsec cryto test to support Denverton hardware crypto device C3xxx. replace HW_cryptodev keyword to HW_DH895xcc or HW_C3xxx, or add more other hardware crypto device type support in the future. Change-Id: I181bca4095757fde7ca45ab3856273ac5930d902 Signed-off-by: Yulong Pei <yulong.pei@intel.com>
2019-03-20CSIT-1386 KernelVM - Part IIPeter Mikus2-36/+83
- Implementation of KernelVM (ThinVM) - nf_density tests for KernelVM with VPP as VNF. Change-Id: Ife5c2e1ab419c55cbcd442792f940db3a41da471 Signed-off-by: Peter Mikus <pmikus@cisco.com>
2019-03-18CSIT-1386 KernelVM - Part IPeter Mikus3-71/+51
- Refactor QemuUtils Full refactor of QemuUtils to prepare for thinner VMs. Change-Id: I171ba11a9e6e9faec582bd58df6819598b0ed0b5 Signed-off-by: Peter Mikus <pmikus@cisco.com>