aboutsummaryrefslogtreecommitdiffstats
AgeCommit message (Collapse)AuthorFilesLines
2020-11-26Report: Reorganize the NAT44 chaptersTibor Frank45-1153/+1866
- packet throughput - speed-up Change-Id: I5d49ae2da54170e48c51db47e035d2121471be0d Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 52e3b0f6b6b4f92c5ee5ecf003623cb59c798290)
2020-11-26Revert "Report: Hide 2n-zn2 results"Tibor Frank30-2521/+2494
This reverts commit 7d6ee11472e8ab675248616fbc90005eaf73128b. Reason for revert: publish 2n-zn2 results Change-Id: I3c5716af92a014efa0901244260099b9f1cf1af9 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-11-26Trending: Fix build number in error message.Tibor Frank1-6/+1
Change-Id: I1708ebd24a6a26db53a14192b23fe51569543333 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 610e96a2afae5c8e58bee00d05c45d57d7bb6021)
2020-11-25Report: Hide 2n-zn2 resultsTibor Frank30-2494/+2521
Change-Id: I26587c8b987be9c7b6fb3a1d79d77a4e1af417f8 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-11-25Report: Update release notes.Vratko Polak1-3/+7
+ Expand details on NAT stateless tests added. + Clarify situation with jumbo ipsec. Change-Id: Id6fb7fa48b5fadca7798ec1da7e1baf44d0827aa Signed-off-by: Vratko Polak <vrpolak@cisco.com> (cherry picked from commit 57476e27b7220d37ca951cdf6b8e12fc0fbc8288) (cherry picked from commit 0a8d967bc3158cb20fe4552537a574fa5f9972af)
2020-11-25Report: Add dataTibor Frank1-0/+2
- 2n-zn2 iter Change-Id: I7412963c918b53e44294f4cf213a806c8435e3b5 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 4aa4a9d82e5eb41823b56bda56a0d34d9a3b26b1)
2020-11-23Report: Add 2n-zn2 DPDK LatencyTibor Frank1-0/+11
Change-Id: Id4738b823a77e59d5efd913f770326726b8a81e2 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit b6340be1caf13025b2aaac06ccf6ad3bb64d593f)
2020-11-23Trendig: Fix alerting for 2n-zn2Tibor Frank1-1/+3
Change-Id: I4e4d156dcb9d93bf0c537a870e6fab6c500f5810 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 379e58a740271d84b15059f7059661fbec13cef8)
2020-11-23Report: Configure rls2009.48Tibor Frank3-2/+5
Change-Id: Ib7c4bc51bea43dafa2bd788c709c800caeb75988 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit ef51e552aa2d0e134646d28aa3ada2da0d467f99)
2020-11-23Report: Add 2n-zn2Tibor Frank76-82/+8289
Change-Id: Ifaba0d689a279ae650ae30531b21533cc6927262 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit f8b3d745368deb92af2b7ff70874e3b0ae80a0c7)
2020-11-20PAL: Remove temporary pdf filesTibor Frank1-0/+6
Change-Id: I65cf505edd346fc5da240be08310064b3755e458 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 4d6dcbbbb9103f15fb55d525f14964c2323c2a03)
2020-11-20Add and reorder methodology sectionsVratko Polak1-3/+4
+ DUT state chapter was not reachable. + Small reorder to put edits together and sooner. Change-Id: I2735c522e92e277298e299176b4cdc9be4044c26 Signed-off-by: Vratko Polak <vrpolak@cisco.com> (cherry picked from commit 0e835cca6f7dbb1b8c3c0a0356ae7546aa5258bf)
2020-11-20Report: Add dataTibor Frank1-1/+2
Change-Id: I7efb7a339e248972604b1c8027d21ea32e9f4070 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 3ef4eb87e921a350361a54b0449b6536c49c5794)
2020-11-19Methodology: Add profile details to nat44 docVratko Polak1-8/+230
Change-Id: I5dc427693f78e4c712ab5cae587d7849dc2fec16 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-11-19Methodology: Edit nat44 test spec docVratko Polak1-33/+97
- Missing profile specifics and transaction counters. Change-Id: I6f7378e5fe9d639599c38545b0503354a8a65198 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-11-19Methodology: DUT state considerationsVratko Polak1-0/+150
Change-Id: I4bdb61cf665579c6b49d79c6fb281b4c0fb5ec89 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-11-19Methodology: Trex modes and transactionsVratko Polak3-49/+199
Change-Id: I43423dea499bce3a298dbbba752c2aee2a322836 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-11-18Report: Add dataTibor Frank1-2/+3
Change-Id: I73a735b3e82aaa01f56c21bf91e5073f144c584f Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit dbae94b3db123a5e86b0a5afb5f5baaa262ce8ad)
2020-11-162n-zn2 Epyc backportpmikus88-77/+6665
- Better to get it fully aligned then cherry-pick spaghetti. Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: If223ef3f0247413d53225eb57f8903a7675632e3
2020-11-16Report: Add dataTibor Frank1-11/+69
Change-Id: I3aaf06de4ae5fb2dbeef5920987bc427bf0bb529 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 6b33c89d362623dd46f3a14c497835de6f669101)
2020-11-13Report: Add dataTibor Frank1-17/+28
Change-Id: I2027c8e0be76c119d88c966f700f9639ed4c1784 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 68d130f9405ed239e624f9dd016381b70ee57001)
2020-11-12Report: Fix spec of latency graphsTibor Frank1-11/+0
Change-Id: Ic2d8c66ec44b89354e4dbf4f9605c8e53bf21481 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit f4621bff99ef04d87dad83a752f55e16423324e0)
2020-11-12Report: Add NAT44 dataTibor Frank4-356/+164
- reorder NAT44 graphs Change-Id: Ied762a85fb1ee69c7d12971e3e15822571a18451 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 6328731a2778c864c6372296449a55dae79cd305)
2020-11-11Report: Add comparison soak vs ndrTibor Frank1-2/+0
Change-Id: I19308ed936246be5b4e61c6e219591d16095cb56 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 1e149e65b7ee02ad5235e4b3c61102ce7d5d89ff)
2020-11-10ASTF: Lessen L7 processing in UDP_CPS profilesVratko Polak5-40/+30
Change-Id: I8b154156120821adb24273db2a232fa82200c0fe Signed-off-by: Vratko Polak <vrpolak@cisco.com> (cherry picked from commit 780612e1eaa2414808fb2cd6c88f7d428a8e3eed)
2020-11-10TCP CPS: Count NAT sessionsVratko Polak1-2/+4
The previous code counted full TCP connections, which need one more packet, leading to worse results. Change-Id: Ifcf78356b6ed54819ea0bf5aa069d7d9cb951183 Signed-off-by: Vratko Polak <vrpolak@cisco.com> (cherry picked from commit b71112bc323b55e39d8a9992a46530e1eb7f6f58)
2020-11-09Report: Add dataTibor Frank1-0/+2
Change-Id: Id59947b8c502e4bf7b7ce66ad326f481346c98d6 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit c70bc9fce0954dce4353d3f99d9fc7d1a385583f)
2020-11-06PAL: Tests order in plotsTibor Frank1-73/+91
Change-Id: Ic9342f0f3deea3d7d175c437993289c86464bb99 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 494f7e4abf409894ccb3652ded0cb0eedeabf85a)
2020-11-06Report: Add NAT44 LatencyTibor Frank3-84/+163
Change-Id: I2c247c5362396bcb4c8b42b1b46a2601091ca968 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 4e056959f9e5a01f1c1cc6816d50609a4aa42366)
2020-11-05Report: Add NAT44, part 2Tibor Frank1-16/+278
Change-Id: I81c3b43407921ed7912b530bb93c725515d3d583 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 44186cb43aaa4131f676d9d18f8f573823ec9512)
2020-11-05Report: Configure rls2009.46Tibor Frank3-2/+19
Change-Id: I7886285de8edbf66e03e1b22d85c2e7a4c4d7224 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 20f419f950606f1848a81d815c251f8180867b50)
2020-11-03Report: Fix bugs in NAT44 graphsTibor Frank6-1371/+227
Change-Id: I6de094fc0cb2acdfb6eeb22e6d85a65d2e9f1478 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit a1a6ce5505840750a34d86ada800367a183cf74b)
2020-11-02job-specs: Update the durationsTibor Frank16-64/+64
Change-Id: Ibed3ad934bc4fd03013404e18b5a0d6917499bf8 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 579bccb6aa18c97f3c1a1636658eb41235a18b7a)
2020-11-02Report: Add dataTibor Frank2-11/+27
Change-Id: I800273d04c3f956e58be727cbf5befa61377fa2b Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 09f0b96e5484a24e11415ef7839dbf8f1dcfd9bb)
2020-11-02Report: Add dataTibor Frank2-3/+20
Change-Id: I99836894960f74f9dc283b812e1106ac512db3ae Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 380ed1f7a22d960ecf4b6a34c15cf3c69a32b800)
2020-11-01Report: NAT44 LatencyTibor Frank2-106/+61
Change-Id: Ifca9e4f5783eea37df30ddfedc80c988f2163ad8 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 2cc09e0ca9c00135144a3774443937eaba3be56c)
2020-11-01Report: Add dataTibor Frank1-4/+68
Change-Id: I1cf61c523d781db3dbeccf89077c50992cb03205 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 41cdf812ff618809e6181a561afbf86b7fae67f0)
2020-10-30Report: Add NAT44 dataTibor Frank2-52/+41
Change-Id: I463bf61d41484738ba5c614b473bcb5457e6abe1 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit d500d77f542b718b0bd7c1dd5e20e70b9f196329)
2020-10-30Report: Add NATT44 testsTibor Frank6-494/+1623
Change-Id: I3920f881286565029f35bcfa700f36c6701db457 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit b0791718d2a3cb7a6c902496f721ecb817013aa5)
2020-10-30Report: Add dataTibor Frank1-0/+4
Change-Id: Iaa85aaa5179136281ab05325efcf0e91c234a774 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit e2e037d5965dabf8b274ed0fe67fb89d5d324d0b)
2020-10-29Support existing test types with ASTFVratko Polak159-2555/+6632
Manual cherry-pick from master [1], reverting impact of [2] and [3]. [1] https://gerrit.fd.io/r/c/csit/+/28208/176 [2] https://gerrit.fd.io/r/c/csit/+/29077 [3] https://gerrit.fd.io/r/c/csit/+/29529 The heap multipliers are left in suites, as that simplifies cherry-picking between branches. Original [0] commit message: Support existing test types with ASTF + Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-10-29job-spec: Add all NAT44 testsTibor Frank10-474/+1790
- the specs: - docs/job_specs/report_iterative/2n-clx/vpp-mlr-00.md - docs/job_specs/report_iterative/2n-clx/vpp-mrr-00.md - docs/job_specs/report_iterative/2n-skx/vpp-mlr-00.md - docs/job_specs/report_iterative/2n-skx/vpp-mrr-00.md take too long. We should consider to remove some tests. - the coverage specs were split to three parts Change-Id: I39e2032c76266e159a781d1225cff1a5119d4c85 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit e31998ea56c55879fbaae8e58b0dad0bc6549dae)
2020-10-28PAL: Remove debug info from result parserTibor Frank1-6/+0
Change-Id: I9b6c2185e43ed0b33dbeafb3d2757656f9e49bc3 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit b9aabb97bb10297ce004f731b7b61b9dc9c83ee6)
2020-10-28PAL: Add processing of PPS and CPS testsTibor Frank3-39/+396
Change-Id: I7b15c69c923d47e83c2dc586cdc03ed4ebaeec74 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 7bcbdcb30d2eea8fe7e1fb60696e39abef897920)
2020-10-28Report: Configure rls2009.45Tibor Frank3-2/+10
Change-Id: I8918f09411254b64abb65a165a0e284f735b306e Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 764041478f165644c0dea744bc4b2d976bb0b840)
2020-10-28Report: Add dataTibor Frank2-5/+7
Change-Id: I2394b42e177758a8f3ce4aeeb912b264cf3b457c Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit bb53fd3a235007eb0ed23f8023d7fa7044c51929)
2020-10-26Report: Add dataTibor Frank1-15/+32
Change-Id: Ib5acc321ac7d5db7598b30b9783cee06ee8c1462 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 0d8a5c07c35ea82566a5392e35c630a6eea6c05d)
2020-10-23Report: Add dataTibor Frank1-0/+4
Change-Id: Iaa5fb969b699985da8ec3356aa32fc0cb757affb Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 905ab17015b24183770e3b55517482e30904d7f4)
2020-10-22Report: Add dataTibor Frank1-14/+24
Change-Id: If03e4adf3672413016058fae85219ae5c7e922db Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 8c840bcaf34a1ed3e43c78208aa3ab053c07719c)
2020-10-22Report: Configure 2009.44Tibor Frank3-2/+25
Change-Id: I7f7e50610e71af7f2b0dc1eed267a421ffd8aef0 Signed-off-by: Tibor Frank <tifrank@cisco.com> (cherry picked from commit 1b25441a3d82a12d01334bd82795bd35a4f7c2e3)