aboutsummaryrefslogtreecommitdiffstats
path: root/docs
AgeCommit message (Collapse)AuthorFilesLines
2020-10-29Support existing test types with ASTFVratko Polak1-27/+74
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites. + Update existing cps traffic profiles. + Add missing traffic profiles. + UDP: + Single burst of 32 packets was confirmed as safe enough for TRex. + Maybe 64 could work, but not enough testing for that. + Multiple bursts have lead to reduced TRex performance, as overlaping bursts (from different client instances) tend to fill up the buffers. + TCP: + Data size set to 11111 bytes, completely arbitrarily. + Results look reasonable, so I have kept that. - MSS not set at all - No tested support for frame size other than 64B. - Frame size does not even factor into TCP profiles. + So other frame sizes are skipped in autogen. + Update tags in related suites. - HOSTS_{n} and SRC_USER_{n} should be unified. - Questionable clarification on difference between IP4BASE and SCALE. + Add NAT state resetters to tests that need them. + Resetter is called (if set) before each measurement. + If ramp-up is detected, resetter is not set. + Rename "mult" argument to "multiplier". + Abstracted from packets to transactions. + Transaction corresponds to profile. + TRex multiplier argument sets target rate in transactions per second. + The familiar STL traffic: + Bidirectional is considered to be 2 packets per transaction. + Unidirectional is considered to be 1 packet per transaction. + The newer ASTF traffic: + 4 subtypes, each has different number of packets per transaction. + For max rate computation: + Packets in the more numerous direction are considered. + Rely on TRex reported traffic duration for ASTF: + Use the server side value. - Client side value is higher by an overhead. - TRex is not sending traffic during that time. + Remove delays from traffic profiles. - Those delays would increase the reprted traffic time. + Support for scale lmited trials. + Only for ASTF profiles, each ASTF profile has limited scale. + Scale defined in suite variables. + For TRex to send all transactions provided duration value is ignored. + The appropriate value is computed in TrafficGenerator. + An ad-hoc time constant is added to match the TRex client side time overhead. + The profile driver receives the computed duration. + Measurement for PLRsearch add a sleep if the computed duration is smaller. + Alternative argument for search algos if scale is limited. + Both need higher timeout to accomodate big scales. + MLRsearch can afford fewer phases. + Added a parameter to optionally shorten the duration. + Use short duration for runtime stats trial and failure stats trial. + Use very large keepalive values in udp profiles to avoid ka packets. + No polling in ASTF profile driver. - Polling could eliminate the time overhead value. + But polling proved to introduce some loss, affecting the results. + Handle duration stretching in ASTF by stopping traffic. + The stop has several steps so that: + The traffic is really stopped entirely. + Late packets do not count (maybe as errors). + Stats are preserved to read for results (and cleared afterwards). + Several quantities added to ReceiveRateMeasurement: + Original target duration is preserved (algos need that). + Input estimate (tps) for early search iterations. + Output estimate (maybe pps) for MRR output. + Strict result (unsent counts as loss) for NDR. + Use L2 counters (opackets, ipackets) where possible. - TRex has trouble processing packets for the L7 ones at high loads. + Remove warmup from profile drivers and keywords. + Suites should call "Send ramp-up traffic" explicitly if needed. + Added parsing for few more counters. + Both to use in formulas or just for debug purposes. - Only 64B cases in autogen, framesize support to be added later. + Latency streams during search can be enabled via PERF_USE_LATENCY env var. + MLRsearch improvments: + Rename argument names to min_rate and max_rate. + Use relative receive rate in initial phase. + PLRsearch improvements: + Careful computation when output (pps) does not match input (tps). + Use geometric distribution (instead of Poisson). + Helps agains math errors. + This should improve estimate stability. - But in practice big losses still lead to significant jumps. + Traffic generator improvements: + send_traffic_on_tg now calls the full set_rate_provider_defaults. + _send_traffic_on_tg_internal for the logic without provider defaults. + As the internal function is re-used by measure() without affecting defaults. + Move _parse_traffic_results just before get_measurement_result. + As the latter uses fields set bu the former, it is now easier to read. + Multiple sources for approximate duration. + Tried from more precise to more available. + Includes logic for _pps tests (added in later change). + Move explicit type conversions to earlier occurences. + Profile driver output field uses semicolons to simplify parsing. + Performance Robot lib file split to several smaller ones. + performance_actions.robot: + Hosts Additional Statistics Action For * keywords. + performance_display.robot: + Hosts keyword for displaying and verifying results. + Change test message to use the correct unit (pps or cps). + performance_limits.robot renamed to performance_vars.robot + Added many keywords, mostly for accessing test variables. + Moved variables for Policer into a new keyword there. + Some keywords need sophisticated logic. - Other are basically Get Variable Value. + But in future more logic can be added, without editing callers. + Documentation for the new keywords acts as a documentation for test variables. + performance_utils.robot has the rest. + Eliminated arguments if the value is in test variable. + Small improvements to documentation. - Still not enough cleanup with respect to arguments and test variables. + Keywords are sorted alphabetically now in each one. + Suites: + Unified variables table: + No colons in comments. + ${n_hosts}, ${n_ports} and use them instead hardcoded numbers. + Add -cps to existing cps suite names. + Remove "trial data overwrite". + Compute max rate as in STL suites. + Each NAT suite has ip4base suite to compare results to. - Those act as indirect TRex calibration. - VPP does not lose packets in those. + Latency in ASTF suites is disabled hard. - As we do not support latency in ASTF profiles yet. + Unidirectional tests governed by suite variable, not an argument. + Write long argument lists vertically. + Prefer to use argument names. + In Python, also the last argument is followed by comma. + It makes renaming and reordering easier. + Similarly applies to prints with long lists of values. + A TODO to update api crc file comments. Change-Id: I84729355edbec051298a9de1162107f88ff5737d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-10-29job-spec: Add all NAT44 testsTibor Frank10-458/+1791
- the specs: - docs/job_specs/report_iterative/2n-clx/vpp-mlr-00.md - docs/job_specs/report_iterative/2n-clx/vpp-mrr-00.md - docs/job_specs/report_iterative/2n-skx/vpp-mlr-00.md - docs/job_specs/report_iterative/2n-skx/vpp-mrr-00.md take too long. We should consider to remove some tests. - the coverage specs were split to three parts Change-Id: I39e2032c76266e159a781d1225cff1a5119d4c85 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-29tests: Add GSO enabled testsMartin Balaz1-0/+4
Change-Id: I409b060f64ae7c6787448ae519fd76d8384e9ffb Signed-off-by: Martin Balaz <martin.balaz@pantheon.tech>
2020-10-28Report: Configure rls2009.45Tibor Frank1-0/+8
Change-Id: I8918f09411254b64abb65a165a0e284f735b306e Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-28Report: Add dataTibor Frank1-5/+0
Change-Id: I2394b42e177758a8f3ce4aeeb912b264cf3b457c Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-21Report: Configure 2009.44Tibor Frank1-0/+23
Change-Id: I7f7e50610e71af7f2b0dc1eed267a421ffd8aef0 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-19Tests: Remove GBPpmikus10-119/+8
- Specs - Tests - Library - TODO: PAL Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I8890940d56c7a9924d4a5f9365b6bb881610db14
2020-10-19Report: Add dataTibor Frank1-0/+5
Change-Id: I17c582de8fa24ffd55b163cfeb727f2951ec647f Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-16Report: Add dataTibor Frank1-0/+4
Change-Id: I82099d75e02369a6b8e3093563f8e8ec62c96c19 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-15Trending: Add csit-dpdk-perf-mrr-weekly-master-3n-tshTibor Frank2-0/+51
Change-Id: Icce2fc809d3f72a6447cc07700df3c6f324af267 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-15Report: Configure Report 2009.43Tibor Frank1-0/+10
Change-Id: I16ce4e2146fa44eace76fb495cc6d007005c215f Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-14Report: Fix static contentTibor Frank2-2/+3
Change-Id: I3ba5ea19efd6d8f878aa67bff9bcd237a7a366ff Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-14Spec: EPYC zn2pmikus3-0/+1083
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I2dd5602c129f3145ee6aa577929afdd5c2815fcd
2020-10-14report: updates to methodology section including nat44, acl, ipsecMaciek Konstantynowicz6-30/+225
Change-Id: I13464728d903cba14feedd3cfb78226d50f3d4a1 Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
2020-10-14Report: Fix bugs in release notesTibor Frank2-5/+5
Change-Id: Ib1175c8eb256136e9d4b5342043c2014d99631e6 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-14Report: Fix bugs in pdf version 2Tibor Frank3-22/+23
Change-Id: I2fb1c5593a72fda406a21b3f6cdebae6a1085817 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-14Report: vpp performance release notes updateJan Gelety3-66/+64
Change-Id: I8c7e3af80a976a252a4fd4df223ab6122decc201 Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-10-13Report: Edit PAL section in the release notesTibor Frank1-3/+3
Change-Id: I8c47ce084ae2cac981e1763b577bc18a406f709e Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-13Report: Add Data, Fix bugs in pdf versionTibor Frank9-135/+37
Change-Id: Ie880ee58313603104b2de9681305fc460939e96a Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-12Report: Add data, fix small bugsTibor Frank5-22/+28
Change-Id: I39981b53640e45c95db20747b61d8af34eb4ce86 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-09Report: Add NAT44 Latency graphsTibor Frank4-0/+578
Change-Id: I23ee9bef5631fde8403650caa96a14d9f920e055 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-09Report: Configure 2009.42Tibor Frank1-1/+1
Change-Id: Ic6f9ba720eb3ed964404789632278f211e576bb2 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-07docs: update arm hardware testbed specificationsJuraj Linkeš1-116/+53
* Remove decommissioned McBin testbed, but leave leftover mgmt IPs. * Mark SoftIrons in mgmt IPs section as decommissioned. * Update Taishan testbed. * Update arm build servers IPMI status (no longer out-of-order). * Update new ThunderX2 servers inventory details. Change-Id: I5f02479745be83158c5507fa740a5c7746090c13 Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2020-10-07Report: Add DNV comparison tables and detailsTibor Frank1-79/+77
Change-Id: I810f22a8bf4dc074e1c01d54fc4f8303b05aef22 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-07report: added description of CSIT environment ver. 5Maciek Konstantynowicz1-3/+14
Change-Id: Ic42bb1904d3cb7407c5d494f4ca53f0b6fd75d32 Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
2020-09-13Report: vpp-device static content updateJan Gelety2-3/+6
Change-Id: If78173c4b7fb4d5a0dedd55defeebf44b110be2a Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-10-06Report: Update test_environment_tgVratko Polak1-10/+16
+ Yaml formatting as used in code. + Mentioned core_mask=STLClient.CORE_MASK_PIN - No mention of ASTF specifics. Change-Id: I653cc383f21195275a92ff233768d92b340e684e Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-10-06Report: Add 2n-skx NAT44Tibor Frank4-0/+576
Change-Id: I15b371a93d7b1e80fab50bd5ffab83ada3a2fdb2 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-05Report: Add dataTibor Frank45-120/+92
- DNV VPP tests - 3n-tsh MRR 2005-5 - MRR Comparisons for skx, hsw and clx Change-Id: Ibb73e0f8234d2eb27b6a386675492efc9f050247 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-05Report: Fix NAT44Tibor Frank2-60/+60
Change-Id: I9c0b8571673a0fc839430a4dce2ef0e8b9b017e2 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-05Trending: Add NAT44Tibor Frank7-76/+476
Change-Id: I57d4b6f0ffa2ebfffbf95fc030bc8da70e46c8bc Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-05Report: Add NAT44Tibor Frank8-86/+618
Change-Id: I9bb8340e7921bce32b3bd51b8a54f33bc74f41ec Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-05Report: Add rls dataTibor Frank1-1/+1
Change-Id: I903075db92a71964f792d2de42a69ca93103e11d Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-02job-spec: Add report_iteratve/3n-hsw/hoststack-00.mdTibor Frank1-0/+41
Change-Id: I98ad76f5e1d1f9b4a7442aa9925ce68869123ac7 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-10-02jop-spec: Add NAT44 testsTibor Frank3-79/+459
Add all which we want to run and disable those which are not ready. Change-Id: I649ab290a1deec5d1c7de34a7bbe367d78a1e3ff Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-30Update 1n-tx2 testbed docs for new thx2 serversJieqiang Wang1-42/+103
Change-Id: I64e1161bd6accbb478cec5314710ac4a68beda29 Signed-off-by: Jieqiang Wang <jieqiang.wang@arm.com>
2020-09-30Report: Add dataTibor Frank1-60/+65
Change-Id: Id55530bb4e9118e73f3630103333b902d07510f6 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-29Report: Add vpp 2n-clx-cx556a, add dpdk 3n-tshTibor Frank12-24/+11
Change-Id: Ia5b0a70f430a4a26f07c1fa6b7506b839f3813bc Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-29Report: Edit the configurationTibor Frank1-3/+3
Change-Id: I3a3317903ee6e22fa0fe7fff9bf7d0208b19c8ae Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-28Report: Add RC2 dataTibor Frank1-1/+1
Change-Id: I6c4ae6b428571a392586f5b817c5392a0de8c727 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-24Report: replace snat by natTibor Frank2-16/+16
Change-Id: Ia329059955134c7fe30135a43bb1e1ec72b7bb6a Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-24Trending: Replace snat by natTibor Frank2-14/+14
Change-Id: Ifcb1311d46b7134502b142933fd1574eb94cc230 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-23Trending: Fix graph file nameTibor Frank1-3/+3
2n-clx-xxv710-64b-8t4c-ip4-features-snat44det-avf.html Change-Id: I93b39f9edf3b0d85b08cec97ad647e398da83207 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-23Fix: use nat44 not snat44Jan Gelety12-510/+510
Change-Id: If141566f76d5b2f309953233e693c34fef8ee12f Signed-off-by: Jan Gelety <jgelety@cisco.com>
2020-09-23Infra: Add 2n-xn2pmikus27-0/+4584
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I2ff6526b5209dabf70c6593f60162d9b8c147014
2020-09-22Trending: Hide 2n-clx-cx556aTibor Frank10-10/+28
Change-Id: I3616ea486c96534534422057e2f80da486ded2b8 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-21PAL: Fix show runtimeTibor Frank1-1/+1
Change-Id: Id42f087b2e3874a362f17596740feab6af8029d9 Signed-off-by: Tibor Frank <tifrank@cisco.com>
2020-09-18Added amd 2n-zen2 testbed specificationMaciek Konstantynowicz1-27/+118
Change-Id: I429c64f89d1c93945da375072f6f9cb702fa687f Signed-off-by: Maciek Konstantynowicz <mkonstan@cisco.com>
2020-09-18job-specs: add 3n-tsh dpdk mrr weekly setJuraj Linkeš1-0/+1
Change-Id: I0cd41129538b5e4ec47f3876813f7df36da2977a Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
2020-09-16Report: Add snat44det tests to graphsTibor Frank2-0/+86
Change-Id: I65b6b210d500c0f5aedd39e6c70c3c48b8654dfd Signed-off-by: Tibor Frank <tifrank@cisco.com>