| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Ia5a097797b54c2e71acdcc8d72706e5540536252
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ib52fab112d458decfecf39c77085bcd85f811eba
|
|
+ Model version 1.0.0.
- Only some result types are exported.
+ MRR, NDRPDR and SOAK.
- Other result types to be added later.
+ In contrast, all test types are detected.
+ Convert custom classes to JSON-serializable equivalents.
+ Sort dict keys before converting to JSON.
+ Override the order for some known keys.
+ Export sets as sorted arrays.
+ Convert to info content from serialized raw content.
+ Also export outputs for suite setups and teardowns.
+ Info files for setup/teardown exist only temporarily.
+ The data is merged into suite.info.json file.
+ This simplifies presentation of total suite duration.
+ Define model via JSON schema:
- Just test case, suite setup/teardown/suite to be added later.
- Just info, raw to be added later.
+ Proper descriptions.
+ Json is generated from yaml.
+ This is a convenience for maintainers.
+ The officially used schema is the .json one.
+ TODOs written into a separate .txt file.
+ Validate exported instance against the schema.
+ Include format checking.
+ Update CSIT requirements for validation dependencies.
+ This needs python-dateutil==2.8.2, only a patch bump.
+ Compute bandwidth also for soak tests.
+ This unifies with NDRPDR to simplify schema definition.
- PAL may need an update for parsing soak test message.
+ Include SSH log items, raw output only.
+ Generate all outputs in a single filesystem tree.
+ Move raw outputs into test_output_raw.tar.xz.
+ Rename existing tar with suites to generated_robot_files.tar.xz.
Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Implementation stub so checker can check already.
+ Also add documentation stub for the implemented model.
+ Checker checks also for bumps in documentation version.
- Not comparing implementation and documentation version yet.
Change-Id: I4d19c00315a1c171de325c4494c28f5210635f32
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
When building documentation using sphinx we see ~1200 similar warnings [0]
[0] - https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-verify-tox-master-ubuntu2004-x86_64/3289/doc_verify.log.gz
These warning are harmless and can be fixed later
Signed-off-by: Viliam Luc <vluc@cisco.com>
Change-Id: I1ac1099d38935971d47491dde905715345d3935c
|
|
+ Add ability to switch between hugepages.
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I84d8eae28ed414a32e5ba82e6c9ed10d7f0ef9cb
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ic249493a39faa8f429ae4fa1de644d74d874151b
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I81958fbf6ef240d53a0fb8708ca882baf02f606c
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I331fe9134c62594189ca231349ab4c5ba43b51e5
|
|
- enabling for fortville, columbiaville
- enabling experimental for mlx
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I1b7ceb54769f4a0089ac7309350499e60c5cca0a
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I5bbb573fb75d0ee7b5e9b21591e4d6ff48df917e
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I2f019a083916aec9f7816266f6ad5b92dcc31fa0
|
|
1. Suite steup add download nginx
2. Add nginx-1.14.2/1.15.0 ldp test suite
3. Add NginxUtils,NginxConfigGenerator method
4. Taskset the PID of nginx to the unused cores in VPP and these cores are under NIC's NUMA ID
5. cleanup add Kill Processes - nohup
Signed-off-by: xizhanx <xix.zhang@intel.com>
Change-Id: Idbf0e4ec3bf63e88281a8e3e34f52e00a6801c85
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I6edd980cb72111a008ae7fa19e1a4df279febdb2
|
|
+ PPS limit for AWS set to 1.2 Mpps.
+ The logic is very similar to that one in ASTF driver.
+ This helps for testbeds with high duration stretching (e.g. AWS).
+ Difference: No transaction scale, and we deal with floats.
+ Update loss counting to count unsent packets as lost.
+ Also count "unsent" transactions for other transaction types.
+ If nonzero, log the number of unsent packets/transactions.
+ Make STL and ASTF time overhead constant (called delay) configurable.
+ Subtract delay from approximated_duration, also for ASTF.
Change-Id: I6ee6aa6fba4f110ba1636e1b0ff76cac64383e33
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Additional configuration can provide performance boosts when running in virtual environments
If set to 0, uses default DPDK value
By tuning this value it's also possible to run T-Rex 2.88 with ENA NICs on AWS
Signed-off-by: Tomas Alexy <tomas.alexy@pantheon.tech>
Change-Id: I43c86ea1d9aa854a1087f07fe544ac77a5b80397
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I0341a1564ba510acf46bda3e24225209abef2f82
|
|
+ Mellanox 4.6 is not for Ubuntu 20.04
+ Mellanox for ubuntu 20.04 is 4.9+
+ T-Rex 2.86 is not for Mellanox 4.6+
+ T-Rex for Mellanox 5.2 is 2.88+
==================================
= Bump
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I902dfc2a43e6718b385e89f31a34260e09d61bd3
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Id56b87ab868f2897a6563914b0beca2acc25e706
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I145a4b5511141f1e2b4e387daa358e32dd2c8015
|
|
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
+ Update existing cps traffic profiles.
+ Add missing traffic profiles.
+ UDP:
+ Single burst of 32 packets was confirmed as safe enough for TRex.
+ Maybe 64 could work, but not enough testing for that.
+ Multiple bursts have lead to reduced TRex performance,
as overlaping bursts (from different client instances)
tend to fill up the buffers.
+ TCP:
+ Data size set to 11111 bytes, completely arbitrarily.
+ Results look reasonable, so I have kept that.
- MSS not set at all
- No tested support for frame size other than 64B.
- Frame size does not even factor into TCP profiles.
+ So other frame sizes are skipped in autogen.
+ Update tags in related suites.
- HOSTS_{n} and SRC_USER_{n} should be unified.
- Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
+ Resetter is called (if set) before each measurement.
+ If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
+ Transaction corresponds to profile.
+ TRex multiplier argument sets target rate in transactions per second.
+ The familiar STL traffic:
+ Bidirectional is considered to be 2 packets per transaction.
+ Unidirectional is considered to be 1 packet per transaction.
+ The newer ASTF traffic:
+ 4 subtypes, each has different number of packets per transaction.
+ For max rate computation:
+ Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
+ Use the server side value.
- Client side value is higher by an overhead.
- TRex is not sending traffic during that time.
+ Remove delays from traffic profiles.
- Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
+ Only for ASTF profiles, each ASTF profile has limited scale.
+ Scale defined in suite variables.
+ For TRex to send all transactions provided duration value is ignored.
+ The appropriate value is computed in TrafficGenerator.
+ An ad-hoc time constant is added to match the TRex client side time overhead.
+ The profile driver receives the computed duration.
+ Measurement for PLRsearch add a sleep if the computed duration is smaller.
+ Alternative argument for search algos if scale is limited.
+ Both need higher timeout to accomodate big scales.
+ MLRsearch can afford fewer phases.
+ Added a parameter to optionally shorten the duration.
+ Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
- Polling could eliminate the time overhead value.
+ But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
+ The stop has several steps so that:
+ The traffic is really stopped entirely.
+ Late packets do not count (maybe as errors).
+ Stats are preserved to read for results (and cleared afterwards).
+ Several quantities added to ReceiveRateMeasurement:
+ Original target duration is preserved (algos need that).
+ Input estimate (tps) for early search iterations.
+ Output estimate (maybe pps) for MRR output.
+ Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
- TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
+ Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
+ Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
+ Rename argument names to min_rate and max_rate.
+ Use relative receive rate in initial phase.
+ PLRsearch improvements:
+ Careful computation when output (pps) does not match input (tps).
+ Use geometric distribution (instead of Poisson).
+ Helps agains math errors.
+ This should improve estimate stability.
- But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
+ send_traffic_on_tg now calls the full set_rate_provider_defaults.
+ _send_traffic_on_tg_internal for the logic without provider defaults.
+ As the internal function is re-used by measure() without affecting defaults.
+ Move _parse_traffic_results just before get_measurement_result.
+ As the latter uses fields set bu the former, it is now easier to read.
+ Multiple sources for approximate duration.
+ Tried from more precise to more available.
+ Includes logic for _pps tests (added in later change).
+ Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
+ performance_actions.robot:
+ Hosts Additional Statistics Action For * keywords.
+ performance_display.robot:
+ Hosts keyword for displaying and verifying results.
+ Change test message to use the correct unit (pps or cps).
+ performance_limits.robot renamed to performance_vars.robot
+ Added many keywords, mostly for accessing test variables.
+ Moved variables for Policer into a new keyword there.
+ Some keywords need sophisticated logic.
- Other are basically Get Variable Value.
+ But in future more logic can be added, without editing callers.
+ Documentation for the new keywords acts as a documentation for test variables.
+ performance_utils.robot has the rest.
+ Eliminated arguments if the value is in test variable.
+ Small improvements to documentation.
- Still not enough cleanup with respect to arguments and test variables.
+ Keywords are sorted alphabetically now in each one.
+ Suites:
+ Unified variables table:
+ No colons in comments.
+ ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
+ Add -cps to existing cps suite names.
+ Remove "trial data overwrite".
+ Compute max rate as in STL suites.
+ Each NAT suite has ip4base suite to compare results to.
- Those act as indirect TRex calibration.
- VPP does not lose packets in those.
+ Latency in ASTF suites is disabled hard.
- As we do not support latency in ASTF profiles yet.
+ Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
+ Prefer to use argument names.
+ In Python, also the last argument is followed by comma.
+ It makes renaming and reordering easier.
+ Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.
Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
The previous 60 Mpps stil leads to ~30% duration stretching.
+ Add comment on why 36 Mpps was chosen as the new limit.
Change-Id: Ic11e8ece03939bdc8680cd7bc4122373583a2f17
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Unifying code structures
- To easily plug another DUT
+ New PCI PassThrough templates
+ Improved perf stat on cores allocated in test.
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I325f17b977314f93cb91818feddfddf3e607eb8a
|
|
+ Bump T-Rex version. We need new features for ASTF test.
+ Apply core pining. Results in a more stable performance.
+ Tweak the number of T-Rex workers.
+ We need an even value to achieve ymmetric performance with pinning.
+ Value 8 was selected as a best compromise.
This is a combination of 3 commits.
This is the 1st commit message:
T-Rex: 2.82
This is the commit message #2:
Change Trex to CORE_MASK_PIN mode to improve performance
https://trex-tgn.cisco.com/trex/doc/trex_stateless.html#_core_masking_per_interface
Above link have below explaination,
"When the profile is symmetric, performance can be improved by pinning half of
the cores to port 0, and half of the cores to port 1, thus avoiding cache
trashing and bouncing."
The reason to change this is that to run CSIT with 100G NIC often failed with
"TRex stateless runtime error timeout", it caused by Trex can not send enough
traffic within the fixed duration.
by change to CORE_MASK_PIN mode fix the issue.
Not editing ASTF, as that supports different options.
This is the commit message #3:
Experiment: Vary number of TRex workers
With CORE_MASK_PIN, we can get more predictable time distribution.
Decided to use 8 workers, that gives good results
both for high end (RDMA-core l2patch) and low end (vhost) tests.
Change-Id: I5c61127799e0624464e960fcb980ad1c4058e744
Signed-off-by: pmikus <pmikus@cisco.com>
Signed-off-by: Yulong Pei <yulong.pei@intel.com>
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
for Intel E810CQ 100G NIC, kernel driver of PF is ice, kernel driver of VF
is iavf, its VF hardware support VPP native avf driver.
Signed-off-by: Yulong Pei <yulong.pei@intel.com>
Change-Id: Ic8d86e5ee00057bbbcd09df619a38bd1371c8fd7
|
|
- continuation of https://gerrit.fd.io/r/c/csit/+/26898 as there was
reached limit of changes (1000)
Jira: CSIT-1711
- udp synthetic profiles w/o data packets
- udp cps perf tests, phase I (no special "search cps" KW)
Part I means that we are using MRR tests to collect traffic data
until there is ready new CPS test type with corresponding algorithm.
Change-Id: I0d30feb9ecf1d0bff937152656f8eb422f831378
Signed-off-by: Jan Gelety <jgelety@cisco.com>
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I1b535ea61ab68f6e37989ffc942979cdfd24f55e
|
|
Change-Id: I3bbe1fe0073ddeead5219993675f24955e8c3dfd
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
- test/suite/global
- binary logic is not working
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: Ia3d81cbf2c5f04d1093a0a408c84a9ffc6f3eef0
|
|
Change-Id: If61783fb717757c6189f06924412bd079e15a08f
Signed-off-by: Tibor Frank <tifrank@cisco.com>
|
|
for example, below will trigger VPP with default code path test in CSIT,
"csit-2n-clx-perftest mrrAND64bANDnic_intel-xxv710ANDeth-l2bdbasemaclrnNOTdrv_avf"
and below will trigger VPP with avx512 code path test in CSIT,
"csit-2n-clx-perftest icl mrrAND64bANDnic_intel-xxv710ANDeth-l2bdbasemaclrnNOTdrv_avf"
Change-Id: Id3abf932203ce00f34f8643c5bc4d5fc4d3e6efb
Signed-off-by: Yulong Pei <yulong.pei@intel.com>
|
|
This reverts commit 925c5bfcffb2bc6b833a8e64b7f3a06bbca52197.
It was affecting 2005 performance, confusing comparisons.
The revert includes the RDMA limit.
Change-Id: I9c31b061ee59bc5cb14d623ac98a65763ef75f1c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Rework BASH scripts (more code to python).
+ Move BASH into libraries.
+ Allows RDMA usage.
+ Fix 9000B tests.
+ Rename confusing l2fwd -> testpmd.
+ Fix suite setup.
+ Fix PCI whitelist to not accidentally pickup wrong interface.
+ Fix deprecated DPDK cli arguments.
- MLX5 jumbo are disabled on NIC (i will increase separately).
https://jenkins.fd.io/job/csit-dpdk-perf-verify-master-2n-clx/6/console (l3fwd still broken)
- MLX5 IMIX seems to be some TRex issue with IMIX for mlx5 (i will handle separately)
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I31d1b67305fa247cb5e1f57e739d3ef30dc1a04b
|
|
- MLX, ENA
- Part I
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I9f4b407770515fe6009ef5dfdba2b27fa2ff654f
|
|
+ Merge single/double link
+ Introduce _pf{n}[0] variables so we can access physical function
same way as virtual function
+ Cleanup code by moving complex logic to python
+ Prepare code for multiple vf functions
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Ic2e74a38bfa146441357de8f0916aeb638941c49
|
|
The previous values lead to duration stretching,
which affects MRR results (making receive rate bigger than it really is).
+ Move timestamp for starting of traffic,
the previous position was too conservative.
Change-Id: Ie983f0d3eacf56d57caa551d1b7fbb87ece6c158
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Modified cpuutils numa to work with aws 18xlarge instances.
Added dmidecode package to calibration role as is not installed by default.
Added required constants
Signed-off-by: Maros Mullner <maros.mullner@pantheon.tech>
Change-Id: I99608d9df3cff0b085ad26c30e339216b17581eb
|
|
- There is currently no other way to support multiple version of
playbooks.
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I2ade4a57d726b793e422996e83bf767ac4315488
|
|
regardless of ports state.
Signed-off-by: Maros Mullner <maros.mullner@pantheon.tech>
Change-Id: Iaafb5ff3a53a290ca79506bdb559a32d39570482
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Ia5c1e40b952c14775f2f038d3a8a293335172f47
|
|
Change-Id: I53425f57fe9ecef9cff2c94642cc7cb24537a961
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
|
|
Change-Id: I46bd4b58bdd02a3deed38c158214e1e9977dde05
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Change-Id: Icff49fb31cce342a2a4ae799e844ec91f9e5e366
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
|
|
+ Include a script to add suite tags to many suites at once.
+ Add suite tags also to device tests (not covered by autogen).
Change-Id: I514ee6178e22999b43460028fe2696738b012f04
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Disallowed -avf- (or -rdma-) as "template" suites.
+ GBP suite switched to DPDK driver in repo.
+ Each NIC has its own list of supported drivers, in Constants.
+ Updated tag expressions for daily jobs:
+ Feature, ipsec, memif, scale, srv6, tunnels, vhost and vts
are tested only with vfio-pci.
+ Other (base, dot1q, dot1ad) tested with all drivers.
+ Setup actions currently depend on driver, generated.
- The performance_rdma action is trivial for now.
- Several tests fail, to be fixed later, e.g. by performance_rdma.
+ Reconf tests are also supported.
+ Added DRV_VFIO_PCI tags missing, mainly in density tests.
- Vhost suites (density, reconf) are failing, but suites look good.
- TCP suites do not support NIC drivers yet.
- DPDK obviously not supported.
+ Use Python 3 in regenerate scripts.
+ Fix typos binded => bound.
+ File open modes set either u"rt" or u"wt" everywhere.
+ Remove a trailing space in an environment variable name.
Change-Id: I290470675dc5c9e88b2eaa5ab6285ecd9ed7827a
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
- DUT only topology (hoststack test
apps are co-located with vpp)
- Make vpp app specific keywords generic
where applicable
- Add IP4 Prefix to topology file
- Support running wrk in linux namespace
- Refactor namespace cleanup
- Remove redundant namespace creation code
- Refactor test/keyword dirs: tcp -> hoststack
- Add hoststack utility keywords
- Refactor wrk suite setup/teardown
- Update tests with recent perf infra changes
Change-Id: Ia1cf07978d579393eef94923819a87c8c1f36f34
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Icaa99e0a71a19ba23697c21a4b5aa046813a718a
|
|
Since https://gerrit.fd.io/r/c/csit/+/24056
csit-vpp perf tests are failing.
Turns out the reason is too large limit_memory value,
but other improvements applied during investigation
are included as well.
+ Cat trex.log if trex fails to start.
+ Improve trex startup command formatting (remove double spaces).
+ Reorder imports.
Change-Id: I2c1106ea6f88a1a275682e73eba212d08c7947c8
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I2b1a0d1455ae74a2ad4fdb553286aef5942f2a99
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Ia9832e5e65dbcf2d4f99c52fd0570d456262ab9a
|
Peter Mikus
Vratko Polak
Viliam Luc
xizhanx
Tomas Alexy
Yulong Pei
Jan Gelety
Tibor Frank
Maros Mullner
Dave Wallace