| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
- No support for IMIX.
+ Fix a bad bug in padding (most ASTF profiles had wrong frame sizes).
+ Fix a big typo in TCP PPS profiles (s->c was not data, just RST).
+ Control transaction size via ASTF_N_DATA_FRAMES env variable.
- Default value 5 leads to transactions smaller than before.
+ It ensures transaction is one burst (per direction) even for jumbo.
+ Edit autogen to set supported frame sizes based on suite id.
+ Both TCP and UDP use the same values:
+ 64B for CPS (exact for UDP, nominal for TCP).
+ 100B, 1518B and 9000B for TPUT and PPS.
- TCP TPUT achievable minimum is 70B.
+ Used 100B to leave room for possible IPv6 ASTF tests.
+ Separate function for code reused by vpp and trex tests.
- I do not really like the new "copy and edit" approach added here.
+ But it is a quick edit, better autogen refactor is low priority.
+ Consider both established and transitory sessions as valid.
- Mostly for compatibility with 2202 behavior and to avoid ramp-ups.
- Assuming both session states have similar enough VPP CPU overhead.
+ Added a TODO to investigate and maybe reconsider later.
+ Update the state timeout value to 240s.
+ That is the default for TCP (for transitory state).
- UDP could keep using 300s.
+ But I prefer UDP and TCP to behave as similarly as possible.
+ Use TRex tunables to get the exact frame size (for data packets).
- It is not clear why the recipe for MSS has to be this complicated.
+ Move code away from profile init, as frame size is not known there.
+ Change internal profile API, so values related to MSS are passed.
+ Lower ramp-up rate for TCP TPUT tests.
+ Because without lower rate, jumbo fails on packet loss in ramp-up.
+ UDP TPUT ramp-up rate also lowered (just to keep suites more similar).
+ Distinguish one-direction and aggregated average frame size.
+ Update keyword documentation where the distiction matters.
+ One-direction is needed for turning bandwidth limit to TPS limit.
+ Aggregated is needed for correct NDRPDR bandwidth result value.
- TCP TPUT will always be few percent below bidirectional maximum.
+ That is unavoidable, as one direction sends more control packets.
+ Add runtime consistency checks so future refactors are safer.
+ Fail if padding requested would be negative.
+ Fail if suite claims unexpected values for packets per transaction.
+ Edit the 4 types of ASTF profiles to keep them similar to each other.
+ Move UDP TPUT limit value from a field back to direct argument.
+ Stop pretending first UDP packet is not data.
+ Apply small improvements where convenient.
+ Replace "aggregate" with "aggregated" where possible.
+ To lower probability of any future typos in variable names.
+ Avoid calling Set Numeric Frame Sizes twice.
+ Code formatting, keyword documentation, code comments, ...
+ Add TODOs for less important code quality improvements.
- Postpone updating of methodology pages to a subsequent change.
Change-Id: I4b381e5210e69669f972326202fdcc5a2c9c923b
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Model version 1.0.0.
- Only some result types are exported.
+ MRR, NDRPDR and SOAK.
- Other result types to be added later.
+ In contrast, all test types are detected.
+ Convert custom classes to JSON-serializable equivalents.
+ Sort dict keys before converting to JSON.
+ Override the order for some known keys.
+ Export sets as sorted arrays.
+ Convert to info content from serialized raw content.
+ Also export outputs for suite setups and teardowns.
+ Info files for setup/teardown exist only temporarily.
+ The data is merged into suite.info.json file.
+ This simplifies presentation of total suite duration.
+ Define model via JSON schema:
- Just test case, suite setup/teardown/suite to be added later.
- Just info, raw to be added later.
+ Proper descriptions.
+ Json is generated from yaml.
+ This is a convenience for maintainers.
+ The officially used schema is the .json one.
+ TODOs written into a separate .txt file.
+ Validate exported instance against the schema.
+ Include format checking.
+ Update CSIT requirements for validation dependencies.
+ This needs python-dateutil==2.8.2, only a patch bump.
+ Compute bandwidth also for soak tests.
+ This unifies with NDRPDR to simplify schema definition.
- PAL may need an update for parsing soak test message.
+ Include SSH log items, raw output only.
+ Generate all outputs in a single filesystem tree.
+ Move raw outputs into test_output_raw.tar.xz.
+ Rename existing tar with suites to generated_robot_files.tar.xz.
Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
It should be in transactions per second, so target rate (min rate).
Change-Id: I56a12d65466320fe5e4b2ee7d3b9946c9b3c414f
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Change some method names and argument types:
+ Do not mention NDR and PDR, except as examples.
+ Return list of ReceiveRateInterval instead of NdrPdrResult.
+ The resulting intervals can be degenerate when hitting min/max rate.
+ Rename quantity name parts from "fraction" to "ratio".
+ Intervals are no longer tracked for each target ratio.
+ They are found dynamically from known results.
+ Add effective_loss_ratio field to avoid loss inversion effects.
+ Move some functions to separate files.
+ Bound search logic moved to MeasurementDatabase.py
+ ProgressState moved to its file.
+ WidthArithmetics.py holds small computation functions.
+ Use parameter expansion_coefficient instead of "doublings".
+ Do uneven bisect to save time when width is not power of two times goal.
+ Timeout now correctly tracked for the whole search,
not just the current phase.
+ Make logging (debug) function pluggable.
+ Added debug log messages for initial phase.
+ Do not mark as subclass if contructor signature differs.
+ Avoid re-measure on scale-limited (ASTF) profiles.
+ Remove outdated comments.
+ Bump copyright years.
Change-Id: I93f693b4f186f59030ee5ac21b78acc890109813
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Add UDP_CPS, TCP_CPS, UDP_PPS and TCP_PPS suites.
+ Update existing cps traffic profiles.
+ Add missing traffic profiles.
+ UDP:
+ Single burst of 32 packets was confirmed as safe enough for TRex.
+ Maybe 64 could work, but not enough testing for that.
+ Multiple bursts have lead to reduced TRex performance,
as overlaping bursts (from different client instances)
tend to fill up the buffers.
+ TCP:
+ Data size set to 11111 bytes, completely arbitrarily.
+ Results look reasonable, so I have kept that.
- MSS not set at all
- No tested support for frame size other than 64B.
- Frame size does not even factor into TCP profiles.
+ So other frame sizes are skipped in autogen.
+ Update tags in related suites.
- HOSTS_{n} and SRC_USER_{n} should be unified.
- Questionable clarification on difference between IP4BASE and SCALE.
+ Add NAT state resetters to tests that need them.
+ Resetter is called (if set) before each measurement.
+ If ramp-up is detected, resetter is not set.
+ Rename "mult" argument to "multiplier".
+ Abstracted from packets to transactions.
+ Transaction corresponds to profile.
+ TRex multiplier argument sets target rate in transactions per second.
+ The familiar STL traffic:
+ Bidirectional is considered to be 2 packets per transaction.
+ Unidirectional is considered to be 1 packet per transaction.
+ The newer ASTF traffic:
+ 4 subtypes, each has different number of packets per transaction.
+ For max rate computation:
+ Packets in the more numerous direction are considered.
+ Rely on TRex reported traffic duration for ASTF:
+ Use the server side value.
- Client side value is higher by an overhead.
- TRex is not sending traffic during that time.
+ Remove delays from traffic profiles.
- Those delays would increase the reprted traffic time.
+ Support for scale lmited trials.
+ Only for ASTF profiles, each ASTF profile has limited scale.
+ Scale defined in suite variables.
+ For TRex to send all transactions provided duration value is ignored.
+ The appropriate value is computed in TrafficGenerator.
+ An ad-hoc time constant is added to match the TRex client side time overhead.
+ The profile driver receives the computed duration.
+ Measurement for PLRsearch add a sleep if the computed duration is smaller.
+ Alternative argument for search algos if scale is limited.
+ Both need higher timeout to accomodate big scales.
+ MLRsearch can afford fewer phases.
+ Added a parameter to optionally shorten the duration.
+ Use short duration for runtime stats trial and failure stats trial.
+ Use very large keepalive values in udp profiles to avoid ka packets.
+ No polling in ASTF profile driver.
- Polling could eliminate the time overhead value.
+ But polling proved to introduce some loss, affecting the results.
+ Handle duration stretching in ASTF by stopping traffic.
+ The stop has several steps so that:
+ The traffic is really stopped entirely.
+ Late packets do not count (maybe as errors).
+ Stats are preserved to read for results (and cleared afterwards).
+ Several quantities added to ReceiveRateMeasurement:
+ Original target duration is preserved (algos need that).
+ Input estimate (tps) for early search iterations.
+ Output estimate (maybe pps) for MRR output.
+ Strict result (unsent counts as loss) for NDR.
+ Use L2 counters (opackets, ipackets) where possible.
- TRex has trouble processing packets for the L7 ones at high loads.
+ Remove warmup from profile drivers and keywords.
+ Suites should call "Send ramp-up traffic" explicitly if needed.
+ Added parsing for few more counters.
+ Both to use in formulas or just for debug purposes.
- Only 64B cases in autogen, framesize support to be added later.
+ Latency streams during search can be enabled via PERF_USE_LATENCY env var.
+ MLRsearch improvments:
+ Rename argument names to min_rate and max_rate.
+ Use relative receive rate in initial phase.
+ PLRsearch improvements:
+ Careful computation when output (pps) does not match input (tps).
+ Use geometric distribution (instead of Poisson).
+ Helps agains math errors.
+ This should improve estimate stability.
- But in practice big losses still lead to significant jumps.
+ Traffic generator improvements:
+ send_traffic_on_tg now calls the full set_rate_provider_defaults.
+ _send_traffic_on_tg_internal for the logic without provider defaults.
+ As the internal function is re-used by measure() without affecting defaults.
+ Move _parse_traffic_results just before get_measurement_result.
+ As the latter uses fields set bu the former, it is now easier to read.
+ Multiple sources for approximate duration.
+ Tried from more precise to more available.
+ Includes logic for _pps tests (added in later change).
+ Move explicit type conversions to earlier occurences.
+ Profile driver output field uses semicolons to simplify parsing.
+ Performance Robot lib file split to several smaller ones.
+ performance_actions.robot:
+ Hosts Additional Statistics Action For * keywords.
+ performance_display.robot:
+ Hosts keyword for displaying and verifying results.
+ Change test message to use the correct unit (pps or cps).
+ performance_limits.robot renamed to performance_vars.robot
+ Added many keywords, mostly for accessing test variables.
+ Moved variables for Policer into a new keyword there.
+ Some keywords need sophisticated logic.
- Other are basically Get Variable Value.
+ But in future more logic can be added, without editing callers.
+ Documentation for the new keywords acts as a documentation for test variables.
+ performance_utils.robot has the rest.
+ Eliminated arguments if the value is in test variable.
+ Small improvements to documentation.
- Still not enough cleanup with respect to arguments and test variables.
+ Keywords are sorted alphabetically now in each one.
+ Suites:
+ Unified variables table:
+ No colons in comments.
+ ${n_hosts}, ${n_ports} and use them instead hardcoded numbers.
+ Add -cps to existing cps suite names.
+ Remove "trial data overwrite".
+ Compute max rate as in STL suites.
+ Each NAT suite has ip4base suite to compare results to.
- Those act as indirect TRex calibration.
- VPP does not lose packets in those.
+ Latency in ASTF suites is disabled hard.
- As we do not support latency in ASTF profiles yet.
+ Unidirectional tests governed by suite variable, not an argument.
+ Write long argument lists vertically.
+ Prefer to use argument names.
+ In Python, also the last argument is followed by comma.
+ It makes renaming and reordering easier.
+ Similarly applies to prints with long lists of values.
+ A TODO to update api crc file comments.
Change-Id: I84729355edbec051298a9de1162107f88ff5737d
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
Vratko Polak