Age | Commit message (Collapse) | Author | Files | Lines |
|
This is mainly for easy detection of known issues,
so only new or unsure failures need closer investigation.
Change-Id: I6ebae2a0917ad229f9cfef5e56614b0a0ca73474
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
This semi-automatic script saves time for RCA after release.
Change-Id: Idda3de00635b3c16b71df024616f1e2e94d7c1d6
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
- Due to divergence from original design path the RAW was never
consumed. It adds too much code complexity and requires processing
on both storage and compute. Removing entirely to make modeling
efficient.
- log (apparently SSH) section will never be consumed in the way it is
coded in model. This section is also not part of model schema itself
due to the point above.
- Introducing telemetry section that is going to carry telemetry
items required for CDash.
Signed-off-by: pmikus <peter.mikus@protonmail.ch>
Change-Id: I7e0256c6c9715de8ee559eed29dce96329aac97d
|
|
+ Model version 1.0.0.
- Only some result types are exported.
+ MRR, NDRPDR and SOAK.
- Other result types to be added later.
+ In contrast, all test types are detected.
+ Convert custom classes to JSON-serializable equivalents.
+ Sort dict keys before converting to JSON.
+ Override the order for some known keys.
+ Export sets as sorted arrays.
+ Convert to info content from serialized raw content.
+ Also export outputs for suite setups and teardowns.
+ Info files for setup/teardown exist only temporarily.
+ The data is merged into suite.info.json file.
+ This simplifies presentation of total suite duration.
+ Define model via JSON schema:
- Just test case, suite setup/teardown/suite to be added later.
- Just info, raw to be added later.
+ Proper descriptions.
+ Json is generated from yaml.
+ This is a convenience for maintainers.
+ The officially used schema is the .json one.
+ TODOs written into a separate .txt file.
+ Validate exported instance against the schema.
+ Include format checking.
+ Update CSIT requirements for validation dependencies.
+ This needs python-dateutil==2.8.2, only a patch bump.
+ Compute bandwidth also for soak tests.
+ This unifies with NDRPDR to simplify schema definition.
- PAL may need an update for parsing soak test message.
+ Include SSH log items, raw output only.
+ Generate all outputs in a single filesystem tree.
+ Move raw outputs into test_output_raw.tar.xz.
+ Rename existing tar with suites to generated_robot_files.tar.xz.
Change-Id: I69ff7b330ed1a14dc435fd0ef008e753c0d7f78c
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Signed-off-by: pmikus <pmikus@cisco.com>
Change-Id: I909942dbb920df7f0fe15c0c92cda92c3cd8d8ad
|
|
It got broken by the previous reservation fix.
Change-Id: I5d9091b7fa4fcc37148c0223fe8eb83f7e5b6770
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Change-Id: I136cc51cbd5a01451e04a39c5d8e5554d6d39305
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Disallowed -avf- (or -rdma-) as "template" suites.
+ GBP suite switched to DPDK driver in repo.
+ Each NIC has its own list of supported drivers, in Constants.
+ Updated tag expressions for daily jobs:
+ Feature, ipsec, memif, scale, srv6, tunnels, vhost and vts
are tested only with vfio-pci.
+ Other (base, dot1q, dot1ad) tested with all drivers.
+ Setup actions currently depend on driver, generated.
- The performance_rdma action is trivial for now.
- Several tests fail, to be fixed later, e.g. by performance_rdma.
+ Reconf tests are also supported.
+ Added DRV_VFIO_PCI tags missing, mainly in density tests.
- Vhost suites (density, reconf) are failing, but suites look good.
- TCP suites do not support NIC drivers yet.
- DPDK obviously not supported.
+ Use Python 3 in regenerate scripts.
+ Fix typos binded => bound.
+ File open modes set either u"rt" or u"wt" everywhere.
+ Remove a trailing space in an environment variable name.
Change-Id: I290470675dc5c9e88b2eaa5ab6285ecd9ed7827a
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ CSIT will only use packages installed from master
PIP requirement.txt file
+ All CSIT subsequent code must use single function
and PIP list for virtualenv
- Report/PAL/doc/trending will follow in next patch
Change-Id: Ia9150fb05a144182a5a163178d553370ede55cdd
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
Change-Id: I96f40fa816055a656209cbbcae8259a74cc4bd80
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
This is the python3 change, the python2 is still used for testing bisect.
+ New version is 0.2.0 due to amount o API changes.
+ Jumpavg is now part of CSIT resource package tree.
+ Perpatch migrated to new jumpavg.
- PAL NOT updated (the update moved to a different Change).
Change-Id: I7d7a8bf8a411196c20c2a40a8c64478d6709bc07
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: Iae97cf4490fea38f5f03bdaa120ee7ae48b941ab
|
|
Change-Id: I1392c06b1d64f62b141d24c0d42a8e36913b15e2
Signed-off-by: Jan Gelety <jgelety@cisco.com>
|
|
Change-Id: Ib7f9dff7bede4f8ec3148f234109132be920de82
Signed-off-by: Tibor Frank <tifrank@cisco.com>
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I60d80564050cb786c7b1e0a08f31f8292f82197e
|
|
Change-Id: I464db6fb6c220f23f2fe69df2c793175975d7b72
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
+ Remove dependency on topo_ scripts that depends on custom SSH()
that depends on framework itself. This way the cleanup is independent
of failure in our SSH libs.
+ Simple ansible command can do cleanup of a machine:
ansible-playbook --inventory inventories/lf_inventory/hosts site.yaml \
--limit '10.32.8.18' --tags 'cleanup'
+ Add vpp_device reset and cleanup.
+ Remove historical scripts.
- Still in testing beta phase.
- Need to add SRIOV cleanup.
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I68e23304c7ad01041f51263c328c6e8d9b555cb7
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I344cbab78e1491b1dc0b64598f1eae23c8d2cdcf
|
|
If Jenkins interrupt pybot during AVF tests (that is happening
more frequently as coverage increased) the interfaces remains
in link UP state. Next run VPP cannot grab interfaces as they
are in UP state in and kernel driver. This results in testcase
failing.
Patch is supposed to do put interfaces in topology down so VPP
can bind them back to vfio-pci.
Change-Id: I2aa9e5944d39474965eac3c5f1b472f58e9aa8b5
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
Last attempt [0] is doing more harm then good. If testbed is down
but recovered quickly enough (reboot, temporary ssh connectivity
issue, ...) it never gets back into pool making other testbeds
overloaded and queued.
This patch is suppose to partially revert previous behavior until
proper patch will follow.
[0] https://gerrit.fd.io/r/c/csit/+/21148
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I8203946d10d3e7dd51e97519d679246b5dae59e3
|
|
Signed-off-by: Peter Mikus <pmikus@cisco.com>
Change-Id: I7a43c3183dc276fa5e4806f31a52cf5844792d4d
|
|
Previously, ssh-inaccesible testbeds
were handled the same way as reserved,
never ending the wait loop.
With this, if no testbed is accessible,
the job fails early.
Change-Id: I01bc79094fe0232a47d795e53e3daa52e8742bac
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
String "Unknown" is used if test run tag or url is not supplied,
so this change should be backward compatible.
+ Used "runtag", as both "test" and "tag" start with taken "-t".
+ Fixed some pylint violations.
+ Docstring explaining the steps taken and reasoning behind two files.
+ Other minor improvements.
Change-Id: If704d02e51374087fe39ad0f69432477d1b310d3
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Change-Id: I19be5008067067c2e7c78e7673a9ceaaf2f52c8f
Signed-off-by: Michal Cmarada <mcmarada@cisco.com>
|
|
CSIT-1485
- Measurement results are inconsistent, creating false positives.
- To avoid false positives, sensitivity has to be lowered.
+ Measurements are no more interleaved.
+ Measurement result are no longer filtered.
+ The job now runs faster.
- Ip6base regressions smaller than 4% are tolerated.
+ Ip6base false positives can happen, but less than 10% of the time.
- Part of job parameters is set on ci-management level.
+ Voter script output is improved.
Change-Id: I0ec7d5df8b397daf7ff0277a1137ee9f36d8d866
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
- Csit is migrating VAT scripts to PAPI, therefore
python package for vpp is needed in honeycomb tests.
Change-Id: I724cdd87e7ba899f9d9f95e08f5ce636435f7780
Signed-off-by: Michal Cmarada <mcmarada@cisco.com>
|
|
Change-Id: I492541a0a0d6cdc08f03e4c47fc86f22716cf065
Signed-off-by: Michal Cmarada <mcmarada@cisco.com>
|
|
Change-Id: I90c491e8039b20bbb0ef23bb1912e275030bef0b
Signed-off-by: Michal Cmarada <mcmarada@cisco.com>
|
|
As part of Bionic update for honeycomb (HC2VPP-372)
scripts need to be updated. Switching to Packagecloud instead
of nexus simplifies this process.
Change-Id: I243367e8c1d7c3de5890d6846c199563cef457bf
Signed-off-by: Michal Cmarada <mcmarada@cisco.com>
|
|
- Incredible why is it so hard to write scripts not hardcoding
anything.
Change-Id: I48cf2cc3d25f9665a9ff7ca26756d3b70cdd2f9c
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
Change-Id: I987dcd5092d8527c9aefbe093e1ed7ae144d191b
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
Change-Id: Ib14d989d5f44a2aeeec122e2f63a9832a213f920
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
Testing has showed SSH access to DUT affects performance.
Downside: No support for non-Linux TGs.
Complication: After merge there might be conflicts with older jobs,
including jobs for other branches.
Change-Id: Icefddc5a9ca8f4d830460e53a82bb45231f7222e
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
This is to reduce sensitivity on testbed unstable performance.
Also add topo_cleanupbefore every pybot run
to avoid issues with VPP uninstall.
TRACE prints are left there to simplify investigation
of false positives if/when they happen.
Change-Id: I9b0cdcfbbe4aa0735a0596746ac32c9e93af0897
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
Change-Id: Ib4e72dc8e3c0fedfcd702f4f97bc4f26cbc642e9
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
+ Scripts do not rely on (other) bootstraps.
+ Perf verify bootstrap is also the new style sript now.
+ Scripts are divided to functions for better re-use.
+ Functions are sourced from small number of large "library" files.
- Still using jumpavg from pipy.
- Perpatch has specific simplified parsing (instead of the PAL one).
- Bash style document is in a separate Change.
Change-Id: If88fa528ce155ea86b614e3d77c0550b91bbdf11
Signed-off-by: Vratko Polak <vrpolak@cisco.com>
|
|
- DPDK and VPP bootstrap merged into one (ligato will follow).
- Added more error detections
- Added topo-cleanup.py to cleanup TB before and after test.
- Remove installation of VPP
Change-Id: I7a86117eb03cbe4ee8fde47baeed58bc86c0dfb2
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
The hc2vpp-csit-verify allows to run HC CSIT tests per hc2vpp patch.
The job should not use most recent vpp artifacts
for respective branch ('RELEASE'),
but version specified in the vpp-version file.
The vpp-version file is used by hc2vpp-integration job
to produce honeycomb package.
Change-Id: If2811a01256e8217de50d220082dbc204443b54a
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
Ubuntu 14.04 packages are not being published since 1710.
Change-Id: I1372fe0eb4eab3c6c3e0f3a7ee435104f3c6f0fb
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
Honeycomb package does not require VPP debug symbols
and typical HC user does not have them installed.
Change-Id: I2cae502cad3216304d6f7ee44913f30c97fb5a20
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
The download_hc_build_pkgs.sh is used by
hc2vpp-csit-verify and hc2vpp-csit-verify-odl jobs.
These jobs download latest VPP packages
and unlike other HC CSIT jobs, build hc2vpp.
This patch removes vpp-dev/vpp-devel packages
from the list of installed VPP packages
to unify the environment with the other HC CSIT jobs.
Change-Id: I28742afda05cd85675619a92fbc10f4565bdb321
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
The vpp-dpdk-dkms is not needed for current VPP versions.
Change-Id: I9c1a79fb940b2ba0f332792d4fc088f49f9ee191
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
Change-Id: Ic382b625e5e7201ecc2a2937c15646806c594c61
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
Reverts https://gerrit.fd.io/r/#/c/12572/
Change-Id: Ic55016e3c257a81440332a212e4c0d83e1990668
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
- Rework VPP package download script
Change-Id: I51aca96fc3b834d8e17cf349a5a7b9d7105f2184
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
This reverts https://gerrit.fd.io/r/#/c/12209/
Change-Id: I1de4ce6e96644881aa14adf6765e8a67b286595d
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
Reverts https://gerrit.fd.io/r/#/c/11088/
Change-Id: Ib7b70d33f4a8110656dd39f25b7f5a67a2f52104
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
NSH artifacts in Nexus are outdated and crash VPP.
Reenable NSH after it is fixed (CSIT-994).
Change-Id: I3a0367682d62d5ccdbf008336ed682e7ab9032a0
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
Change-Id: Icd149927c6b2893f8f716df29d8de475e809be49
Signed-off-by: Peter Mikus <pmikus@cisco.com>
|
|
Since
https://gerrit.fd.io/r/#/c/10668/
and
https://gerrit.fd.io/r/#/c/10669/
honeycomb packages from stable branches
no longer depend on single VPP version.
This patch updates download_hc_pkgs.hc script,
used by hc2vpp-csit jobs, to use the latest honeycomb package
in such case.
Change-Id: If5b006e72caa8cc905137b1bf02af7d9e927e37a
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|