summaryrefslogtreecommitdiffstats
path: root/jjb/scripts
AgeCommit message (Collapse)AuthorFilesLines
2021-09-20FIX: Do not fail to copy directorypmikus1-5/+8
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I3cff2ca3feef7fc1083584249cdb2866e13ea836
2021-09-16Merge "FIX: Improve ContentEncoding handling"Dave Wallace1-10/+10
2021-09-16FIX: Improve ContentEncoding handlingpmikus1-10/+10
+ Each bucket has different settings that is accepted. This patch helps to make all buckets and jobs works as expected for their settings. + Tested on sandbox on all kind of jobs (tox,docs,trending) Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Ibf7e81f06c9bd9b3f17194231cbd4f7e83596305
2021-09-16CSIT: cleanup unused scriptspmikus5-181/+0
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I121b8fdf94ebbaa65ed999c9f907581ca7690210
2021-09-14Docs: Cleanuppmikus3-72/+119
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: I9209a4c739feb9aae713d0f3654d3cf6fa04c92d
2021-08-25Delete jjb/scripts/csit/cpta.shVratko Polak1-89/+0
It is not used since https://gerrit.fd.io/r/c/ci-management/+/33118 Change-Id: I62b167eff32fd27158e2dd7ae6deec2c25d8db47 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-08-20FIX: CSIT trending pathpmikus1-1/+1
Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: If7f9889fce320c2d53c7ff1c25d1a505e5f80eaf
2021-08-17Global: Docs generation in s3pmikus6-156/+142
+ Rework report/docs/trending jobs in CSIT to utilize AWS S3 cloudfront Signed-off-by: pmikus <pmikus@cisco.com> Change-Id: Ica2757ba8dc31792631c8d6b53b332ebf688b15e
2021-08-06Fix workarounds for archive pathsVratko Polak3-3/+38
https://gerrit.fd.io/r/c/ci-management/+/33060 has introduced workarounds for jobs which create Robot outputs outside /archives/. It got merged without verifying the workarounds work correctly. + The workaround for device job does not seem to be needed. + Perform the copy also when robot failed. Fixes: 37a55a05cd6ac7ca15bbb99f5ddc9ca61874d5f8 Change-Id: I5e08545bd290fa917e85a375865a130c3f2ff996 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2021-08-05Global: Rework archive artifactspmikus8-263/+296
This patch removes archive-artifacts-parameter macro and ARCHVIVE_ARTIFACTS env var from csit and vpp project. All project specific artifacts to be uploaded with the log files SHOULD BE copied to $WORKSPACE/archives. The next step once this is merged will be to remove NEXUS entirely via JCasC. + Remove archive-artifacts from all csit/vpp yaml files. + Add fdio-infra-ship-backup-logs macro + Remove unused jjb/include-raw-deploy-archives.sh + CSIT: - copy job artifacts to $WORKSPACE/archives + HC2VPP - remove CSIT + TLDK: - remove CSIT Signed-off-by: pmikus <pmikus@cisco.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: Iada020cf269714c34f9ce32d764d991827e3b003
2021-06-15Change 'Downloading' to 'Retrieving' in backup_upload_archives.shDave Wallace1-2/+2
- To prevent false positives when searching console logs for jobs which are downloading content over the internet. Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I0d9d1e3138380453eabab6a0fe2844311304491f
2021-06-03Merge "JJB: Include perf analysis per job"Dave Wallace3-1/+41
2021-06-03JJB: Include perf analysis per jobpmikus3-1/+41
+ Useful for performance analysis before run. This should help to analyze machine load before starting payload by comparing conditions from running jobs. - Requirements: sysstat (apt install sysstat) - Remove extraneous python import from backup_upload_archives.sh - Fix sar issue on centos-7 - Fix debian9 lftools install / upgrade issue Signed-off-by: pmikus <pmikus@cisco.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I6b61f99b9b2cde2bac3dacdefaa926e9afbb157e
2021-06-02api-checkstyle: do not send the notification about the successful API checksAndrew Yourtchenko1-4/+0
the api-checkstyle was initially done with notification for both successful and failed api-checkstyle runs, just as a matter of precaution. It's time to keep the notification only for the fail case, thus fulfilling the point of notifications. Change-Id: I41a12e7a30bc852c9280fc08a6eae426438f7d92 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-05-13Fix make test docs upload & clean up docs jobsDave Wallace1-1/+6
- make test DOCS_DIR in vpp/master changed location, fix upload for master and subsequent release branches. - Don't archive docs build output to avoid stress on nexus - Sphinx docs are generated and pushed to github by hand for publishing. Therefore remove sphinx docs jobs and include building of sphinx docs in the vpp-docs-verify-* jobs. Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I3b2badbf3e2b242ad6b4efcc7d6d281c8b26136b
2021-05-08Backup console logs to consul.Dave Wallace2-10/+27
- Add additional executor info to console output for consul log storage. Change-Id: Ifc9174ff69f26aa31f92e3680606f7a2303b5924 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-04-27Merge "Separate VPP stacktrace generation from artifact archiving"Vratko Polak1-21/+17
2021-04-23Separate VPP stacktrace generation from artifact archivingDave Wallace1-21/+17
- Separate removal of socket files in workspace archive directory from archiving of artifacts - Clean up deprecated workspace-archive-artifacts-parameter macro Change-Id: I11c7fe55d3412fb7261da9f0cfb809a152ae1d51 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-04-21Refactor ccache config to use per-os ccache dirs.Dave Wallace1-17/+11
- Simplify ccache state output to console log by using ccache stats command. - Bump max size of ccache dir to 10G (centos8 is approximately 5.7G) - Clean up extraneous ccache dir after docker build is complete. Change-Id: Ia2b55b7cd47212fce91b4567d0cca69d2ce892ed Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-04-16Fix backup archive directory structure.Dave Wallace1-18/+7
Change-Id: I9fd1194bdb68239df2a6e2c4432ec1849ca12aac Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-04-08Use fdio-infra-publish for nomad executor based jobsDave Wallace2-22/+47
- Separate post build executor info dump into its own script file. - Remove backup_upload_archives.sh from fdio-infra-shiplogs which is now only used for openstack executors. - Add setup_executor_env.sh to run first for all jobs which run on nomad/dockerhub images. Change-Id: I564201b8c578e6541152e6af74537d1d007ba1a7 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-04-06disable ASAN in the debug buildsAndrew Yourtchenko1-1/+3
The move to unix sockets has broken ASAN fairly thoroughly, so disable it Change-Id: Icf3822c5927309f8574865008173a66019a3ed55 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-03-26Merge "Include container id in console log."Vanessa Rene Valderrama2-4/+19
2021-03-26Include container id in console log.Dave Wallace2-4/+19
- Output executor runtime attributes in post_build_deploy_archives.sh for the case when a job fails prior to setup_executor_env.sh Change-Id: Icb54c039db21a621d44808221a3100c509583090 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-03-25Enable CCACHE in builder images / labelsDave Wallace1-0/+22
- Remove aarch64 labels which do not have any executor docker images. Change-Id: I1f46929b1c8393aa7a25bad215a95b85f2e00bd9 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-03-24Remove temporary DNS fix.Dave Wallace2-7/+2
- Use search domain in /etc/resolv.conf as hostname. Change-Id: Iaf156a2096a1b60f6222fa2d7f99101d72e2572c Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-03-23Force DNS nameserver in CI jobs to use local consul server.Dave Wallace1-0/+3
Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I76b7f4668ad0bfb11e9045de2d7f8e150132083a
2021-03-08Display DNS config in console logs.Dave Wallace1-0/+3
Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I2f7b549a903d3cd572a4711901ce1d138e922aa5
2021-02-23Merge "vpp: print "git diff" before running API checkstyle"Vratko Polak1-0/+3
2021-02-23vpp: print "git diff" before running API checkstyleAndrew Yourtchenko1-0/+3
API checkstyle is unhappy if the repo is not clean. An example: https://jenkins.fd.io/job/vpp-checkstyle-verify-master-ubuntu2004-x86_64/598/console Output the changes to make diagnostics easier. Change-Id: I3fe1dfd0adde47e2140151687a6ab31885fc851c Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-02-15Merge "Delete remains of DMM jobs"Dave Wallace1-42/+0
2021-02-15vpp: enable multi-worker make test on Debian10Andrew Yourtchenko1-0/+11
This patch enables the multi-worker make test on the branches that support it, as a voting job directly, to capture the working tests and avoid the new non-working multiworker testcases. Change-Id: Ibeedd8f8aa2851c022b045d37505bb75bf460de8 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-02-10vpp build.sh: move from "make verify" to "make pkg-verify" + "make test"Andrew Yourtchenko1-17/+36
Problem: The VPP CI merge and verify jobs use the "make verify", which has special handling depending on the OS. So, to make meaningful changes like running the test suite on more platforms, one would need to make the changes in two places, which is error-prone and slow. Solution: move the logic on which OS to run the "make test" into the ci-management scripts, and stop using "make verify" Change-Id: Icc7daa9364198f455ed8ca5bae5e5da5f15baf48 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-02-09Add the core file decoding+cleanup for the verify jobsAndrew Yourtchenko4-45/+185
- Add better failure reporting and dry run handling for build scripts. Change-Id: Ia19bae15ff4880b07094f4f665e5e00030eda27c Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-01-29backup upload archives: avoid gzip compressing the filesAndrew Yourtchenko1-2/+4
if they are already compressed Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Change-Id: Ie0ce0a303f3d681b8aae697c1b358534e68b2c28
2021-01-26Skip VPP build for merge jobs with Gerrit ID != git HEAD.Dave Wallace5-49/+101
- Since merge jobs are serialized after the patches have been merged into git, all merge jobs which do not have the same Gerrit ID as git HEAD are redundant. Skip them to save useless cycles and to reduce the size of the build queues when multiple patches are merged in a short amount of time. - This also fixes the issue with packagecloud packages containing different contents then the associated Gerrit ID when built & pushed in this state. Change-Id: I822492bd1d523576373bb268258c181a9c5edb42 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-01-08upload the logs to backup storage location as wellAndrew Yourtchenko1-0/+193
Change-Id: Ib3e9c9ae1fd053c06840f7bc5124aae48fe58e76 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-12-15Delete remains of DMM jobsVratko Polak1-42/+0
Change-Id: I45efc7227d4efd37e26cae50495306834ac17d0d Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-12-16Merge "delete the references to packagecloud.io .deb repositories after ↵Dave Wallace1-0/+3
installing vpp-ext-deps"
2020-12-16delete the references to packagecloud.io .deb repositories after installing ↵Andrew Yourtchenko1-0/+3
vpp-ext-deps If a new empty packagecloud.io/fdio/XXYY empty repository is being added during the stable/XXYY branch creation process, the jobs will fail unless a dummy artifact is added to this repository. Since we only use this repository to install the right version of vpp-ext-deps, delete the references after that portion is complete. Change-Id: Ia5dfa5bf0193c0af8671d561d27f9f02c4e641f4 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-12-15Use 'lftools deploy archives' for log uploadDave Wallace2-0/+61
- This is a pre-requisite for using amazon S3 for log storage. - Requires docker images built by updated docker scripts to include packages required by 'lftools' or other project specific docker images to be upgraded with same packages. This change only updates jobs which are using fdiotools docker images. - Add FD.io specific global macros to avoid inflating docker images with packages required by unneeded global-jjb scripts. - Sort VPP job parameters alphabetically - Dump installed python3 packages in setup_executor_env.sh Change-Id: Ifddd8f2b70b10acf834ffc837b27ef5453213a86 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-12-04run ASAN within the non-voting VPP debug verify jobAndrew Yourtchenko1-3/+3
Address sanitizer catches a wide range of failures like buffer overruns, etc. So makes sense to run it during the debug test run. However, it is a bit picky to the compiler - currently it only works well only on GCC 8+, so this patch accounts for that by moving the job to 20.04 executor which has gcc-9 by default. Thanks to Benoit for the help getting it working. Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Change-Id: If086907d1f20e99503aa0e811fd2ae95c168d731
2020-12-01Do not fail on base64 encoded commentsVratko Polak1-1/+2
+ Add job parameters for Sandbox testing. Change-Id: I3f8c088bd49b1926393a6fa9d3327dd6c7533656 Signed-off-by: Vratko Polak <vrpolak@cisco.com>
2020-11-12Fix package_cloud push of vpp-ext-depsDave Wallace1-3/+6
- Fix regression in previous change to packagecloud_push.sh Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: Idfb47bd49bc59467ba3f28499edcdd41a00d04ce
2020-11-10Fix packagecloud_push.sh upload failure for vpp-ext-depsDave Wallace1-3/+22
- When a new vpp-ext-deps version is created, the packagecloud_push.sh script will declare a failure for merge jobs if the package already has been pushed by another job running in parallel. Since vpp-ext-deps will be automatically rebuilt by the VPP build, there is no reason to fail the merge job if the packagecloud push fails for the vpp-ext-deps package. Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I96edc6948b7f8b654cc66154412e5eb00927266a
2020-11-04Install vpp-ext-deps rpms from local packages.Dave Wallace1-1/+12
- In setup_vpp_dpdk_dev_env.sh if the required vpp-ext-deps package version is available in /root/Downloads, then install the local package instead of downloading and installing it from packagecloud.io. Change-Id: Iffb8394288524732a2fd2795eec6324d85b17ef5 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-11-04Fix apt-cache show cmd failure on Debian-10Dave Wallace1-5/+8
- The version of apt-cache on debian-10 does not produce the same output as the versions in Ubuntu distros. Refactor extraction of vpp-ext-deps package name from apt-cache output. Change-Id: I92c297d91f6acf332be6a54bc1eaac83a4d3d7da Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-10-16Fix setup_vpp_ubuntu_docker_test.sh on vpp stable branchesDave Wallace1-2/+3
- Fix set -e bug on snergster executors for vpp stable/* branches introduced by https://gerrit.fd.io/r/c/ci-management/+/29209 Change-Id: Icb261cf0a198f9c76feca689e4013a5dd0247f8b Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-10-13Clean up vpp jenkins scripts & docker dirDave Wallace39-367/+452
- Remove docker/vpp Dockerfile definitions and colateral files for image now being build by https://gerrit.fd.io/r/c/ci-management/+/28022 - Remove opensuse support which has been elided from the vpp build in https://gerrit.fd.io/r/c/vpp/+/28999 - Remove unused scripts. - Output script name marker for all jjb scripts following LFIT practices. - Remove old/unused output from scripts. - Add logic to use vpp-ext-deps download cache in the executor images to speed up build jobs. - Add copyright notifcations to modified files without an existing copyright. - Make all vpp/docs jobs use '-{stream}-{os}-{executor}' suffix in their names. - Consolidate executor logging into setup_executor_env.sh and call it first for all jobs. Change-Id: I4ffb67ed476659f0127ca8c84f20a305d06149bf Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2020-10-12Promote vpp debian-9 jobs from beta to production.Dave Wallace1-0/+5
Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I99905d7a00b11d81715789af64774b375bd404d7