src/vnet/tcp/tcp_newreno.c

#!/usr/bin/env bash

# Copyright (c) 2023 Cisco and/or its affiliates.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at:
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -exuo pipefail

# This entry script does not change which CSIT branch is used,
# use "with_oper_for_vpp.sh" wrapper for that.
#
# This script is to be used for locating performance regressions
# (or breakages, or progressions, or fixes).
# It uses "git bisect" commands on the VPP repository,
# between the triggered VPP patch and a commit specified in the first argument
# of the gerrit comment text.
# The other arguments are used as tag expressions for selecting tests as usual.
# Many different result types are supported.
#
# Logs are present in the archive directory, but usually the main output
# is the offending commit as identified by "git bisect", visible in console.
#
# While selecting just one testcase is the intended use,
# this script should be able to deal with multiple testcases as well,
# grouping all the values together. This usually inflates
# the standard deviation, but it is not clear how that affects the bisection.
#
# For the bisection decision, jumpavg library is used,
# deciding whether shorter description is achieved by forcefully grouping
# the middle results with the old, or with the new ones.
# If the shortest description is achieved with 3 separate groups,
# bisect interval focuses on biggest relative change
# (with respect to pairwise maximum).
#
# If a test fails, an artificial result is used to distinguish
# from normal results. Currently, the value 1.0, with the multiplicity of 4.
#
# Note that if there was a VPP API change that affects tests in the interval,
# there frequently is no good way for single CSIT commit to work there.
# You can try manually reverting the CSIT changes to make tests pass,
# possibly needing to search over multiple subintervals.
# Using and older CSIT commit (possibly cherry-picking the bisect Change
# if it was not present in CSIT compatible with old enough VPP builds)
# is the fastest solution; but beware of CSIT-induced performance effects
# (e.g. TRex settings).
#
# If a regression happens during a subinterval where the test fails
# due to a bug in VPP, you may try to create a new commit chain
# with the fix cherry-picked to the start of the interval.
# Do not do that as a chain in Gerrit, it would be long and Gerrit will refuse
# edits of already merged Changes.
# Instead, add a block of bash code to do the manipulation
# on local git history between checkout and bisect.
#
# At the start, the script executes first bisect iteration in an attempt
# to avoid work if the search interval has only one commit (or is invalid).
# Only when the work is needed, earliest and latest commits are built
# and tested. Branches "earliest", "middle" and "latest" are temporarily created
# as a way to remember which commits to check out.
#
# Test results are parsed from json files,
# symlinks are used to tell python script which results to compare.
#
# Assumptions:
# + There is a directory holding VPP repo with patch under test checked out.
# + It contains csit subdirectory with CSIT code to use (this script is there).
# + Everything needed to build VPP is already installed locally.
# Consequences:
# + Working directory is switched to the VPP repo root.
# + At the end, VPP repo has checked out and built some commit,
#   as chosen by "git bisect".
# + Directories build_root, build and csit are reset during the run.
# + The following directories (relative to VPP repo) are (re)created:
# ++ csit_{earliest,middle,latest}, build_{earliest,latest},
# ++ archive, csit/archive, csit/download_dir.
# + Symlinks csit_{early,late,mid} are also created.
# Arguments:
# - ${1} - If present, override JOB_NAME to simplify manual usage.

# "set -eu" handles failures from the following two lines.
BASH_ENTRY_DIR="$(dirname $(readlink -e "${BASH_SOURCE[0]}"))"
BASH_FUNCTION_DIR="$(readlink -e "${BASH_ENTRY_DIR}/../function")"
source "${BASH_FUNCTION_DIR}/common.sh" || {
    echo "Source failed." >&2
    exit 1
}
source "${BASH_FUNCTION_DIR}/per_patch.sh" || die "Source failed."
# Cleanup needs ansible.
source "${BASH_FUNCTION_DIR}/ansible.sh" || die "Source failed."
common_dirs || die
check_prerequisites || die
set_perpatch_vpp_dir || die
get_test_code "${1-}" || die
get_test_tag_string || die
# Unfortunately, git bisect only works at the top of the repo.
cd "${VPP_DIR}" || die

# Save the current commit.
git checkout -b "latest"
# Save the lower bound commit.
git checkout -b "earliest"
git reset --hard "${GIT_BISECT_FROM}"

# This is the place for custom code manipulating local git history.

#git checkout -b "alter"
#...
#git checkout "latest"
#git rebase "alter" || git rebase --skip
#git branch -D "alter"

git bisect start || die
# TODO: Can we add a trap for "git bisect reset" or even "deactivate",
# without affecting the inner trap for unreserve and cleanup?
git checkout "latest"
git status || die
git describe || die
git bisect new || die
# Performing first iteration early to avoid testing or even building.
git checkout "earliest" || die "Failed to checkout earliest commit."
git status || die
git describe || die
# The first iteration.
git bisect old | tee "git.log" || die "Invalid bisect interval?"
git checkout -b "middle" || die "Failed to create branch: middle"
git status || die
git describe || die
if head -n 1 "git.log" | cut -b -11 | fgrep -q "Bisecting:"; then
    echo "Building and testing initial bounds."
else
    echo "Single commit, no work needed."
    exit 0
fi
# Building latest first, good for avoiding DPDK rebuilds.
git checkout "latest" || die "Failed to checkout latest commit."
build_vpp_ubuntu "LATEST" || die
set_aside_build_artifacts "latest" || die
git checkout "earliest" || die "Failed to checkout earliest commit."
git status || die
git describe || die
build_vpp_ubuntu "EARLIEST" || die
set_aside_build_artifacts "earliest" || die
git checkout "middle" || die "Failed to checkout middle commit."
git branch -D "earliest" "latest" || die "Failed to remove branches."
# Done with repo manipulation for now, testing commences.
initialize_csit_dirs "earliest" "middle" "latest" || die
set_perpatch_dut || die
select_topology || die
select_arch_os || die
activate_virtualenv "${VPP_DIR}" || die
generate_tests || die
archive_tests || die

# TODO: Does it matter which build is tested first?

select_build "build_earliest" || die
check_download_dir || die
reserve_and_cleanup_testbed || die
run_robot || die
move_test_results "csit_earliest" || die
ln -s -T "csit_earliest" "csit_early" || die

# Explicit cleanup, in case the previous test left the testbed in a bad shape.
ansible_playbook "cleanup"

select_build "build_latest" || die
check_download_dir || die
run_robot || die
move_test_results "csit_latest" || die
ln -s -T "csit_latest" "csit_late" || die
untrap_and_unreserve_testbed || die

# See function documentation for the logic in the loop.
main_bisect_loop || die
# In worst case, the middle branch is still checked out.
# TODO: Is there a way to ensure "middle" branch is always deleted?
git branch -D "middle" || true
# Delete symlinks to prevent duplicate archiving.
rm -vrf "csit_early" "csit_late" "csit_mid"
Age	Commit message (Collapse)	Author	Files	Lines
2019-04-08	host stack: update stale copyright	Florin Coras	1	-1/+1
	Change-Id: I33cd6e44d126c73c1f4c16b2041ea607b4d7f39f Signed-off-by: Florin Coras <fcoras@cisco.com>
2018-11-13	tcp: cubic fast convergence	Florin Coras	1	-0/+1
	Change-Id: I3a15960fe346763faf13e8728ce36c2f3bf7b05a Signed-off-by: Florin Coras <fcoras@cisco.com>
2018-11-09	tcp: basic cubic implementation	Florin Coras	1	-9/+1
	Because the code is not optimized, newreno is still the default congestion control algorithm. Change-Id: I7061cc80c5a75fa8e8265901fae4ea2888e35173 Signed-off-by: Florin Coras <fcoras@cisco.com>
2018-05-26	tcp: loss recovery improvements/fixes	Florin Coras	1	-1/+1
	- fix newreno cwnd computation - reset snd_una_max on entering recovery - accept acks beyond snd_nxt but less than snd_congestion when in recovery - avoid entering fast recovery multiple times when using sacks - avoid as much as possible sending small segments when doing fast retransmit - more event logging Change-Id: I19dd151d7704e39d4eae06de3a26f5e124875366 Signed-off-by: Florin Coras <fcoras@cisco.com>
2018-05-23	tcp: cc improvements and fixes	Florin Coras	1	-0/+2
	Change-Id: I6615bb612bcc3f795b5f822ea55209bb30ef35b5 Signed-off-by: Florin Coras <fcoras@cisco.com>
2018-04-20	tcp: make newreno byte instead of acks dependent	Florin Coras	1	-2/+8
	Should be more resilient to ack losses Change-Id: Icec3b93c1d290dec437fcc4e6fe5171906c9ba8a Signed-off-by: Florin Coras <fcoras@cisco.com>
2017-07-11	Horizontal (nSessions) scaling draft	Dave Barach	1	-2/+2
	- Data structure preallocation. - Input state machine fixes for mid-stream 3-way handshake retries. - Batch connections in the builtin_client - Multiple private fifo segment support - Fix elog simultaneous event type registration - Fix sacks when segment hole is added after highest sacked - Add "accepting" session state for sessions pending accept - Add ssvm non-recursive locking - Estimate RTT for syn-ack - Don't init fifo pointers. We're using relative offsets for ooo segments - CLI to dump individual session Change-Id: Ie0598563fd246537bafba4feed7985478ea1d415 Signed-off-by: Dave Barach <dbarach@cisco.com> Signed-off-by: Florin Coras <fcoras@cisco.com>
2017-06-19	Overall tcp performance improvements (VPP-846)	Florin Coras	1	-2/+2
	- limit minimum rto per connection - cleanup sack scoreboard - switched svm fifo out-of-order data handling from absolute offsets to relative offsets. - improve cwnd handling when using sacks - add cc event debug stats - improved uri tcp test client/server: bugfixes and added half-duplex mode - expanded builtin client/server - updated uri socket client/server code to work in half-duplex - ensure session node unsets fifo event for empty fifo - fix session detach Change-Id: Ia446972340e32a65e0694ee2844355167d0c170d Signed-off-by: Florin Coras <fcoras@cisco.com>
2017-06-09	Implement sack based tcp loss recovery (RFC 6675)	Florin Coras	1	-3/+17
	- refactor existing congestion control code (RFC 6582/5681). Handling of ack feedback now consists of: ack parsing, cc event detection, event handling, congestion control update - extend sack scoreboard to support sack based retransmissions - basic implementation of Eifel detection algorithm (RFC 3522) for detecting spurious retransmissions - actually initialize the per-thread frame freelist hash tables - increase worker stack size to 2mb - fix session queue node out-of-buffer handling - ensure that the local buffer cache vec_len matches reality - avoid 2x spurious event requeues when short of buffers - count out-of-buffer events - make the builtin server thread-safe - fix bihash template threading issue: need to paint -1 across uninitialized working_copy_length vector elements (via rebase from master) Change-Id: I646cb9f1add9a67d08f4a87badbcb117980ebfc4 Signed-off-by: Florin Coras <fcoras@cisco.com> Signed-off-by: Dave Barach <dbarach@cisco.com>
2017-05-07	Fix TCP loss recovery, VPP-745	Florin Coras	1	-1/+1
	Allows pure loss recovery retransmits only on timeout. Change-Id: I563cdbf9e7b890a6569350bdbda4f746ace0544e Signed-off-by: Florin Coras <fcoras@cisco.com>
2017-03-01	VPP-598: tcp stack initial commit	Dave Barach	1	-0/+93
	Change-Id: I49e5ce0aae6e4ff634024387ceaf7dbc432a0351 Signed-off-by: Dave Barach <dave@barachs.net> Signed-off-by: Florin Coras <fcoras@cisco.com>