Proposal to automate VPP-API "Flag Day"

Change-Id: I7bfce00762017b1aa8fd91ce1c8e4c0b810ee7b7
Signed-off-by: Dave Wallace <dwallacelf@gmail.com>

1 files changed, 224 insertions, 0 deletions

diff --git a/docs/automating_vpp_api_flag_day.rst b/docs/automating_vpp_api_flag_day.rst
new file mode 100755
index 0000000000..2e8612669e
--- /dev/null
+++ b/docs/automating_vpp_api_flag_day.rst
@@ -0,0 +1,224 @@
+..
+   Copyright (c) 2019 Cisco and/or its affiliates.
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at:
+..
+       http://www.apache.org/licenses/LICENSE-2.0
+..
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+
+AUTOMATING VPP-API FLAG DAY IN CSIT
+===================================
+
+**ABSTRACT**
+
+This document is intended to provide a solution to the problem of
+automating the detection of VPP API changes which are not backwards
+compatible with existing CSIT tests and to automate the "Flag Day"
+process of deploying a new set of CSIT tests which are compatible
+with the new version of the VPP API without causing a halt to the
+normal VPP/CSIT operational CI process. This shall initially be
+limited to changes in \*.api files contained in the vpp repo.
+Eventually the detection algorithm could be extended to include
+other integration points such as "directory" structure of stats
+segment or PAPI python library dependencies.
+
+**VPP API FLAG DAY ALGORITHM USE CASE**
+
+The following steps describe the use case of the proposed
+implementation for automating the VPP API "Flag Day" algorithm:
+
+#. A VPP patch with VPP API changes is submitted to
+   gerrit for review
+#. CSIT verify job detects the VPP API change(s).
+   Note: This requires a new CSIT test described below in the
+   section "CSIT VPP API CHANGE DETECTION TEST".
+#. CSIT verify job fails before starting to run any tests and
+   an email with the appropriate reason is sent to the VPP patch
+   submitter and vpp-api-dev@lists.fd.io including the VPP patch
+   information and API change(s) that were detected.
+#. The VPP patch developer and CSIT team create a CSIT JIRA ticket
+   to identify the work required to support the new VPP API version.
+#. CSIT developer creates a patch to existing CSIT tests to support
+   the new VPP API version and new features in the VPP patch as required.
+#. CSIT developer runs CI verification tests for the CSIT patch against
+   the VPP patch (aka "Patch-On-Patch verification).
+   This process verifies support for the new VPP API, associated VPP
+   features and CSIT tests.  Both developers iterate until the
+   verification passes.
+#. CSIT committer updates the supported API signature(s) such that
+   all signatures included in the CSIT branch are supported by the
+   current version of the CSIT tests. See "API FLAG DAY SCENERIOS" below
+   for examples.
+#. CSIT committer merges CSIT patch and creates a new operational
+   branch in the CSIT repo.
+#. VPP developer issues a recheck on the VPP patch and a VPP
+   Committer merges the patch.
+#. Recheck of existing VPP patches in gerrit may cause the "VPP
+   API Incompatible Change Test" to send an email to the patch
+   submitter to rebase the patch to pick up the compatible VPP API
+   version files.
+
+**FEATURES REQUIRED FOR IMPLEMENTATION**
+
+**VPP API SIGNATURE GENERATION**
+
+The VPP PAPI generation already produces the complete set of
+signatures in JSON format for all api files and includes them in the
+vpp-api-python.deb package.  Upon installation all of the \*.api.json
+files are installed in the /usr/share/vpp/api directory.  Each record
+in the .api.json file contains the name of the api message, the fields
+and their data types, and a CRC of the json object.
+
+**VPP API CLIENT SIGNATURES**
+
+In each CSIT branch, all of the VPP API client signatures that are supported
+by the CSIT tests in that branch are contained in separate directories
+under the .../csit/resources/api/vpp directory. The CSIT VPP API
+client signature directory structure is the same as the one published in
+/usr/share/vpp/api as generated by vppapigen.
+
+The granularity of the CSIT VPP API client signature support
+will initially be on a per VPP API JSON file.  In the future, a per VPP
+api message level of granularity may be added.  If CSIT is capable of
+supporting more than one version of a VPP API JSON file, then a new
+CSIT VPP api client signature directory will be created containing
+all of the supported VPP API JSON files.  Typically this will be identical
+to the previous version with the exception of the VPP API JSON files
+which have been changed in the VPP patch which triggered the VPP API FLAG
+day algorithm.
+
+See https://gerrit.fd.io/r/19027 for the baseline implementation.
+
+**VPP API CHANGES FILE INCLUDED in VPP PACKAGE**
+
+The VPP build system shall add a file in the /usr/share/vpp/api
+directory of the vpp package which is the same directory in which
+the api JSON files are published.  This file will include the list of
+VPP api files which were included in the patch to be verified.
+
+See https://gerrit.fd.io/r/19479 for the baseline implementation.
+
+**CSIT VPP API CHANGE DETECTION TEST**
+
+The set of VPP api signatures which are supported by the CSIT tests in
+a given CSIT branch shall be stored in .../csit/resources/api/vpp which
+mirrors the same directory structure as the API signature directory
+generated by vppapigen (e.g. /usr/share/api/vpp/core &
+/usr/share/api/vpp/plugins).
+
+The test compares the VPP patch's API signature directory with each of
+the CSIT VPP API signabture directory and determine the following state:
+
+- No Change
+- Changed
+- Rebase or Merge Parent VPP Patch [0]
+
+[0] The Rebase or Merge Parent VPP Patch result occurs when there is no valid API
+signature found in .../csit/resources/api/vpp AND there are no VPP API changes
+included in the patch.  This could be the result of a patch whose parent does not
+include the API changes merged in another VPP patch and supported by the new CSIT
+oper branch.  This case would be resolved by rebasing the patch to HEAD.  The other
+possibility is that the patch is a descendent of a patch with an incompatible API
+change that has not been merged yet.  This case is resolved by completing the API
+Flag Day algorithm on the parent patch such that the latest CSIT oper branch supports
+the API in the parent.  This importance of the detection of this state is to provide
+direct feedback to the VPP patch author about how to resolve the issue in a timely
+manner.
+
+Any condition other than "No Change" shall cause an email to be sent
+to the VPP patch submitter.  If the condition is "Changed" then
+vpp-api-dev@lists.fd.io shall also be copied on the notification email.
+
+**RUN CSIT VERIFY JOB AGAINST A SPECIFIC VPP PATCH IN GERRIT REVIEW BRANCH**
+
+This is the "Patch-On-Patch" methodology documented in [TBD]?
+
+
+**VPP API FLAG DAY SCENARIOS**
+
+In the beginning, let's assume there is a single VPP API Client signature
+directory in the current oper branch called vpp-api-client.sig.1 which
+contains core/vpe.api.json and plugin/acl.api.json which are supported
+by the CSIT tests.
+
+**VPP PATCH CONTAINS INCOMPATIBLE API CHANGES**
+
+Next, a VPP developer modifies vpe.api with a whole set of
+new type definitions.  When the patch is submitted to gerrit.fd.io, the
+"CSIT VPP API CHANGE DETECTION TEST" detects the changed api file and
+votes Verified -1.  Once CSIT has been updated to support the new type
+definitions and verified against the VPP patch,
+vpp-api-client.sig.1/core/vpe.api.json is replaced with the vpe.api.json
+file from the patch. The CSIT changes are committed into CSIT master and a
+new oper branch is created. The VPP patch is then rechecked and merged
+into VPP master as soon as practicable. All existing VPP patches and any
+new patches not including the VPP api change patch will fail verification
+with a "Rebase or Merge Parent" notification upon recheck or initial
+submission to gerrit.  Rebasing is then required in order to pass
+verification of the new api changes.
+
+**VPP PATCH CONTAINS BACKWARDS COMPATIBLE CHANGES**
+
+The next day, a VPP developer finds a need to add a new
+attribute to an api message in vpe.api with a default value defined.
+This is a backwards compatible change for CSIT.  Since the "CSIT VPP
+API CHANGE DETECTION TEST" only works on a per api file level of granularity,
+the change is flagged with Verified -1.  However, in this case, the
+CSIT developer can resolve the verify failure by adding a second VPP API
+client signature directory, vpp-api-client.sig.2 which is a copy of
+vpp-api-client.sig.1 with the vpe.api.json file updated with the contents
+of the copy from the VPP patch.  After the CSIT changes are merged and a new
+CSIT oper branch is created, the VPP patch will pass verification upon recheck.
+All other patches will continue to pass verification upon recheck or initial
+submission to gerrit by matching the signature in  vpp-api-client.sig.1 --
+life is good.
+
+**CSIT REMOVES SUPPORT FOR A VPP API VERSION**
+
+Since it is not desirable to maintain a bazillion CSIT VPP API client
+signatures, after a reasonable period of time (let's say a week), a
+CSIT developer deletes vpp-api-client.sig.1 and renames
+vpp-api-client.sig.2 to vpp-api-client.sig.1, merges to CSIT master,
+and creates a new oper branch.  At this point, VPP patches that do not
+contain the new vpe.api file will fail verification upon recheck or initial
+submission to gerrit with a "Rebase or Merge Parent" notification and
+will require rebasing to pass verification.
+
+**CSIT ADDS SUPPORT FOR A NEW FEATURE API PRIOR TO VPP**
+
+A VPP developer has lots of ideas and decides to add a new
+plugin and api which supports the "Super-Duper Feature" to VPP in
+a new plugin called the "Super-Duper Plugin" and associated super_duper.api
+VPP binary APi message definition file. Being a thoughtful and
+helpful developer, the VPP developer notifies the CSIT team providing
+them with the super_duper.api.json file. A CSIT developer
+quickly produces the Super-Duper Feature CSIT test suite and updates the VPP
+API Client signature with vpe-api-client.sig.1/plugin/super_duper.api.json.
+In the meantime, the VPP developer pushes the Super-Duper VPP patch which
+fails the CSIT VPP API CHANGE DETECTION TEST. Both developers then work
+together to verify both CSIT and the VPP patch.  The CSIT developer
+then merges the CSIT code into master and creates a new oper branch.  Our
+VPP developer is very pleased when the VPP patch containing
+the Super-Duper Plugin verifies upon recheck. All other VPP patches without
+api file changes continue to pass the CSIT VPP API CHANGE DETECTION TEST
+before and after the Super-Duper VPP patch is merged.
+
+**VPP PATCH CONTAINS A NEW FEATURE API BEFORE CSIT SUPPORT**
+
+Now let's assume that the VPP developer was having a bad day
+and forgot to notify the CSIT team about the new Super-Duper Plugin.
+Upon pushing the VPP patch to gerrit, the VPP developer is pleased that
+there is no nastygram email from the CSIT VPP API CHANGE DETECTION TEST.
+All VPP patches without api file changes continue to pass the CSIT VPP
+API CHANGE DETECTION TEST. Eventually a Super-Duper Plugin test suite is
+added to CSIT along with vpe-api-client.sig.1/plugin/super_duper.api.json
+and release in a new CSIT oper branch. All VPP patches that are do not contain
+api changes and are verified via recheck or initial submission, continue to
+pass the CSIT VPP API CHANGE DETECTION TEST.