diff options
Diffstat (limited to 'docs/automating_vpp_api_flag_day.rst')
| -rwxr-xr-x | docs/automating_vpp_api_flag_day.rst | 224 |
1 files changed, 224 insertions, 0 deletions
diff --git a/docs/automating_vpp_api_flag_day.rst b/docs/automating_vpp_api_flag_day.rst new file mode 100755 index 0000000000..2e8612669e --- /dev/null +++ b/docs/automating_vpp_api_flag_day.rst @@ -0,0 +1,224 @@ +.. + Copyright (c) 2019 Cisco and/or its affiliates. + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at: +.. + http://www.apache.org/licenses/LICENSE-2.0 +.. + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + + +AUTOMATING VPP-API FLAG DAY IN CSIT +=================================== + +**ABSTRACT** + +This document is intended to provide a solution to the problem of +automating the detection of VPP API changes which are not backwards +compatible with existing CSIT tests and to automate the "Flag Day" +process of deploying a new set of CSIT tests which are compatible +with the new version of the VPP API without causing a halt to the +normal VPP/CSIT operational CI process. This shall initially be +limited to changes in \*.api files contained in the vpp repo. +Eventually the detection algorithm could be extended to include +other integration points such as "directory" structure of stats +segment or PAPI python library dependencies. + +**VPP API FLAG DAY ALGORITHM USE CASE** + +The following steps describe the use case of the proposed +implementation for automating the VPP API "Flag Day" algorithm: + +#. A VPP patch with VPP API changes is submitted to + gerrit for review +#. CSIT verify job detects the VPP API change(s). + Note: This requires a new CSIT test described below in the + section "CSIT VPP API CHANGE DETECTION TEST". +#. CSIT verify job fails before starting to run any tests and + an email with the appropriate reason is sent to the VPP patch + submitter and vpp-api-dev@lists.fd.io including the VPP patch + information and API change(s) that were detected. +#. The VPP patch developer and CSIT team create a CSIT JIRA ticket + to identify the work required to support the new VPP API version. +#. CSIT developer creates a patch to existing CSIT tests to support + the new VPP API version and new features in the VPP patch as required. +#. CSIT developer runs CI verification tests for the CSIT patch against + the VPP patch (aka "Patch-On-Patch verification). + This process verifies support for the new VPP API, associated VPP + features and CSIT tests. Both developers iterate until the + verification passes. +#. CSIT committer updates the supported API signature(s) such that + all signatures included in the CSIT branch are supported by the + current version of the CSIT tests. See "API FLAG DAY SCENERIOS" below + for examples. +#. CSIT committer merges CSIT patch and creates a new operational + branch in the CSIT repo. +#. VPP developer issues a recheck on the VPP patch and a VPP + Committer merges the patch. +#. Recheck of existing VPP patches in gerrit may cause the "VPP + API Incompatible Change Test" to send an email to the patch + submitter to rebase the patch to pick up the compatible VPP API + version files. + +**FEATURES REQUIRED FOR IMPLEMENTATION** + +**VPP API SIGNATURE GENERATION** + +The VPP PAPI generation already produces the complete set of +signatures in JSON format for all api files and includes them in the +vpp-api-python.deb package. Upon installation all of the \*.api.json +files are installed in the /usr/share/vpp/api directory. Each record +in the .api.json file contains the name of the api message, the fields +and their data types, and a CRC of the json object. + +**VPP API CLIENT SIGNATURES** + +In each CSIT branch, all of the VPP API client signatures that are supported +by the CSIT tests in that branch are contained in separate directories +under the .../csit/resources/api/vpp directory. The CSIT VPP API +client signature directory structure is the same as the one published in +/usr/share/vpp/api as generated by vppapigen. + +The granularity of the CSIT VPP API client signature support +will initially be on a per VPP API JSON file. In the future, a per VPP +api message level of granularity may be added. If CSIT is capable of +supporting more than one version of a VPP API JSON file, then a new +CSIT VPP api client signature directory will be created containing +all of the supported VPP API JSON files. Typically this will be identical +to the previous version with the exception of the VPP API JSON files +which have been changed in the VPP patch which triggered the VPP API FLAG +day algorithm. + +See https://gerrit.fd.io/r/19027 for the baseline implementation. + +**VPP API CHANGES FILE INCLUDED in VPP PACKAGE** + +The VPP build system shall add a file in the /usr/share/vpp/api +directory of the vpp package which is the same directory in which +the api JSON files are published. This file will include the list of +VPP api files which were included in the patch to be verified. + +See https://gerrit.fd.io/r/19479 for the baseline implementation. + +**CSIT VPP API CHANGE DETECTION TEST** + +The set of VPP api signatures which are supported by the CSIT tests in +a given CSIT branch shall be stored in .../csit/resources/api/vpp which +mirrors the same directory structure as the API signature directory +generated by vppapigen (e.g. /usr/share/api/vpp/core & +/usr/share/api/vpp/plugins). + +The test compares the VPP patch's API signature directory with each of +the CSIT VPP API signabture directory and determine the following state: + +- No Change +- Changed +- Rebase or Merge Parent VPP Patch [0] + +[0] The Rebase or Merge Parent VPP Patch result occurs when there is no valid API +signature found in .../csit/resources/api/vpp AND there are no VPP API changes +included in the patch. This could be the result of a patch whose parent does not +include the API changes merged in another VPP patch and supported by the new CSIT +oper branch. This case would be resolved by rebasing the patch to HEAD. The other +possibility is that the patch is a descendent of a patch with an incompatible API +change that has not been merged yet. This case is resolved by completing the API +Flag Day algorithm on the parent patch such that the latest CSIT oper branch supports +the API in the parent. This importance of the detection of this state is to provide +direct feedback to the VPP patch author about how to resolve the issue in a timely +manner. + +Any condition other than "No Change" shall cause an email to be sent +to the VPP patch submitter. If the condition is "Changed" then +vpp-api-dev@lists.fd.io shall also be copied on the notification email. + +**RUN CSIT VERIFY JOB AGAINST A SPECIFIC VPP PATCH IN GERRIT REVIEW BRANCH** + +This is the "Patch-On-Patch" methodology documented in [TBD]? + + +**VPP API FLAG DAY SCENARIOS** + +In the beginning, let's assume there is a single VPP API Client signature +directory in the current oper branch called vpp-api-client.sig.1 which +contains core/vpe.api.json and plugin/acl.api.json which are supported +by the CSIT tests. + +**VPP PATCH CONTAINS INCOMPATIBLE API CHANGES** + +Next, a VPP developer modifies vpe.api with a whole set of +new type definitions. When the patch is submitted to gerrit.fd.io, the +"CSIT VPP API CHANGE DETECTION TEST" detects the changed api file and +votes Verified -1. Once CSIT has been updated to support the new type +definitions and verified against the VPP patch, +vpp-api-client.sig.1/core/vpe.api.json is replaced with the vpe.api.json +file from the patch. The CSIT changes are committed into CSIT master and a +new oper branch is created. The VPP patch is then rechecked and merged +into VPP master as soon as practicable. All existing VPP patches and any +new patches not including the VPP api change patch will fail verification +with a "Rebase or Merge Parent" notification upon recheck or initial +submission to gerrit. Rebasing is then required in order to pass +verification of the new api changes. + +**VPP PATCH CONTAINS BACKWARDS COMPATIBLE CHANGES** + +The next day, a VPP developer finds a need to add a new +attribute to an api message in vpe.api with a default value defined. +This is a backwards compatible change for CSIT. Since the "CSIT VPP +API CHANGE DETECTION TEST" only works on a per api file level of granularity, +the change is flagged with Verified -1. However, in this case, the +CSIT developer can resolve the verify failure by adding a second VPP API +client signature directory, vpp-api-client.sig.2 which is a copy of +vpp-api-client.sig.1 with the vpe.api.json file updated with the contents +of the copy from the VPP patch. After the CSIT changes are merged and a new +CSIT oper branch is created, the VPP patch will pass verification upon recheck. +All other patches will continue to pass verification upon recheck or initial +submission to gerrit by matching the signature in vpp-api-client.sig.1 -- +life is good. + +**CSIT REMOVES SUPPORT FOR A VPP API VERSION** + +Since it is not desirable to maintain a bazillion CSIT VPP API client +signatures, after a reasonable period of time (let's say a week), a +CSIT developer deletes vpp-api-client.sig.1 and renames +vpp-api-client.sig.2 to vpp-api-client.sig.1, merges to CSIT master, +and creates a new oper branch. At this point, VPP patches that do not +contain the new vpe.api file will fail verification upon recheck or initial +submission to gerrit with a "Rebase or Merge Parent" notification and +will require rebasing to pass verification. + +**CSIT ADDS SUPPORT FOR A NEW FEATURE API PRIOR TO VPP** + +A VPP developer has lots of ideas and decides to add a new +plugin and api which supports the "Super-Duper Feature" to VPP in +a new plugin called the "Super-Duper Plugin" and associated super_duper.api +VPP binary APi message definition file. Being a thoughtful and +helpful developer, the VPP developer notifies the CSIT team providing +them with the super_duper.api.json file. A CSIT developer +quickly produces the Super-Duper Feature CSIT test suite and updates the VPP +API Client signature with vpe-api-client.sig.1/plugin/super_duper.api.json. +In the meantime, the VPP developer pushes the Super-Duper VPP patch which +fails the CSIT VPP API CHANGE DETECTION TEST. Both developers then work +together to verify both CSIT and the VPP patch. The CSIT developer +then merges the CSIT code into master and creates a new oper branch. Our +VPP developer is very pleased when the VPP patch containing +the Super-Duper Plugin verifies upon recheck. All other VPP patches without +api file changes continue to pass the CSIT VPP API CHANGE DETECTION TEST +before and after the Super-Duper VPP patch is merged. + +**VPP PATCH CONTAINS A NEW FEATURE API BEFORE CSIT SUPPORT** + +Now let's assume that the VPP developer was having a bad day +and forgot to notify the CSIT team about the new Super-Duper Plugin. +Upon pushing the VPP patch to gerrit, the VPP developer is pleased that +there is no nastygram email from the CSIT VPP API CHANGE DETECTION TEST. +All VPP patches without api file changes continue to pass the CSIT VPP +API CHANGE DETECTION TEST. Eventually a Super-Duper Plugin test suite is +added to CSIT along with vpe-api-client.sig.1/plugin/super_duper.api.json +and release in a new CSIT oper branch. All VPP patches that are do not contain +api changes and are verified via recheck or initial submission, continue to +pass the CSIT VPP API CHANGE DETECTION TEST. |