path: root/docs/automating_vpp_api_flag_day.rst

..
   Copyright (c) 2019 Cisco and/or its affiliates.
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at:
..
       http://www.apache.org/licenses/LICENSE-2.0
..
   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.


AUTOMATING VPP-API FLAG DAY IN CSIT
===================================

**ABSTRACT**

This document is intended to provide a solution to the problem of
automating the detection of VPP API changes which are not backwards
compatible with existing CSIT tests and to automate the "Flag Day"
process of deploying a new set of CSIT tests which are compatible
with the new version of the VPP API without causing a halt to the
normal VPP/CSIT operational CI process. This shall initially be
limited to changes in \*.api files contained in the vpp repo.
Eventually the detection algorithm could be extended to include
other integration points such as "directory" structure of stats
segment or PAPI python library dependencies.

**MOTIVATION**

Aside of per-release activities (release report), CSIT also provides testing
that requires somewhat tight coupling to the latest (merged but not released)
VPP code. Currently, the coupling is not as tight as possible.
Instead of testing HEAD builds of VPP directly with HEAD CSIT code,
HEAD of one project is run with somewhat older codebase of the other project.
Definition of what is the older codebase to use is maintained by CSIT project.
For older CSIT codebase, the project creates so-called "oper" branches.
For older VPP codebase, CSIT master HEAD contains identifiers
for "stable" VPP builds. Such older codebases are also used for verify jobs,
where HEAD of the other project is replaced by the commit under review.

One particular type of jobs useful for VPP development is trending jobs.
They test latests VPP build with oper branch of CSIT,
and analytics is applied to detect regressions in preformance.
For this to work properly, VPP project needs a warning against breaking
the assumptions the current oper branch makes about VPP behavior.
In the past, the most frequent type of such breakage was API change.

Earlier attempts to create a process to minimize breakage have focused
on creating a new verify job for VPP (called api-crc job) that
votes -1 on a change that affect CRC values for API messages CSIT uses.
The list of messages and CRC values (multiple values are allowed)
is maintained in CSIT repository (in oper branch).
The process was less explicit on how should CSIT project maintain such list.
As CSIT was not willing to support two incpompatible API messages
by the same codebase (commit), there were unavoidable windows
where either trenging jobs, or CSIT verify jobs were failing.

Practice showed that human (or infra) errors can create two kinds of breakages.
Either the unavoidable short window gets long, affecting a trending job run
or two, or the api-crc job starts giving -1 to innocent changes
because oper branch went out of sync with VPP HEAD codebase.
This second type of failure prevents any merges to VPP for a long time
(12 hours is the typical time, give time zone differences).

The current version of this document is still being somewhat vague
on the CSIT side of the process, but introduces a new requirement:
The api-crc job should not give false -1, under any reasonable circumstance.
That means, if a VPP change (nor any of its unmerged ancestor commits)
does not affect any CRC values for messages used by CSIT,
-1 should only mean "rebase is needed", and rebasing to HEAD should result
in +1 from the api-crc job.

**VPP API FLAG DAY ALGORITHM USE CASE**

The following steps describe the use case of the proposed
implementation for automating the VPP API "Flag Day" algorithm:

#. A VPP patch with VPP API changes is submitted to
   gerrit for review. This usually means .api files are edited,
   but a patch that affects the way CRCs are computed is also an API change.
#. The api-crc job detects the API CRC values have changed
   for some messages used by CSIT.
   Note: This requires a new CSIT test described below in the
   section "CSIT VPP API CHANGE DETECTION TEST".
#. The api-crc job runs in parallel with any other VPP verify job,
   so other jobs can hint at the impact on CSIT.
   Some API changes do not need any edit on CSIT side, except for the CRC value.
   If the api-crc job fails, an email with the appropriate reason
   is sent to the VPP patch submitter and vpp-api-dev@lists.fd.io
   including the VPP patch information and API change(s) that were detected.
#. The VPP patch developer and CSIT team create a CSIT JIRA ticket
   to identify the work required to support the new VPP API version.
#. CSIT developer creates a patch to existing CSIT tests to support
   the new VPP API version and new features in the VPP patch as required.
#. CSIT developer runs CI verification tests (without CRC checks)
   for the CSIT patch against the VPP patch (aka "Patch-On-Patch verification").
   This process verifies support for the new VPP API, associated VPP
   features and CSIT tests.  Both developers iterate until the
   verification passes.
#. CSIT developer creates a separate change, that only edits the CSIT CRC list
   to include both stable VPP CRCs, and CRCs of the VPP change under review.
#. CSIT committer requires proofs both CRCs are included correctly
   (for example the api-crc job is run with overriden CSIT_REF parameter).
#. When CSIT committer sees both CSIT changes are ready,
   the CRC-editing change is merged to CSIT master branch
   and cherry-picked to the latest oper branch.
   This does not break any jobs.
#. VPP developer issues a recheck on the VPP patch
#. On success, VPP Committer merges the patch.
#. VPP committer sends an e-mail to vpp-api-dev stating the support for
   the previous CRC values will soon be removed.
#. VPP merge jobs create and upload new VPP packages.
   This breaks trending jobs, but both VPP and CSIT verify jobs still work.
#. CSIT developer bumps stable vpp version in the test affecting change
   and rechecks it still works.
#. CSIT committer merges this change (to both master and oper).
   This fixes trending jobs. Both VPP and CSIT verify jobs continue to work.
#. CSIT committer creates a patch that removes the old CRC values.
#. CSIT committer merges that patch.
   This breaks verify jobs for old changes in VPP.
#. CSIT committer sends an e-mail to vpp-api-dev stating the support for
   the previous CRC values has been removed, and rebase is needed
   for all affected VPP changes.
#. Recheck of existing VPP patches in gerrit may cause the "VPP
   API Incompatible Change Test" to send an email to the patch
   submitter to rebase the patch to pick up the compatible VPP API
   version files.

**FEATURES REQUIRED FOR IMPLEMENTATION**

**VPP API SIGNATURE GENERATION**

The VPP PAPI generation already produces the complete set of
signatures in JSON format for all api files and includes them in the
vpp-api-python.deb package.  Upon installation all of the \*.api.json
files are installed in the /usr/share/vpp/api directory.  Each record
in the .api.json file contains the name of the api message, the fields
and their data types, and a CRC of the json object.

**VPP API CLIENT SIGNATURES**

In each CSIT branch, all of the VPP API client signatures that are supported
by the CSIT tests in that branch are contained in separate directories
under the .../csit/resources/api/vpp directory. The CSIT VPP API
client signature directory structure is the same as the one published in
/usr/share/vpp/api as generated by vppapigen.

The granularity of the CSIT VPP API client signature support
will initially be on a per VPP API JSON file.  In the future, a per VPP
api message level of granularity may be added.  If CSIT is capable of
supporting more than one version of a VPP API JSON file, then a new
CSIT VPP api client signature directory will be created containing
all of the supported VPP API JSON files.  Typically this will be identical
to the previous version with the exception of the VPP API JSON files
which have been changed in the VPP patch which triggered the VPP API FLAG
day algorithm.

See https://gerrit.fd.io/r/19027 for the baseline implementation.

**VPP API CHANGES FILE INCLUDED in VPP PACKAGE**

The VPP build system shall add a file in the /usr/share/vpp/api
directory of the vpp package which is the same directory in which
the api JSON files are published.  This file will include the list of
VPP api files which were included in the patch to be verified.

See https://gerrit.fd.io/r/19479 for the baseline implementation.

**CSIT VPP API CHANGE DETECTION TEST**

The set of VPP api signatures which are supported by the CSIT tests in
a given CSIT branch shall be stored in .../csit/resources/api/vpp which
mirrors the same directory structure as the API signature directory
generated by vppapigen (e.g. /usr/share/api/vpp/core &
/usr/share/api/vpp/plugins).

The test compares the VPP patch's API signature directory with each of
the CSIT VPP API signabture directory and determine the following state:

- No Change
- Changed
- Rebase or Merge Parent VPP Patch [0]

[0] The Rebase or Merge Parent VPP Patch result occurs when there is no valid API
signature found in .../csit/resources/api/vpp AND there are no VPP API changes
included in the patch.  This could be the result of a patch whose parent does not
include the API changes merged in another VPP patch and supported by the new CSIT
oper branch.  This case would be resolved by rebasing the patch to HEAD.  The other
possibility is that the patch is a descendent of a patch with an incompatible API
change that has not been merged yet.  This case is resolved by completing the API
Flag Day algorithm on the parent patch such that the latest CSIT oper branch supports
the API in the parent.  This importance of the detection of this state is to provide
direct feedback to the VPP patch author about how to resolve the issue in a timely
manner.

Any condition other than "No Change" shall cause an email to be sent
to the VPP patch submitter.  If the condition is "Changed" then
vpp-api-dev@lists.fd.io shall also be copied on the notification email.

**RUN CSIT VERIFY JOB AGAINST A SPECIFIC VPP PATCH IN GERRIT REVIEW BRANCH**

This is the "Patch-On-Patch" methodology documented in [TBD]?


**VPP API FLAG DAY SCENARIOS**

In the beginning, let's assume there is a single VPP API Client signature
directory in the current oper branch called vpp-api-client.sig.1 which
contains core/vpe.api.json and plugin/acl.api.json which are supported
by the CSIT tests.

**VPP PATCH CONTAINS INCOMPATIBLE API CHANGES**

Next, a VPP developer modifies vpe.api with a whole set of
new type definitions.  When the patch is submitted to gerrit.fd.io, the
"CSIT VPP API CHANGE DETECTION TEST" detects the changed api file and
votes Verified -1.  Once CSIT has been updated to support the new type
definitions and verified against the VPP patch,
vpp-api-client.sig.1/core/vpe.api.json is replaced with the vpe.api.json
file from the patch. The CSIT changes are committed into CSIT master and a
new oper branch is created. The VPP patch is then rechecked and merged
into VPP master as soon as practicable. All existing VPP patches and any
new patches not including the VPP api change patch will fail verification
with a "Rebase or Merge Parent" notification upon recheck or initial
submission to gerrit.  Rebasing is then required in order to pass
verification of the new api changes.

**VPP PATCH CONTAINS BACKWARDS COMPATIBLE CHANGES**

The next day, a VPP developer finds a need to add a new
attribute to an api message in vpe.api with a default value defined.
This is a backwards compatible change for CSIT.  Since the "CSIT VPP
API CHANGE DETECTION TEST" only works on a per api file level of granularity,
the change is flagged with Verified -1.  However, in this case, the
CSIT developer can resolve the verify failure by adding a second VPP API
client signature directory, vpp-api-client.sig.2 which is a copy of
vpp-api-client.sig.1 with the vpe.api.json file updated with the contents
of the copy from the VPP patch.  After the CSIT changes are merged and a new
CSIT oper branch is created, the VPP patch will pass verification upon recheck.
All other patches will continue to pass verification upon recheck or initial
submission to gerrit by matching the signature in  vpp-api-client.sig.1 --
life is good.

**CSIT REMOVES SUPPORT FOR A VPP API VERSION**

Since it is not desirable to maintain a bazillion CSIT VPP API client
signatures, after a reasonable period of time (let's say a week), a
CSIT developer deletes vpp-api-client.sig.1 and renames
vpp-api-client.sig.2 to vpp-api-client.sig.1, merges to CSIT master,
and creates a new oper branch.  At this point, VPP patches that do not
contain the new vpe.api file will fail verification upon recheck or initial
submission to gerrit with a "Rebase or Merge Parent" notification and
will require rebasing to pass verification.

**CSIT ADDS SUPPORT FOR A NEW FEATURE API PRIOR TO VPP**

A VPP developer has lots of ideas and decides to add a new
plugin and api which supports the "Super-Duper Feature" to VPP in
a new plugin called the "Super-Duper Plugin" and associated super_duper.api
VPP binary APi message definition file. Being a thoughtful and
helpful developer, the VPP developer notifies the CSIT team providing
them with the super_duper.api.json file. A CSIT developer
quickly produces the Super-Duper Feature CSIT test suite and updates the VPP
API Client signature with vpe-api-client.sig.1/plugin/super_duper.api.json.
In the meantime, the VPP developer pushes the Super-Duper VPP patch which
fails the CSIT VPP API CHANGE DETECTION TEST. Both developers then work
together to verify both CSIT and the VPP patch.  The CSIT developer
then merges the CSIT code into master and creates a new oper branch.  Our
VPP developer is very pleased when the VPP patch containing
the Super-Duper Plugin verifies upon recheck. All other VPP patches without
api file changes continue to pass the CSIT VPP API CHANGE DETECTION TEST
before and after the Super-Duper VPP patch is merged.

**VPP PATCH CONTAINS A NEW FEATURE API BEFORE CSIT SUPPORT**

Now let's assume that the VPP developer was having a bad day
and forgot to notify the CSIT team about the new Super-Duper Plugin.
Upon pushing the VPP patch to gerrit, the VPP developer is pleased that
there is no nastygram email from the CSIT VPP API CHANGE DETECTION TEST.
All VPP patches without api file changes continue to pass the CSIT VPP
API CHANGE DETECTION TEST. Eventually a Super-Duper Plugin test suite is
added to CSIT along with vpe-api-client.sig.1/plugin/super_duper.api.json
and release in a new CSIT oper branch. All VPP patches that are do not contain
api changes and are verified via recheck or initial submission, continue to
pass the CSIT VPP API CHANGE DETECTION TEST.